2024-04-22T01:01:03.999 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T01:01:04.000 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quota} --archive /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725 --name teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/orig.config.yaml 2024-04-22T01:01:04.044 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T01:01:04.129 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725 branch: squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quota} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667725' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 676 sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b sleep_before_teardown: 0 subset: 46/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBT/4mDsuiWTW/0UQNTgYB632NwPv23SHEKXswKvSLs3w6WQsINlxKnxQ+USx0MW9l9A7O6oXbDltROIEBWszbE= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBVkidFhzskkhvOaYJ2UFXFdVXUNhYyYHmYfy3KlfDCrPJLBxoM8mqDNyR85QobvOMariv66l2yHH/XOKE5kW0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_quota teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T01:01:04.129 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa; will attempt to use it 2024-04-22T01:01:04.130 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks 2024-04-22T01:01:04.130 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T01:01:04.131 INFO:teuthology.task.internal:Checking packages... 2024-04-22T01:01:04.155 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '820a2f8ced66152ed3d765c31444b8076ea28c6b' 2024-04-22T01:01:04.155 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T01:01:04.155 INFO:teuthology.packaging:ref: None 2024-04-22T01:01:04.155 INFO:teuthology.packaging:tag: None 2024-04-22T01:01:04.155 INFO:teuthology.packaging:branch: squid 2024-04-22T01:01:04.155 INFO:teuthology.packaging:sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:04.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-22T01:01:04.495 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2407-g430e09df-1jammy 2024-04-22T01:01:04.497 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T01:01:04.503 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T01:01:04.503 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T01:01:04.529 INFO:teuthology.task.internal:Saving configuration 2024-04-22T01:01:04.545 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T01:01:04.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T01:01:04.577 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 00:55:38.023187', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMtMMBKJGq/Et7E9ztLi+iw+dqK1VfnoLTNi1yjBLPxp'} 2024-04-22T01:01:04.601 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 00:55:38.021172', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFvwh4L7z6xnLGUd6T+fi+3PWQ+R0Yls8lBOwBoqV0Qu'} 2024-04-22T01:01:04.601 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T01:01:04.608 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-22T01:01:04.608 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-22T01:01:04.608 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T01:01:04.669 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe9cc53aee0>, signals=[15]) 2024-04-22T01:01:04.670 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T01:01:04.676 INFO:teuthology.task.internal:Opening connections... 2024-04-22T01:01:04.676 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:01:04.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:01:04.759 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:01:04.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:01:04.836 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T01:01:04.847 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-22T01:01:04.853 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-22T01:01:04.853 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-22T01:01:04.899 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-22T01:01:04.899 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="Ubuntu" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="22.04" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_CODENAME=jammy 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:ID=ubuntu 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE=debian 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-22T01:01:04.900 INFO:teuthology.orchestra.run.smithi070.stdout:UBUNTU_CODENAME=jammy 2024-04-22T01:01:04.901 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-22T01:01:04.919 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-04-22T01:01:04.924 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-04-22T01:01:04.924 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-04-22T01:01:04.976 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Ubuntu" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="22.04" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_CODENAME=jammy 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:ID=ubuntu 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE=debian 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-22T01:01:04.977 INFO:teuthology.orchestra.run.smithi098.stdout:UBUNTU_CODENAME=jammy 2024-04-22T01:01:04.977 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-04-22T01:01:05.002 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T01:01:05.012 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T01:01:05.017 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T01:01:05.017 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-22T01:01:05.020 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-04-22T01:01:05.025 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T01:01:05.032 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T01:01:05.032 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-22T01:01:05.065 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-04-22T01:01:05.169 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T01:01:05.210 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T01:01:05.224 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T01:01:05.224 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T01:01:05.224 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T01:01:05.225 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T01:01:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:01:05.230 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-22T01:01:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:01:05.231 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-22T01:01:05.276 INFO:teuthology.orchestra.run.smithi070.stdout:5.15.0-56-generic 2024-04-22T01:01:05.276 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.15.0-56-generic 2024-04-22T01:01:05.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get clean 2024-04-22T01:01:05.277 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-56-generic 2024-04-22T01:01:05.277 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.15.0-56-generic 2024-04-22T01:01:05.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-04-22T01:01:05.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-04-22T01:01:05.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get update 2024-04-22T01:01:05.563 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T01:01:05.563 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-22T01:01:05.572 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-22T01:01:05.684 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T01:01:05.684 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-22T01:01:05.716 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-22T01:01:05.748 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-22T01:01:05.816 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,381 kB] 2024-04-22T01:01:05.888 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-22T01:01:05.971 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [450 kB] 2024-04-22T01:01:05.979 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [242 kB] 2024-04-22T01:01:05.985 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-22T01:01:05.986 INFO:teuthology.orchestra.run.smithi070.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-22T01:01:05.986 INFO:teuthology.orchestra.run.smithi070.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,744 kB] 2024-04-22T01:01:05.989 INFO:teuthology.orchestra.run.smithi070.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-22T01:01:06.030 INFO:teuthology.orchestra.run.smithi070.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [294 kB] 2024-04-22T01:01:06.040 INFO:teuthology.orchestra.run.smithi070.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-22T01:01:06.041 INFO:teuthology.orchestra.run.smithi070.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-22T01:01:06.059 INFO:teuthology.orchestra.run.smithi070.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-22T01:01:06.069 INFO:teuthology.orchestra.run.smithi070.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-22T01:01:06.071 INFO:teuthology.orchestra.run.smithi070.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-22T01:01:06.088 INFO:teuthology.orchestra.run.smithi070.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-22T01:01:06.088 INFO:teuthology.orchestra.run.smithi070.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-22T01:01:06.088 INFO:teuthology.orchestra.run.smithi070.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-22T01:01:06.089 INFO:teuthology.orchestra.run.smithi070.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-22T01:01:06.094 INFO:teuthology.orchestra.run.smithi070.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-22T01:01:06.094 INFO:teuthology.orchestra.run.smithi070.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-22T01:01:06.105 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-22T01:01:06.106 INFO:teuthology.orchestra.run.smithi070.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-22T01:01:06.108 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [450 kB] 2024-04-22T01:01:06.136 INFO:teuthology.orchestra.run.smithi070.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-22T01:01:06.137 INFO:teuthology.orchestra.run.smithi070.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-22T01:01:06.142 INFO:teuthology.orchestra.run.smithi070.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-22T01:01:06.143 INFO:teuthology.orchestra.run.smithi070.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-22T01:01:06.155 INFO:teuthology.orchestra.run.smithi070.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-22T01:01:06.172 INFO:teuthology.orchestra.run.smithi070.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-22T01:01:06.176 INFO:teuthology.orchestra.run.smithi070.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-22T01:01:06.177 INFO:teuthology.orchestra.run.smithi070.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-22T01:01:06.177 INFO:teuthology.orchestra.run.smithi070.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-22T01:01:06.186 INFO:teuthology.orchestra.run.smithi070.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-22T01:01:06.187 INFO:teuthology.orchestra.run.smithi070.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-22T01:01:06.187 INFO:teuthology.orchestra.run.smithi070.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-22T01:01:06.188 INFO:teuthology.orchestra.run.smithi070.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-22T01:01:06.189 INFO:teuthology.orchestra.run.smithi070.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-22T01:01:06.207 INFO:teuthology.orchestra.run.smithi070.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-22T01:01:06.207 INFO:teuthology.orchestra.run.smithi070.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-22T01:01:06.207 INFO:teuthology.orchestra.run.smithi070.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-22T01:01:06.208 INFO:teuthology.orchestra.run.smithi070.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-22T01:01:06.208 INFO:teuthology.orchestra.run.smithi070.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-22T01:01:06.241 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-22T01:01:06.405 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,381 kB] 2024-04-22T01:01:06.573 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [242 kB] 2024-04-22T01:01:06.585 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-22T01:01:06.585 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,744 kB] 2024-04-22T01:01:06.638 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-22T01:01:06.667 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-22T01:01:06.668 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [294 kB] 2024-04-22T01:01:06.676 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-22T01:01:06.677 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-22T01:01:06.678 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-22T01:01:06.695 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-22T01:01:06.715 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-22T01:01:06.715 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-22T01:01:06.716 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-22T01:01:06.770 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-22T01:01:06.776 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-22T01:01:06.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-22T01:01:06.940 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-22T01:01:07.290 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-22T01:01:07.295 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-22T01:01:07.309 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-22T01:01:07.310 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-22T01:01:07.312 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-22T01:01:07.522 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-22T01:01:07.635 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-22T01:01:07.655 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-22T01:01:07.656 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-22T01:01:07.660 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-22T01:01:07.660 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-22T01:01:07.661 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-22T01:01:07.661 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-22T01:01:07.665 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-22T01:01:07.670 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-22T01:01:07.838 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-22T01:01:07.921 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-22T01:01:08.028 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-22T01:01:08.086 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-22T01:01:08.121 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-22T01:01:11.742 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 13.1 MB in 2s (6,980 kB/s) 2024-04-22T01:01:12.456 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 13.1 MB in 3s (4,957 kB/s) 2024-04-22T01:01:13.076 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:01:13.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-22T01:01:13.155 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:01:13.355 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-04-22T01:01:13.356 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-04-22T01:01:13.535 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:01:13.535 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:01:13.535 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:01:13.536 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:01:13.536 INFO:teuthology.orchestra.run.smithi070.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-22T01:01:13.536 INFO:teuthology.orchestra.run.smithi070.stdout: python2.7-minimal 2024-04-22T01:01:13.536 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:01:13.537 INFO:teuthology.orchestra.run.smithi070.stdout:The following additional packages will be installed: 2024-04-22T01:01:13.537 INFO:teuthology.orchestra.run.smithi070.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-22T01:01:13.537 INFO:teuthology.orchestra.run.smithi070.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-22T01:01:13.538 INFO:teuthology.orchestra.run.smithi070.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-22T01:01:13.539 INFO:teuthology.orchestra.run.smithi070.stdout:Suggested packages: 2024-04-22T01:01:13.539 INFO:teuthology.orchestra.run.smithi070.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-22T01:01:13.539 INFO:teuthology.orchestra.run.smithi070.stdout:Recommended packages: 2024-04-22T01:01:13.539 INFO:teuthology.orchestra.run.smithi070.stdout: thermald 2024-04-22T01:01:13.598 INFO:teuthology.orchestra.run.smithi070.stdout:The following NEW packages will be installed: 2024-04-22T01:01:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-22T01:01:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-22T01:01:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-22T01:01:13.600 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages will be upgraded: 2024-04-22T01:01:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-22T01:01:13.716 INFO:teuthology.orchestra.run.smithi070.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-22T01:01:13.716 INFO:teuthology.orchestra.run.smithi070.stdout:Need to get 113 MB of archives. 2024-04-22T01:01:13.716 INFO:teuthology.orchestra.run.smithi070.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-22T01:01:13.716 INFO:teuthology.orchestra.run.smithi070.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-22T01:01:13.781 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:01:13.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-22T01:01:13.861 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:01:14.060 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-22T01:01:14.061 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-22T01:01:14.234 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:01:14.234 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:01:14.234 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:01:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:01:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-22T01:01:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: python2.7-minimal 2024-04-22T01:01:14.235 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:01:14.236 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-04-22T01:01:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-22T01:01:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-22T01:01:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-22T01:01:14.238 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-04-22T01:01:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-22T01:01:14.238 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-04-22T01:01:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: thermald 2024-04-22T01:01:14.284 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-04-22T01:01:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-22T01:01:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-22T01:01:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-22T01:01:14.286 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-04-22T01:01:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-22T01:01:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-22T01:01:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 113 MB of archives. 2024-04-22T01:01:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-22T01:01:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-22T01:01:15.079 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-22T01:01:15.391 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-22T01:01:15.510 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-22T01:01:15.739 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-22T01:01:17.579 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-22T01:01:17.579 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-22T01:01:17.580 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-22T01:01:17.771 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-22T01:01:17.786 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-22T01:01:17.786 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-22T01:01:17.786 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-22T01:01:17.815 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-22T01:01:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-22T01:01:18.042 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-22T01:01:18.217 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 113 MB in 4s (26.9 MB/s) 2024-04-22T01:01:18.424 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-22T01:01:18.434 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 113 MB in 4s (30.2 MB/s) 2024-04-22T01:01:18.587 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-22T01:01:20.393 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-22T01:01:20.398 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:20.464 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:20.573 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-22T01:01:20.577 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:20.644 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-22T01:01:23.167 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:23.280 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:23.341 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-22T01:01:23.362 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:23.441 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:23.758 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-22T01:01:23.778 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:23.816 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:23.911 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-22T01:01:23.931 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:23.961 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:30.302 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T01:01:30.348 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T01:01:30.427 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T01:01:30.474 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T01:01:30.785 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T01:01:30.814 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T01:01:30.914 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T01:01:30.936 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T01:01:31.200 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-22T01:01:31.205 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-22T01:01:31.216 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-22T01:01:31.222 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-22T01:01:31.255 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-22T01:01:31.266 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-22T01:01:36.564 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-22T01:01:36.600 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:36.647 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:36.652 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-22T01:01:36.689 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T01:01:36.718 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:39.787 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T01:01:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T01:01:39.915 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T01:01:39.927 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T01:01:40.260 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-22T01:01:40.282 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-22T01:01:40.382 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:40.403 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:40.491 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-22T01:01:40.529 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-22T01:01:40.595 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:40.658 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-22T01:01:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-22T01:01:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-22T01:01:41.939 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-22T01:01:41.967 INFO:teuthology.orchestra.run.smithi070.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-22T01:01:41.967 INFO:teuthology.orchestra.run.smithi070.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-22T01:01:41.967 INFO:teuthology.orchestra.run.smithi070.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-22T01:01:41.967 INFO:teuthology.orchestra.run.smithi070.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-22T01:01:42.097 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:42.119 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:43.475 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:43.512 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:44.842 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-22T01:01:44.888 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-22T01:01:44.943 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-22T01:01:45.014 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-22T01:01:45.044 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:45.113 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-22T01:01:45.113 INFO:teuthology.orchestra.run.smithi098.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-22T01:01:45.141 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T01:01:45.214 INFO:teuthology.orchestra.run.smithi070.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-22T01:01:45.214 INFO:teuthology.orchestra.run.smithi070.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-22T01:01:59.287 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-22T01:01:59.288 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub' 2024-04-22T01:01:59.309 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-22T01:01:59.333 INFO:teuthology.orchestra.run.smithi098.stdout:Generating grub configuration file ... 2024-04-22T01:01:59.410 INFO:teuthology.orchestra.run.smithi070.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-22T01:01:59.410 INFO:teuthology.orchestra.run.smithi070.stdout:Sourcing file `/etc/default/grub' 2024-04-22T01:01:59.432 INFO:teuthology.orchestra.run.smithi070.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-22T01:01:59.458 INFO:teuthology.orchestra.run.smithi070.stdout:Generating grub configuration file ... 2024-04-22T01:01:59.605 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-22T01:01:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-22T01:01:59.739 INFO:teuthology.orchestra.run.smithi070.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-22T01:01:59.758 INFO:teuthology.orchestra.run.smithi070.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-22T01:01:59.890 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-22T01:01:59.895 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-22T01:01:59.960 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-22T01:01:59.965 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-22T01:02:00.030 INFO:teuthology.orchestra.run.smithi070.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-22T01:02:00.035 INFO:teuthology.orchestra.run.smithi070.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-22T01:02:00.101 INFO:teuthology.orchestra.run.smithi070.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-22T01:02:00.107 INFO:teuthology.orchestra.run.smithi070.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-22T01:02:00.119 INFO:teuthology.orchestra.run.smithi098.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-22T01:02:00.119 INFO:teuthology.orchestra.run.smithi098.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-22T01:02:00.119 INFO:teuthology.orchestra.run.smithi098.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-22T01:02:00.170 INFO:teuthology.orchestra.run.smithi098.stdout:done 2024-04-22T01:02:00.286 INFO:teuthology.orchestra.run.smithi070.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-22T01:02:00.286 INFO:teuthology.orchestra.run.smithi070.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-22T01:02:00.286 INFO:teuthology.orchestra.run.smithi070.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-22T01:02:00.322 INFO:teuthology.orchestra.run.smithi070.stdout:done 2024-04-22T01:02:00.585 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-04-22T01:02:00.611 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 21 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.15.0.105.102 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-04-22T01:02:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-04-22T01:02:00.613 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-22T01:02:00.613 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-22T01:02:00.613 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-22T01:02:00.613 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-22T01:02:00.613 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-04-22T01:02:00.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-04-22T01:02:00.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-04-22T01:02:00.803 DEBUG:teuthology.orchestra.run.smithi070:> dpkg -s linux-image-generic 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Package: linux-image-generic 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Status: install ok installed 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Priority: optional 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Section: kernel 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Installed-Size: 21 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Maintainer: Ubuntu Kernel Team 2024-04-22T01:02:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: amd64 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout:Source: linux-meta 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout:Version: 5.15.0.105.102 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout:Recommends: thermald 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout:Description: Generic Linux kernel image 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout: This package will always depend on the latest generic kernel image 2024-04-22T01:02:00.831 INFO:teuthology.orchestra.run.smithi070.stdout: available. 2024-04-22T01:02:00.831 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-22T01:02:00.831 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-22T01:02:00.831 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-22T01:02:00.831 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-22T01:02:00.831 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-04-22T01:02:00.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get clean 2024-04-22T01:02:01.004 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T01:02:01.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get update 2024-04-22T01:02:01.030 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-22T01:02:01.068 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-22T01:02:01.133 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-22T01:02:01.150 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-22T01:02:01.238 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T01:02:01.322 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-22T01:02:01.401 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-22T01:02:02.368 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:02:02.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-22T01:02:02.441 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:02:02.444 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:02:02.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-22T01:02:02.510 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:02:02.644 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-22T01:02:02.644 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-22T01:02:02.686 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-04-22T01:02:02.687 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-04-22T01:02:02.810 INFO:teuthology.orchestra.run.smithi098.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-22T01:02:02.810 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:02:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:02:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:02:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:02:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T01:02:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T01:02:02.811 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:02:02.841 INFO:teuthology.orchestra.run.smithi070.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-22T01:02:02.841 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:02:02.841 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:02:02.841 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:02:02.841 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:02:02.842 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T01:02:02.842 INFO:teuthology.orchestra.run.smithi070.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T01:02:02.842 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:02:02.849 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-22T01:02:02.851 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-04-22T01:02:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-04-22T01:02:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-04-22T01:02:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-04-22T01:02:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-04-22T01:02:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 21 2024-04-22T01:02:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.15.0.105.102 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-04-22T01:02:02.867 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-04-22T01:02:02.867 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-04-22T01:02:02.879 INFO:teuthology.orchestra.run.smithi070.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-22T01:02:02.881 DEBUG:teuthology.orchestra.run.smithi070:> dpkg -s linux-image-generic 2024-04-22T01:02:02.912 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.Hil7GXHj5d 2024-04-22T01:02:02.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Hil7GXHj5d 2024-04-22T01:02:02.933 INFO:teuthology.orchestra.run.smithi070.stdout:Package: linux-image-generic 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Status: install ok installed 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Priority: optional 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Section: kernel 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Installed-Size: 21 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Maintainer: Ubuntu Kernel Team 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: amd64 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Source: linux-meta 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Version: 5.15.0.105.102 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Recommends: thermald 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout:Description: Generic Linux kernel image 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout: This package will always depend on the latest generic kernel image 2024-04-22T01:02:02.934 INFO:teuthology.orchestra.run.smithi070.stdout: available. 2024-04-22T01:02:02.935 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-22T01:02:02.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.Hil7GXHj5d 2024-04-22T01:02:02.978 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.Ko4aR6qHss 2024-04-22T01:02:02.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Ko4aR6qHss 2024-04-22T01:02:03.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.Ko4aR6qHss 2024-04-22T01:02:03.095 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.Hil7GXHj5d is 10KB 2024-04-22T01:02:03.109 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.Hil7GXHj5d 2024-04-22T01:02:03.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-22T01:02:03.157 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.Ko4aR6qHss is 10KB 2024-04-22T01:02:03.167 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:02:03.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-22T01:02:03.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-22T01:02:03.170 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.Ko4aR6qHss 2024-04-22T01:02:03.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-22T01:02:03.228 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:02:03.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-22T01:02:03.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-22T01:02:03.244 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-22T01:02:03.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo update-grub 2024-04-22T01:02:03.292 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-22T01:02:03.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo update-grub 2024-04-22T01:02:04.465 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2024-04-22T01:02:04.466 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-22T01:02:04.469 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-04-22T01:02:04.474 INFO:teuthology.orchestra.run.smithi070.stderr:Sourcing file `/etc/default/grub' 2024-04-22T01:02:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-22T01:02:04.478 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-04-22T01:02:04.647 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-22T01:02:04.652 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-22T01:02:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-22T01:02:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-22T01:02:04.883 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-22T01:02:04.888 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-22T01:02:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-22T01:02:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-22T01:02:04.926 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-22T01:02:04.931 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-22T01:02:04.931 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-22T01:02:04.935 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-22T01:02:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-22T01:02:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-22T01:02:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-22T01:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-22T01:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-22T01:02:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-22T01:02:05.074 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-04-22T01:02:05.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-04-22T01:02:05.093 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-04-22T01:02:05.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-04-22T01:02:35.122 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-22T01:02:35.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-22T01:02:35.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:02:35.126 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-22T01:02:35.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:02:35.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:02:53.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-22T01:02:53.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-04-22T01:03:02.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-22T01:03:02.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:03:02.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:03:02.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:03:05.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-04-22T01:03:05.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-22T01:03:17.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:03:17.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:03:17.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-22T01:03:17.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:03:35.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-04-22T01:03:49.264 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-22T01:03:49.751 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-22T01:03:49.751 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-22T01:03:49.751 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-22T01:03:49.798 INFO:teuthology.orchestra.run.smithi070.stdout:5.15.0-105-generic 2024-04-22T01:03:49.798 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-22T01:03:49.798 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-22T01:03:49.798 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-22T01:03:50.799 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T01:03:50.799 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T01:03:50.888 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-04-22T01:03:50.920 DEBUG:teuthology.parallel:result is None 2024-04-22T01:03:50.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:03:50.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:03:51.581 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-22T01:03:52.359 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:03:52.359 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-22T01:03:52.360 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-22T01:03:52.407 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-105-generic 2024-04-22T01:03:52.408 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-22T01:03:52.408 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-22T01:03:52.408 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-22T01:03:53.409 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-22T01:03:53.409 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T01:03:53.499 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-04-22T01:03:53.533 DEBUG:teuthology.parallel:result is None 2024-04-22T01:03:53.533 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T01:03:53.541 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T01:03:53.541 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T01:03:53.544 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T01:03:53.549 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T01:03:53.556 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T01:03:53.566 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T01:03:53.566 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T01:03:53.591 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T01:03:53.691 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T01:03:53.697 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T01:03:53.697 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T01:03:53.700 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T01:03:53.720 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:03:53.726 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:03:53.728 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:03:53.734 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:03:53.736 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T01:03:53.743 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T01:03:53.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T01:03:53.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T01:03:53.792 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T01:03:53.803 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T01:03:53.804 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T01:03:53.832 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T01:03:53.839 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:03:53.882 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:03:53.931 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:03:53.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T01:03:53.986 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:03:53.992 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:03:54.043 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:03:54.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T01:03:54.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-22T01:03:54.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-04-22T01:03:54.195 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T01:03:54.208 INFO:teuthology.task.internal:Starting timer... 2024-04-22T01:03:54.208 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T01:03:54.217 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T01:03:54.226 DEBUG:teuthology.task.selinux:Excluding smithi070: OS 'ubuntu' does not support SELinux 2024-04-22T01:03:54.226 DEBUG:teuthology.task.selinux:Excluding smithi098: OS 'ubuntu' does not support SELinux 2024-04-22T01:03:54.227 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T01:03:54.227 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-22T01:03:54.227 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T01:03:54.227 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T01:03:54.236 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T01:03:54.255 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T01:03:54.256 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T01:10:51.283 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-04-22T01:10:51.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-22T01:10:51.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:10:51.368 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-22T01:10:51.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-22T01:10:51.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:10:51.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:10:51.527 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-22T01:10:51.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T01:10:51.602 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T01:10:51.613 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T01:10:51.613 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T01:10:51.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:10:51.616 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T01:10:51.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:10:51.642 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Command line: ntpd -gq 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: ---------------------------------------------------- 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: ntp-4 is maintained by Network Time Foundation, 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: corporation. Support and training for ntp-4 are 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: available at https://www.nwtime.org/support 2024-04-22T01:10:51.643 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: ---------------------------------------------------- 2024-04-22T01:10:51.644 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: proto: precision = 0.045 usec (-24) 2024-04-22T01:10:51.644 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: basedate set to 2022-02-04 2024-04-22T01:10:51.644 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: gps base set to 2022-02-06 (week 2196) 2024-04-22T01:10:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-22T01:10:51.644 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-22T01:10:51.644 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: restrict ::: KOD does nothing without LIMITED. 2024-04-22T01:10:51.645 INFO:teuthology.orchestra.run.smithi070.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listen and drop on 0 v6wildcard [::]:123 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listen normally on 2 lo 127.0.0.1:123 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listen normally on 3 enp3s0f1 172.21.15.70:123 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listen normally on 4 lo [::1]:123 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3e7%5]:123 2024-04-22T01:10:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:51 ntpd[17199]: Listening on routing socket on fd #22 for interface updates 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Command line: ntpd -gq 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: ---------------------------------------------------- 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: ntp-4 is maintained by Network Time Foundation, 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: corporation. Support and training for ntp-4 are 2024-04-22T01:10:51.670 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: available at https://www.nwtime.org/support 2024-04-22T01:10:51.671 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: ---------------------------------------------------- 2024-04-22T01:10:51.671 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: proto: precision = 0.060 usec (-24) 2024-04-22T01:10:51.671 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: basedate set to 2022-02-04 2024-04-22T01:10:51.671 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: gps base set to 2022-02-06 (week 2196) 2024-04-22T01:10:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-22T01:10:51.672 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-22T01:10:51.672 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: restrict ::: KOD does nothing without LIMITED. 2024-04-22T01:10:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-22T01:10:51.673 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listen and drop on 0 v6wildcard [::]:123 2024-04-22T01:10:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-22T01:10:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listen normally on 2 lo 127.0.0.1:123 2024-04-22T01:10:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listen normally on 3 enp3s0f1 172.21.15.98:123 2024-04-22T01:10:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listen normally on 4 lo [::1]:123 2024-04-22T01:10:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d26b%5]:123 2024-04-22T01:10:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:51 ntpd[17200]: Listening on routing socket on fd #22 for interface updates 2024-04-22T01:10:52.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:52 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:52.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:52 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:52.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:52 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:52.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:52 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:52.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:52 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:52.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:52 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:52 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:53.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:53 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:53.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:53 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:53.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:53 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:53.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:53 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:53.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:53 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:53.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:53 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:54 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:54.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:54 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:55.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:55 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:55.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:55 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:55.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:55 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:55.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:55 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:55.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:55 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:55.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:55 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:55.670 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:55 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:55.670 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:55 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:56 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:56 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:57 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:57 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:57 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:57 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:57 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:57 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:58.644 INFO:teuthology.orchestra.run.smithi070.stdout:22 Apr 01:10:58 ntpd[17199]: ntpd: time slew -0.002319 s 2024-04-22T01:10:58.645 INFO:teuthology.orchestra.run.smithi070.stdout:ntpd: time slew -0.002319s 2024-04-22T01:10:58.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-22T01:10:58.645 INFO:teuthology.orchestra.run.smithi070.stderr:22 Apr 01:10:58 ntpd[17199]: can't open /var/log/ntpstats/loopstats.20240422: Permission denied 2024-04-22T01:10:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stdout:22 Apr 01:10:58 ntpd[17200]: ntpd: time slew -0.000032 s 2024-04-22T01:10:58.672 INFO:teuthology.orchestra.run.smithi098.stdout:ntpd: time slew -0.000032s 2024-04-22T01:10:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-22T01:10:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:22 Apr 01:10:58 ntpd[17200]: can't open /var/log/ntpstats/loopstats.20240422: Permission denied 2024-04-22T01:10:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: remote refid st t when poll reach delay offset jitter 2024-04-22T01:10:58.703 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================== 2024-04-22T01:10:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.724 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-04-22T01:10:58.725 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-04-22T01:10:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.725 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:10:58.726 INFO:teuthology.run_tasks:Running task install... 2024-04-22T01:10:58.735 DEBUG:teuthology.task.install:project ceph 2024-04-22T01:10:58.735 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}} 2024-04-22T01:10:58.735 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'} 2024-04-22T01:10:58.735 INFO:teuthology.task.install:Using flavor: default 2024-04-22T01:10:58.741 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T01:10:58.741 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-22T01:10:58.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-key list | grep Ceph 2024-04-22T01:10:58.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-key list | grep Ceph 2024-04-22T01:10:58.805 INFO:teuthology.orchestra.run.smithi070.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-22T01:10:58.823 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-22T01:10:58.834 INFO:teuthology.orchestra.run.smithi070.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-22T01:10:58.835 INFO:teuthology.orchestra.run.smithi070.stdout:uid [ unknown] Ceph.com (release key) 2024-04-22T01:10:58.835 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-22T01:10:58.836 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-22T01:10:58.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:10:58.853 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-22T01:10:58.853 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph.com (release key) 2024-04-22T01:10:58.854 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-22T01:10:58.854 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-22T01:10:58.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:10:59.064 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default/ 2024-04-22T01:10:59.064 INFO:teuthology.task.install.deb:Package version is 19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:10:59.075 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default/ 2024-04-22T01:10:59.075 INFO:teuthology.task.install.deb:Package version is 19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:10:59.185 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:10:59.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-22T01:10:59.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-04-22T01:10:59.236 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:10:59.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-22T01:10:59.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get update 2024-04-22T01:10:59.376 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-22T01:10:59.424 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T01:10:59.425 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy InRelease 2024-04-22T01:10:59.439 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-22T01:10:59.446 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-22T01:10:59.459 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-22T01:10:59.468 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-22T01:10:59.489 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-22T01:10:59.496 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T01:10:59.517 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-04-22T01:10:59.531 INFO:teuthology.orchestra.run.smithi070.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy InRelease 2024-04-22T01:10:59.539 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-22T01:10:59.567 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-22T01:10:59.572 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-22T01:10:59.604 INFO:teuthology.orchestra.run.smithi070.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-22T01:10:59.641 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-22T01:10:59.653 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-22T01:10:59.686 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-04-22T01:10:59.795 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 28.9 kB in 0s (79.8 kB/s) 2024-04-22T01:10:59.796 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 28.9 kB in 0s (70.1 kB/s) 2024-04-22T01:11:00.747 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:11:00.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2405-g820a2f8c-1jammy cephadm=19.0.0-2405-g820a2f8c-1jammy ceph-mds=19.0.0-2405-g820a2f8c-1jammy ceph-mgr=19.0.0-2405-g820a2f8c-1jammy ceph-common=19.0.0-2405-g820a2f8c-1jammy ceph-fuse=19.0.0-2405-g820a2f8c-1jammy ceph-test=19.0.0-2405-g820a2f8c-1jammy ceph-volume=19.0.0-2405-g820a2f8c-1jammy radosgw=19.0.0-2405-g820a2f8c-1jammy python3-rados=19.0.0-2405-g820a2f8c-1jammy python3-rgw=19.0.0-2405-g820a2f8c-1jammy python3-cephfs=19.0.0-2405-g820a2f8c-1jammy python3-rbd=19.0.0-2405-g820a2f8c-1jammy libcephfs2=19.0.0-2405-g820a2f8c-1jammy libcephfs-dev=19.0.0-2405-g820a2f8c-1jammy librados2=19.0.0-2405-g820a2f8c-1jammy librbd1=19.0.0-2405-g820a2f8c-1jammy rbd-fuse=19.0.0-2405-g820a2f8c-1jammy python3-cephfs=19.0.0-2405-g820a2f8c-1jammy cephfs-shell=19.0.0-2405-g820a2f8c-1jammy cephfs-top=19.0.0-2405-g820a2f8c-1jammy cephfs-mirror=19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:11:00.785 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:11:00.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2405-g820a2f8c-1jammy cephadm=19.0.0-2405-g820a2f8c-1jammy ceph-mds=19.0.0-2405-g820a2f8c-1jammy ceph-mgr=19.0.0-2405-g820a2f8c-1jammy ceph-common=19.0.0-2405-g820a2f8c-1jammy ceph-fuse=19.0.0-2405-g820a2f8c-1jammy ceph-test=19.0.0-2405-g820a2f8c-1jammy ceph-volume=19.0.0-2405-g820a2f8c-1jammy radosgw=19.0.0-2405-g820a2f8c-1jammy python3-rados=19.0.0-2405-g820a2f8c-1jammy python3-rgw=19.0.0-2405-g820a2f8c-1jammy python3-cephfs=19.0.0-2405-g820a2f8c-1jammy python3-rbd=19.0.0-2405-g820a2f8c-1jammy libcephfs2=19.0.0-2405-g820a2f8c-1jammy libcephfs-dev=19.0.0-2405-g820a2f8c-1jammy librados2=19.0.0-2405-g820a2f8c-1jammy librbd1=19.0.0-2405-g820a2f8c-1jammy rbd-fuse=19.0.0-2405-g820a2f8c-1jammy python3-cephfs=19.0.0-2405-g820a2f8c-1jammy cephfs-shell=19.0.0-2405-g820a2f8c-1jammy cephfs-top=19.0.0-2405-g820a2f8c-1jammy cephfs-mirror=19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:11:00.814 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:11:00.861 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:11:00.988 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-22T01:11:00.989 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-22T01:11:01.069 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-04-22T01:11:01.070 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:11:01.160 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:11:01.160 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:11:01.160 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:11:01.160 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T01:11:01.160 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T01:11:01.160 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:11:01.162 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-04-22T01:11:01.162 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-22T01:11:01.162 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-22T01:11:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-22T01:11:01.164 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-04-22T01:11:01.165 INFO:teuthology.orchestra.run.smithi098.stdout: btrfs-tools 2024-04-22T01:11:01.261 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:11:01.261 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:11:01.262 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:11:01.262 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:11:01.263 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T01:11:01.263 INFO:teuthology.orchestra.run.smithi070.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T01:11:01.263 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:11:01.263 INFO:teuthology.orchestra.run.smithi070.stdout:The following additional packages will be installed: 2024-04-22T01:11:01.263 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-22T01:11:01.263 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-22T01:11:01.264 INFO:teuthology.orchestra.run.smithi070.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-22T01:11:01.265 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-22T01:11:01.266 INFO:teuthology.orchestra.run.smithi070.stdout:Suggested packages: 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout:Recommended packages: 2024-04-22T01:11:01.267 INFO:teuthology.orchestra.run.smithi070.stdout: btrfs-tools 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-22T01:11:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-22T01:11:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-22T01:11:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-22T01:11:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: socat unzip xmlstarlet zip 2024-04-22T01:11:01.345 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-22T01:11:01.345 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 180 MB of archives. 2024-04-22T01:11:01.345 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 823 MB of additional disk space will be used. 2024-04-22T01:11:01.345 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-22T01:11:01.366 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-22T01:11:01.369 INFO:teuthology.orchestra.run.smithi070.stdout:The following NEW packages will be installed: 2024-04-22T01:11:01.369 INFO:teuthology.orchestra.run.smithi070.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-22T01:11:01.369 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-22T01:11:01.370 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-22T01:11:01.370 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-22T01:11:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-22T01:11:01.372 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-22T01:11:01.373 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-22T01:11:01.373 INFO:teuthology.orchestra.run.smithi070.stdout: socat unzip xmlstarlet zip 2024-04-22T01:11:01.380 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-22T01:11:01.412 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2405-g820a2f8c-1jammy [3,541 kB] 2024-04-22T01:11:01.430 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-22T01:11:01.438 INFO:teuthology.orchestra.run.smithi070.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-22T01:11:01.438 INFO:teuthology.orchestra.run.smithi070.stdout:Need to get 180 MB of archives. 2024-04-22T01:11:01.438 INFO:teuthology.orchestra.run.smithi070.stdout:After this operation, 823 MB of additional disk space will be used. 2024-04-22T01:11:01.438 INFO:teuthology.orchestra.run.smithi070.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-22T01:11:01.455 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-22T01:11:01.459 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-22T01:11:01.459 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-22T01:11:01.471 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-22T01:11:01.472 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-22T01:11:01.521 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-22T01:11:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-22T01:11:01.534 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-22T01:11:01.546 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-22T01:11:01.548 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-22T01:11:01.549 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-22T01:11:01.550 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-22T01:11:01.550 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-22T01:11:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-22T01:11:01.560 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-22T01:11:01.562 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-22T01:11:01.563 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-22T01:11:01.563 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-22T01:11:01.570 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-22T01:11:01.574 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-22T01:11:01.577 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-22T01:11:01.579 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-22T01:11:01.579 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-22T01:11:01.595 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-22T01:11:01.596 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-22T01:11:01.597 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-22T01:11:01.597 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-22T01:11:01.598 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-22T01:11:01.598 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-22T01:11:01.610 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2405-g820a2f8c-1jammy [3,541 kB] 2024-04-22T01:11:01.617 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-22T01:11:01.617 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-22T01:11:01.618 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2405-g820a2f8c-1jammy [3,218 kB] 2024-04-22T01:11:01.620 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-22T01:11:01.620 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-22T01:11:01.626 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-22T01:11:01.630 INFO:teuthology.orchestra.run.smithi070.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-22T01:11:01.642 INFO:teuthology.orchestra.run.smithi070.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-22T01:11:01.647 INFO:teuthology.orchestra.run.smithi070.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-22T01:11:01.647 INFO:teuthology.orchestra.run.smithi070.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-22T01:11:01.648 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-22T01:11:01.648 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-22T01:11:01.649 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-22T01:11:01.650 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-22T01:11:01.650 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-22T01:11:01.652 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-22T01:11:01.656 INFO:teuthology.orchestra.run.smithi070.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-22T01:11:01.658 INFO:teuthology.orchestra.run.smithi070.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-22T01:11:01.660 INFO:teuthology.orchestra.run.smithi070.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-22T01:11:01.661 INFO:teuthology.orchestra.run.smithi070.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-22T01:11:01.662 INFO:teuthology.orchestra.run.smithi070.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-22T01:11:01.668 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-22T01:11:01.669 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-22T01:11:01.674 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-22T01:11:01.675 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-22T01:11:01.677 INFO:teuthology.orchestra.run.smithi070.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-22T01:11:01.679 INFO:teuthology.orchestra.run.smithi070.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-22T01:11:01.681 INFO:teuthology.orchestra.run.smithi070.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-22T01:11:01.681 INFO:teuthology.orchestra.run.smithi070.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-22T01:11:01.689 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-22T01:11:01.690 INFO:teuthology.orchestra.run.smithi098.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-22T01:11:01.691 INFO:teuthology.orchestra.run.smithi098.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-22T01:11:01.693 INFO:teuthology.orchestra.run.smithi098.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-22T01:11:01.694 INFO:teuthology.orchestra.run.smithi098.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-22T01:11:01.695 INFO:teuthology.orchestra.run.smithi098.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-22T01:11:01.697 INFO:teuthology.orchestra.run.smithi070.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-22T01:11:01.699 INFO:teuthology.orchestra.run.smithi070.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-22T01:11:01.699 INFO:teuthology.orchestra.run.smithi070.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-22T01:11:01.699 INFO:teuthology.orchestra.run.smithi070.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-22T01:11:01.699 INFO:teuthology.orchestra.run.smithi070.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-22T01:11:01.700 INFO:teuthology.orchestra.run.smithi070.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-22T01:11:01.710 INFO:teuthology.orchestra.run.smithi098.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-22T01:11:01.712 INFO:teuthology.orchestra.run.smithi098.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-22T01:11:01.713 INFO:teuthology.orchestra.run.smithi098.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-22T01:11:01.714 INFO:teuthology.orchestra.run.smithi098.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-22T01:11:01.718 INFO:teuthology.orchestra.run.smithi098.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2405-g820a2f8c-1jammy [711 kB] 2024-04-22T01:11:01.718 INFO:teuthology.orchestra.run.smithi070.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-22T01:11:01.719 INFO:teuthology.orchestra.run.smithi070.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-22T01:11:01.721 INFO:teuthology.orchestra.run.smithi070.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-22T01:11:01.722 INFO:teuthology.orchestra.run.smithi070.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-22T01:11:01.730 INFO:teuthology.orchestra.run.smithi098.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-22T01:11:01.744 INFO:teuthology.orchestra.run.smithi098.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2405-g820a2f8c-1jammy [356 kB] 2024-04-22T01:11:01.752 INFO:teuthology.orchestra.run.smithi070.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-22T01:11:01.753 INFO:teuthology.orchestra.run.smithi070.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-22T01:11:01.755 INFO:teuthology.orchestra.run.smithi070.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-22T01:11:01.755 INFO:teuthology.orchestra.run.smithi070.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-22T01:11:01.755 INFO:teuthology.orchestra.run.smithi070.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-22T01:11:01.756 INFO:teuthology.orchestra.run.smithi070.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-22T01:11:01.756 INFO:teuthology.orchestra.run.smithi098.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2405-g820a2f8c-1jammy [32.8 kB] 2024-04-22T01:11:01.757 INFO:teuthology.orchestra.run.smithi098.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2405-g820a2f8c-1jammy [180 kB] 2024-04-22T01:11:01.762 INFO:teuthology.orchestra.run.smithi098.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2405-g820a2f8c-1jammy [65.8 kB] 2024-04-22T01:11:01.765 INFO:teuthology.orchestra.run.smithi098.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2405-g820a2f8c-1jammy [330 kB] 2024-04-22T01:11:01.776 INFO:teuthology.orchestra.run.smithi098.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2405-g820a2f8c-1jammy [6,776 kB] 2024-04-22T01:11:01.791 INFO:teuthology.orchestra.run.smithi070.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-22T01:11:01.792 INFO:teuthology.orchestra.run.smithi098.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-22T01:11:01.792 INFO:teuthology.orchestra.run.smithi070.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-22T01:11:01.793 INFO:teuthology.orchestra.run.smithi098.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-22T01:11:01.794 INFO:teuthology.orchestra.run.smithi098.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-22T01:11:01.794 INFO:teuthology.orchestra.run.smithi098.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-22T01:11:01.801 INFO:teuthology.orchestra.run.smithi098.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-22T01:11:01.802 INFO:teuthology.orchestra.run.smithi070.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-22T01:11:01.802 INFO:teuthology.orchestra.run.smithi070.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-22T01:11:01.829 INFO:teuthology.orchestra.run.smithi098.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-22T01:11:01.832 INFO:teuthology.orchestra.run.smithi098.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-22T01:11:01.832 INFO:teuthology.orchestra.run.smithi098.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-22T01:11:01.840 INFO:teuthology.orchestra.run.smithi070.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-22T01:11:01.841 INFO:teuthology.orchestra.run.smithi070.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-22T01:11:01.843 INFO:teuthology.orchestra.run.smithi070.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-22T01:11:01.847 INFO:teuthology.orchestra.run.smithi070.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-22T01:11:01.849 INFO:teuthology.orchestra.run.smithi070.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi070.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-22T01:11:01.865 INFO:teuthology.orchestra.run.smithi098.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-22T01:11:01.865 INFO:teuthology.orchestra.run.smithi098.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-22T01:11:01.876 INFO:teuthology.orchestra.run.smithi070.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-22T01:11:01.880 INFO:teuthology.orchestra.run.smithi070.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-22T01:11:01.881 INFO:teuthology.orchestra.run.smithi070.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-22T01:11:01.882 INFO:teuthology.orchestra.run.smithi098.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-22T01:11:01.883 INFO:teuthology.orchestra.run.smithi098.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-22T01:11:01.883 INFO:teuthology.orchestra.run.smithi070.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-22T01:11:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-22T01:11:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-22T01:11:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-22T01:11:01.885 INFO:teuthology.orchestra.run.smithi098.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-22T01:11:01.889 INFO:teuthology.orchestra.run.smithi098.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-22T01:11:01.906 INFO:teuthology.orchestra.run.smithi070.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-22T01:11:01.920 INFO:teuthology.orchestra.run.smithi098.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-22T01:11:01.922 INFO:teuthology.orchestra.run.smithi098.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-22T01:11:01.923 INFO:teuthology.orchestra.run.smithi098.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-22T01:11:01.963 INFO:teuthology.orchestra.run.smithi098.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-22T01:11:01.968 INFO:teuthology.orchestra.run.smithi098.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-22T01:11:01.969 INFO:teuthology.orchestra.run.smithi098.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-22T01:11:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-22T01:11:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-22T01:11:01.980 INFO:teuthology.orchestra.run.smithi098.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-22T01:11:01.982 INFO:teuthology.orchestra.run.smithi098.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-22T01:11:01.998 INFO:teuthology.orchestra.run.smithi070.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-22T01:11:02.000 INFO:teuthology.orchestra.run.smithi070.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-22T01:11:02.003 INFO:teuthology.orchestra.run.smithi070.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-22T01:11:02.003 INFO:teuthology.orchestra.run.smithi070.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-22T01:11:02.005 INFO:teuthology.orchestra.run.smithi070.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-22T01:11:02.006 INFO:teuthology.orchestra.run.smithi098.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2405-g820a2f8c-1jammy [112 kB] 2024-04-22T01:11:02.009 INFO:teuthology.orchestra.run.smithi098.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2405-g820a2f8c-1jammy [466 kB] 2024-04-22T01:11:02.015 INFO:teuthology.orchestra.run.smithi098.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-22T01:11:02.016 INFO:teuthology.orchestra.run.smithi098.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-22T01:11:02.017 INFO:teuthology.orchestra.run.smithi098.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-22T01:11:02.023 INFO:teuthology.orchestra.run.smithi098.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2405-g820a2f8c-1jammy [25.6 MB] 2024-04-22T01:11:02.034 INFO:teuthology.orchestra.run.smithi070.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-22T01:11:02.037 INFO:teuthology.orchestra.run.smithi070.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-22T01:11:02.038 INFO:teuthology.orchestra.run.smithi070.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-22T01:11:02.059 INFO:teuthology.orchestra.run.smithi098.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-22T01:11:02.063 INFO:teuthology.orchestra.run.smithi098.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-22T01:11:02.064 INFO:teuthology.orchestra.run.smithi098.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-22T01:11:02.073 INFO:teuthology.orchestra.run.smithi070.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-22T01:11:02.073 INFO:teuthology.orchestra.run.smithi070.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-22T01:11:02.079 INFO:teuthology.orchestra.run.smithi098.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-22T01:11:02.080 INFO:teuthology.orchestra.run.smithi098.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-22T01:11:02.081 INFO:teuthology.orchestra.run.smithi098.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-22T01:11:02.085 INFO:teuthology.orchestra.run.smithi098.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-22T01:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-22T01:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-22T01:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-22T01:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-22T01:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-22T01:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-22T01:11:02.096 INFO:teuthology.orchestra.run.smithi070.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-22T01:11:02.097 INFO:teuthology.orchestra.run.smithi098.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-22T01:11:02.128 INFO:teuthology.orchestra.run.smithi070.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-22T01:11:02.130 INFO:teuthology.orchestra.run.smithi070.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-22T01:11:02.131 INFO:teuthology.orchestra.run.smithi070.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-22T01:11:02.152 INFO:teuthology.orchestra.run.smithi070.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-22T01:11:02.156 INFO:teuthology.orchestra.run.smithi070.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-22T01:11:02.156 INFO:teuthology.orchestra.run.smithi070.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-22T01:11:02.158 INFO:teuthology.orchestra.run.smithi070.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-22T01:11:02.158 INFO:teuthology.orchestra.run.smithi070.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-22T01:11:02.166 INFO:teuthology.orchestra.run.smithi070.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-22T01:11:02.166 INFO:teuthology.orchestra.run.smithi070.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-22T01:11:02.176 INFO:teuthology.orchestra.run.smithi070.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-22T01:11:02.176 INFO:teuthology.orchestra.run.smithi070.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-22T01:11:02.177 INFO:teuthology.orchestra.run.smithi070.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-22T01:11:02.198 INFO:teuthology.orchestra.run.smithi070.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-22T01:11:02.201 INFO:teuthology.orchestra.run.smithi070.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-22T01:11:02.202 INFO:teuthology.orchestra.run.smithi070.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-22T01:11:02.213 INFO:teuthology.orchestra.run.smithi070.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-22T01:11:02.214 INFO:teuthology.orchestra.run.smithi070.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-22T01:11:02.214 INFO:teuthology.orchestra.run.smithi070.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-22T01:11:02.217 INFO:teuthology.orchestra.run.smithi070.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-22T01:11:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-22T01:11:03.020 INFO:teuthology.orchestra.run.smithi098.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2405-g820a2f8c-1jammy [5,084 kB] 2024-04-22T01:11:03.111 INFO:teuthology.orchestra.run.smithi098.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2405-g820a2f8c-1jammy [244 kB] 2024-04-22T01:11:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2405-g820a2f8c-1jammy [124 kB] 2024-04-22T01:11:03.122 INFO:teuthology.orchestra.run.smithi098.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2405-g820a2f8c-1jammy [1,526 kB] 2024-04-22T01:11:03.177 INFO:teuthology.orchestra.run.smithi098.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2405-g820a2f8c-1jammy [6,246 kB] 2024-04-22T01:11:03.406 INFO:teuthology.orchestra.run.smithi098.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2405-g820a2f8c-1jammy [22.7 MB] 2024-04-22T01:11:03.477 INFO:teuthology.orchestra.run.smithi070.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2405-g820a2f8c-1jammy [3,218 kB] 2024-04-22T01:11:04.337 INFO:teuthology.orchestra.run.smithi070.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2405-g820a2f8c-1jammy [711 kB] 2024-04-22T01:11:04.346 INFO:teuthology.orchestra.run.smithi098.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2405-g820a2f8c-1jammy [14.1 kB] 2024-04-22T01:11:04.347 INFO:teuthology.orchestra.run.smithi098.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2405-g820a2f8c-1jammy [869 kB] 2024-04-22T01:11:04.364 INFO:teuthology.orchestra.run.smithi098.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2405-g820a2f8c-1jammy [2,435 kB] 2024-04-22T01:11:04.404 INFO:teuthology.orchestra.run.smithi098.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2405-g820a2f8c-1jammy [791 kB] 2024-04-22T01:11:04.414 INFO:teuthology.orchestra.run.smithi098.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2405-g820a2f8c-1jammy [147 kB] 2024-04-22T01:11:04.416 INFO:teuthology.orchestra.run.smithi098.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2405-g820a2f8c-1jammy [2,295 kB] 2024-04-22T01:11:04.483 INFO:teuthology.orchestra.run.smithi070.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2405-g820a2f8c-1jammy [356 kB] 2024-04-22T01:11:04.574 INFO:teuthology.orchestra.run.smithi098.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2405-g820a2f8c-1jammy [8,625 kB] 2024-04-22T01:11:04.606 INFO:teuthology.orchestra.run.smithi070.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2405-g820a2f8c-1jammy [32.8 kB] 2024-04-22T01:11:04.607 INFO:teuthology.orchestra.run.smithi070.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2405-g820a2f8c-1jammy [180 kB] 2024-04-22T01:11:04.611 INFO:teuthology.orchestra.run.smithi070.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2405-g820a2f8c-1jammy [65.8 kB] 2024-04-22T01:11:04.612 INFO:teuthology.orchestra.run.smithi070.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2405-g820a2f8c-1jammy [330 kB] 2024-04-22T01:11:04.632 INFO:teuthology.orchestra.run.smithi070.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2405-g820a2f8c-1jammy [6,776 kB] 2024-04-22T01:11:04.846 INFO:teuthology.orchestra.run.smithi098.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2405-g820a2f8c-1jammy [14.2 kB] 2024-04-22T01:11:04.847 INFO:teuthology.orchestra.run.smithi098.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2405-g820a2f8c-1jammy [50.7 MB] 2024-04-22T01:11:05.983 INFO:teuthology.orchestra.run.smithi070.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2405-g820a2f8c-1jammy [112 kB] 2024-04-22T01:11:06.005 INFO:teuthology.orchestra.run.smithi070.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2405-g820a2f8c-1jammy [466 kB] 2024-04-22T01:11:06.092 INFO:teuthology.orchestra.run.smithi070.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2405-g820a2f8c-1jammy [25.6 MB] 2024-04-22T01:11:07.114 INFO:teuthology.orchestra.run.smithi098.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2405-g820a2f8c-1jammy [121 kB] 2024-04-22T01:11:07.116 INFO:teuthology.orchestra.run.smithi098.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2405-g820a2f8c-1jammy [229 kB] 2024-04-22T01:11:07.121 INFO:teuthology.orchestra.run.smithi098.stdout:Get:120 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2405-g820a2f8c-1jammy [34.0 kB] 2024-04-22T01:11:07.122 INFO:teuthology.orchestra.run.smithi098.stdout:Get:121 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2405-g820a2f8c-1jammy [23.4 kB] 2024-04-22T01:11:07.122 INFO:teuthology.orchestra.run.smithi098.stdout:Get:122 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2405-g820a2f8c-1jammy [39.0 kB] 2024-04-22T01:11:07.124 INFO:teuthology.orchestra.run.smithi098.stdout:Get:123 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2405-g820a2f8c-1jammy [13.4 MB] 2024-04-22T01:11:07.528 INFO:teuthology.orchestra.run.smithi098.stdout:Get:124 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2405-g820a2f8c-1jammy [91.4 kB] 2024-04-22T01:11:08.137 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 180 MB in 6s (28.8 MB/s) 2024-04-22T01:11:08.277 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-22T01:11:08.365 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-22T01:11:08.370 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-22T01:11:08.401 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-22T01:11:08.729 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-22T01:11:08.755 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-22T01:11:08.796 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-22T01:11:08.874 INFO:teuthology.orchestra.run.smithi070.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2405-g820a2f8c-1jammy [5,084 kB] 2024-04-22T01:11:09.116 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cmd2. 2024-04-22T01:11:09.141 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-22T01:11:09.182 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-22T01:11:09.222 INFO:teuthology.orchestra.run.smithi070.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2405-g820a2f8c-1jammy [244 kB] 2024-04-22T01:11:09.235 INFO:teuthology.orchestra.run.smithi070.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2405-g820a2f8c-1jammy [124 kB] 2024-04-22T01:11:09.238 INFO:teuthology.orchestra.run.smithi070.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2405-g820a2f8c-1jammy [1,526 kB] 2024-04-22T01:11:09.364 INFO:teuthology.orchestra.run.smithi070.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2405-g820a2f8c-1jammy [6,246 kB] 2024-04-22T01:11:09.527 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-22T01:11:09.553 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-22T01:11:09.635 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-22T01:11:09.832 INFO:teuthology.orchestra.run.smithi070.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2405-g820a2f8c-1jammy [22.7 MB] 2024-04-22T01:11:09.958 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-22T01:11:09.984 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-22T01:11:10.024 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-22T01:11:10.360 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-22T01:11:10.386 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-22T01:11:10.427 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-22T01:11:10.839 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-22T01:11:10.865 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T01:11:10.914 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:11:11.301 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-22T01:11:11.327 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T01:11:11.367 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:11:11.627 INFO:teuthology.orchestra.run.smithi070.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2405-g820a2f8c-1jammy [14.1 kB] 2024-04-22T01:11:11.627 INFO:teuthology.orchestra.run.smithi070.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2405-g820a2f8c-1jammy [869 kB] 2024-04-22T01:11:11.644 INFO:teuthology.orchestra.run.smithi070.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2405-g820a2f8c-1jammy [2,435 kB] 2024-04-22T01:11:11.704 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-22T01:11:11.730 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T01:11:11.770 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:11:11.812 INFO:teuthology.orchestra.run.smithi070.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2405-g820a2f8c-1jammy [791 kB] 2024-04-22T01:11:11.879 INFO:teuthology.orchestra.run.smithi070.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2405-g820a2f8c-1jammy [147 kB] 2024-04-22T01:11:11.883 INFO:teuthology.orchestra.run.smithi070.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2405-g820a2f8c-1jammy [2,295 kB] 2024-04-22T01:11:12.060 INFO:teuthology.orchestra.run.smithi070.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2405-g820a2f8c-1jammy [8,625 kB] 2024-04-22T01:11:12.082 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-22T01:11:12.108 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-22T01:11:12.140 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-22T01:11:12.451 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librados2. 2024-04-22T01:11:12.478 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../010-librados2_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:12.518 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librados2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:12.711 INFO:teuthology.orchestra.run.smithi070.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2405-g820a2f8c-1jammy [14.2 kB] 2024-04-22T01:11:12.712 INFO:teuthology.orchestra.run.smithi070.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2405-g820a2f8c-1jammy [50.7 MB] 2024-04-22T01:11:12.971 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librbd1. 2024-04-22T01:11:12.997 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../011-librbd1_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librbd1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:13.482 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs2. 2024-04-22T01:11:13.508 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:13.540 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:13.835 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rados. 2024-04-22T01:11:13.850 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:13.893 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rados (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-22T01:11:14.222 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:14.262 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-argparse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:14.515 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cephfs. 2024-04-22T01:11:14.530 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:14.641 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cephfs (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:14.968 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-22T01:11:14.995 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:15.035 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:15.346 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-prettytable. 2024-04-22T01:11:15.361 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-22T01:11:15.404 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-22T01:11:15.657 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rbd. 2024-04-22T01:11:15.672 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:15.707 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rbd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:16.044 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-22T01:11:16.070 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-22T01:11:16.110 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-22T01:11:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-22T01:11:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-22T01:11:16.429 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-22T01:11:16.732 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-22T01:11:16.748 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-22T01:11:16.782 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-22T01:11:16.903 INFO:teuthology.orchestra.run.smithi070.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2405-g820a2f8c-1jammy [121 kB] 2024-04-22T01:11:16.905 INFO:teuthology.orchestra.run.smithi070.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2405-g820a2f8c-1jammy [229 kB] 2024-04-22T01:11:16.910 INFO:teuthology.orchestra.run.smithi070.stdout:Get:120 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2405-g820a2f8c-1jammy [34.0 kB] 2024-04-22T01:11:16.911 INFO:teuthology.orchestra.run.smithi070.stdout:Get:121 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2405-g820a2f8c-1jammy [23.4 kB] 2024-04-22T01:11:16.911 INFO:teuthology.orchestra.run.smithi070.stdout:Get:122 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2405-g820a2f8c-1jammy [39.0 kB] 2024-04-22T01:11:16.912 INFO:teuthology.orchestra.run.smithi070.stdout:Get:123 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2405-g820a2f8c-1jammy [13.4 MB] 2024-04-22T01:11:17.103 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua5.1. 2024-04-22T01:11:17.129 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-22T01:11:17.169 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-22T01:11:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-any. 2024-04-22T01:11:17.512 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-22T01:11:17.555 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-22T01:11:17.658 INFO:teuthology.orchestra.run.smithi070.stdout:Get:124 https://1.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2405-g820a2f8c-1jammy [91.4 kB] 2024-04-22T01:11:17.800 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package zip. 2024-04-22T01:11:17.815 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-22T01:11:17.850 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking zip (3.0-12build2) ... 2024-04-22T01:11:18.128 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package unzip. 2024-04-22T01:11:18.154 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-22T01:11:18.195 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-22T01:11:18.275 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 180 MB in 16s (11.1 MB/s) 2024-04-22T01:11:18.394 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-22T01:11:18.492 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-22T01:11:18.497 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-22T01:11:18.542 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-22T01:11:18.582 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package luarocks. 2024-04-22T01:11:18.608 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-22T01:11:18.648 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-22T01:11:18.844 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-22T01:11:18.871 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-22T01:11:18.911 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-22T01:11:19.023 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librgw2. 2024-04-22T01:11:19.050 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../027-librgw2_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:19.090 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librgw2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:19.189 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cmd2. 2024-04-22T01:11:19.215 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-22T01:11:19.247 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-22T01:11:19.534 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-22T01:11:19.559 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-22T01:11:19.633 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-22T01:11:19.651 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rgw. 2024-04-22T01:11:19.666 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:19.743 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:20.030 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-22T01:11:20.056 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-22T01:11:20.079 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-22T01:11:20.105 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-22T01:11:20.145 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-22T01:11:20.270 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-22T01:11:20.440 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libradosstriper1. 2024-04-22T01:11:20.467 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:20.507 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libradosstriper1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:20.623 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-22T01:11:20.650 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-22T01:11:20.690 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-22T01:11:20.902 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-common. 2024-04-22T01:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:21.126 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-22T01:11:21.153 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T01:11:21.202 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:11:21.588 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-22T01:11:21.614 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T01:11:21.663 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:11:21.999 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-22T01:11:22.025 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T01:11:22.065 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:11:22.299 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-base. 2024-04-22T01:11:22.325 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:22.369 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-22T01:11:22.375 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-base (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:22.395 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-22T01:11:22.427 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-22T01:11:22.746 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librados2. 2024-04-22T01:11:22.772 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../010-librados2_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:22.821 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librados2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:22.946 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-22T01:11:23.582 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librbd1. 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../011-librbd1_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librbd1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cheroot. 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-22T01:11:23.583 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-22T01:11:23.687 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-22T01:11:23.712 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-22T01:11:23.751 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-22T01:11:23.985 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libcephfs2. 2024-04-22T01:11:24.000 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:24.021 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-22T01:11:24.036 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-22T01:11:24.060 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libcephfs2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:24.071 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-22T01:11:24.313 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rados. 2024-04-22T01:11:24.339 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:24.366 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-22T01:11:24.373 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rados (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:24.392 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-22T01:11:24.433 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-22T01:11:24.682 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-22T01:11:24.708 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:24.744 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempora. 2024-04-22T01:11:24.748 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-ceph-argparse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:24.770 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-22T01:11:24.810 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-22T01:11:25.001 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cephfs. 2024-04-22T01:11:25.027 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:25.063 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-portend. 2024-04-22T01:11:25.067 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cephfs (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-22T01:11:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-22T01:11:25.378 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-22T01:11:25.391 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-22T01:11:25.404 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:25.417 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-22T01:11:25.445 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:25.457 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-22T01:11:25.748 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-prettytable. 2024-04-22T01:11:25.752 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-22T01:11:25.774 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-22T01:11:25.779 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-22T01:11:25.814 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-22T01:11:25.819 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-22T01:11:26.025 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rbd. 2024-04-22T01:11:26.051 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:26.083 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rbd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:26.164 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-natsort. 2024-04-22T01:11:26.191 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-22T01:11:26.230 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-22T01:11:26.428 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-22T01:11:26.454 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-22T01:11:26.492 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-logutils. 2024-04-22T01:11:26.495 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-22T01:11:26.519 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-22T01:11:26.559 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-22T01:11:26.781 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-22T01:11:26.807 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-22T01:11:26.847 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-22T01:11:26.853 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-22T01:11:26.880 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-22T01:11:26.920 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-22T01:11:27.150 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-22T01:11:27.165 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-22T01:11:27.173 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-mako. 2024-04-22T01:11:27.188 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-22T01:11:27.200 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-22T01:11:27.223 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-22T01:11:27.470 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua5.1. 2024-04-22T01:11:27.496 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-22T01:11:27.535 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-22T01:11:27.536 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-22T01:11:27.561 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-22T01:11:27.601 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-22T01:11:27.868 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua-any. 2024-04-22T01:11:27.894 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-22T01:11:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-22T01:11:27.930 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-22T01:11:27.939 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-22T01:11:27.970 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-22T01:11:28.183 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package zip. 2024-04-22T01:11:28.199 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-22T01:11:28.215 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webob. 2024-04-22T01:11:28.241 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-22T01:11:28.242 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking zip (3.0-12build2) ... 2024-04-22T01:11:28.273 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-22T01:11:28.593 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-waitress. 2024-04-22T01:11:28.620 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-22T01:11:28.620 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package unzip. 2024-04-22T01:11:28.646 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-22T01:11:28.663 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-22T01:11:28.678 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-22T01:11:28.971 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempita. 2024-04-22T01:11:28.997 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-22T01:11:29.038 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-22T01:11:29.040 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package luarocks. 2024-04-22T01:11:29.066 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-22T01:11:29.107 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-22T01:11:29.341 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-paste. 2024-04-22T01:11:29.367 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-22T01:11:29.407 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-22T01:11:29.498 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librgw2. 2024-04-22T01:11:29.524 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../027-librgw2_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:29.564 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librgw2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:29.727 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-22T01:11:29.743 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-22T01:11:29.794 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-22T01:11:30.081 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-22T01:11:30.107 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-22T01:11:30.108 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rgw. 2024-04-22T01:11:30.135 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:30.147 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-22T01:11:30.175 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:30.453 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-22T01:11:30.475 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webtest. 2024-04-22T01:11:30.479 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-22T01:11:30.501 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-22T01:11:30.511 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-22T01:11:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-22T01:11:30.814 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libradosstriper1. 2024-04-22T01:11:30.840 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:30.861 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pecan. 2024-04-22T01:11:30.880 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libradosstriper1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:30.888 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-22T01:11:30.928 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-22T01:11:31.291 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-certifi. 2024-04-22T01:11:31.300 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-common. 2024-04-22T01:11:31.318 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-22T01:11:31.327 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-22T01:11:31.367 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:31.644 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-idna. 2024-04-22T01:11:31.659 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-22T01:11:31.694 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-22T01:11:31.997 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-urllib3. 2024-04-22T01:11:32.023 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-22T01:11:32.064 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-22T01:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests. 2024-04-22T01:11:32.410 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-22T01:11:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-22T01:11:32.672 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-base. 2024-04-22T01:11:32.699 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:32.748 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-base (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:32.762 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-22T01:11:32.788 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-22T01:11:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-22T01:11:33.124 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-22T01:11:33.138 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:33.181 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:33.334 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-22T01:11:33.361 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-22T01:11:33.401 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-22T01:11:33.538 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-22T01:11:33.563 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:33.595 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:33.662 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cheroot. 2024-04-22T01:11:33.689 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-22T01:11:33.728 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-22T01:11:33.856 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-22T01:11:33.872 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-22T01:11:33.906 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-22T01:11:34.049 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-22T01:11:34.075 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-22T01:11:34.115 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-22T01:11:34.192 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-openssl. 2024-04-22T01:11:34.218 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-22T01:11:34.259 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-22T01:11:34.418 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-22T01:11:34.445 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-22T01:11:34.484 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-22T01:11:35.288 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr. 2024-04-22T01:11:35.315 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:35.471 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-22T01:11:35.497 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-22T01:11:35.504 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-22T01:11:35.823 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-tempora. 2024-04-22T01:11:35.841 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mon. 2024-04-22T01:11:35.850 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-22T01:11:35.856 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:35.890 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-22T01:11:35.891 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mon (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:36.193 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-portend. 2024-04-22T01:11:36.219 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-22T01:11:36.259 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-22T01:11:36.486 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-osd. 2024-04-22T01:11:36.513 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-osd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:36.562 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-22T01:11:36.588 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-22T01:11:36.628 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-22T01:11:36.890 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-22T01:11:36.905 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-22T01:11:36.940 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-22T01:11:37.285 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-natsort. 2024-04-22T01:11:37.311 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-22T01:11:37.351 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-22T01:11:37.598 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph. 2024-04-22T01:11:37.621 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-logutils. 2024-04-22T01:11:37.624 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../068-ceph_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:37.647 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-22T01:11:37.664 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:37.679 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-22T01:11:37.942 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-fuse. 2024-04-22T01:11:37.949 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-22T01:11:37.968 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:37.975 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-22T01:11:38.000 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:38.015 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-22T01:11:38.318 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-mako. 2024-04-22T01:11:38.345 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-22T01:11:38.345 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mds. 2024-04-22T01:11:38.372 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:38.384 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-22T01:11:38.412 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mds (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:38.823 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephadm. 2024-04-22T01:11:38.839 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../071-cephadm_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:38.879 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-22T01:11:38.881 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:38.906 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-22T01:11:38.946 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-22T01:11:39.176 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-22T01:11:39.202 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-22T01:11:39.232 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-22T01:11:39.242 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-22T01:11:39.258 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-22T01:11:39.299 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-22T01:11:39.545 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-22T01:11:39.568 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-webob. 2024-04-22T01:11:39.572 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-22T01:11:39.583 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-22T01:11:39.612 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-22T01:11:39.626 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-22T01:11:39.946 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-waitress. 2024-04-22T01:11:39.972 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-22T01:11:40.015 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-22T01:11:40.332 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-tempita. 2024-04-22T01:11:40.359 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-22T01:11:40.399 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-22T01:11:40.685 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-paste. 2024-04-22T01:11:40.711 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-22T01:11:40.751 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-22T01:11:41.051 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-babel. 2024-04-22T01:11:41.077 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-22T01:11:41.079 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-22T01:11:41.106 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-22T01:11:41.116 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-22T01:11:41.148 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-22T01:11:41.425 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-22T01:11:41.437 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jinja2. 2024-04-22T01:11:41.441 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-22T01:11:41.463 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-22T01:11:41.475 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-22T01:11:41.503 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-22T01:11:41.720 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-webtest. 2024-04-22T01:11:41.746 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-22T01:11:41.786 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-22T01:11:41.865 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-22T01:11:41.891 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:41.931 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:42.089 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pecan. 2024-04-22T01:11:42.116 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-22T01:11:42.156 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-22T01:11:42.251 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-22T01:11:42.278 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-22T01:11:42.318 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-22T01:11:42.527 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-certifi. 2024-04-22T01:11:42.554 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-22T01:11:42.594 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-22T01:11:42.629 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-routes. 2024-04-22T01:11:42.656 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-22T01:11:42.696 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-22T01:11:42.913 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-idna. 2024-04-22T01:11:42.940 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-22T01:11:42.980 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-22T01:11:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-22T01:11:43.042 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:43.283 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-urllib3. 2024-04-22T01:11:43.310 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-22T01:11:43.349 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-22T01:11:43.644 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-requests. 2024-04-22T01:11:43.659 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-22T01:11:43.694 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-22T01:11:43.947 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-22T01:11:43.968 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-22T01:11:43.997 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-22T01:11:44.275 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-22T01:11:44.291 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:44.326 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:44.329 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-22T01:11:44.345 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-22T01:11:44.379 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-22T01:11:44.679 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-22T01:11:44.706 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:44.746 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:44.816 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-joblib. 2024-04-22T01:11:44.832 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-22T01:11:44.866 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-22T01:11:45.049 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-22T01:11:45.075 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-22T01:11:45.115 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-22T01:11:45.228 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-22T01:11:45.255 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-22T01:11:45.359 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-openssl. 2024-04-22T01:11:45.375 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-22T01:11:45.380 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-22T01:11:45.418 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-22T01:11:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn. 2024-04-22T01:11:45.668 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-22T01:11:45.708 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-22T01:11:45.746 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr. 2024-04-22T01:11:45.772 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:45.812 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:46.207 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mon. 2024-04-22T01:11:46.235 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:46.274 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mon (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:46.384 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-22T01:11:46.411 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:46.450 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:46.877 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-osd. 2024-04-22T01:11:46.904 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:46.943 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-osd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:47.414 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cachetools. 2024-04-22T01:11:47.441 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-22T01:11:47.481 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-22T01:11:47.734 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-22T01:11:47.761 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-22T01:11:47.800 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-22T01:11:48.013 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph. 2024-04-22T01:11:48.029 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../068-ceph_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:48.145 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-22T01:11:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-22T01:11:48.212 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-22T01:11:48.391 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-fuse. 2024-04-22T01:11:48.417 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:48.457 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:48.540 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rsa. 2024-04-22T01:11:48.567 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-22T01:11:48.606 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-22T01:11:48.827 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mds. 2024-04-22T01:11:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:48.927 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-google-auth. 2024-04-22T01:11:48.954 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-22T01:11:48.993 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-22T01:11:49.027 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mds (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:49.314 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-22T01:11:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-22T01:11:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-22T01:11:49.414 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package cephadm. 2024-04-22T01:11:49.439 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../071-cephadm_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:49.479 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:49.692 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-websocket. 2024-04-22T01:11:49.719 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-22T01:11:49.758 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-22T01:11:49.765 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-22T01:11:49.781 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-22T01:11:49.815 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-22T01:11:50.070 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-22T01:11:50.097 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-22T01:11:50.126 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-22T01:11:50.136 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-22T01:11:50.153 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-22T01:11:50.193 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-22T01:11:50.759 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-22T01:11:50.775 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:50.817 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:51.223 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-babel. 2024-04-22T01:11:51.250 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-22T01:11:51.287 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-22T01:11:51.297 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-22T01:11:51.314 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-22T01:11:51.344 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-22T01:11:51.584 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jinja2. 2024-04-22T01:11:51.611 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-22T01:11:51.642 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-22T01:11:51.690 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-22T01:11:51.717 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-22T01:11:51.756 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-22T01:11:51.936 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-22T01:11:51.963 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:51.995 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:52.017 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package jq. 2024-04-22T01:11:52.033 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-22T01:11:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-22T01:11:52.314 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-22T01:11:52.341 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-22T01:11:52.345 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package socat. 2024-04-22T01:11:52.360 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-22T01:11:52.389 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-22T01:11:52.395 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-22T01:11:52.634 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-routes. 2024-04-22T01:11:52.660 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-22T01:11:52.665 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package xmlstarlet. 2024-04-22T01:11:52.680 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-22T01:11:52.700 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-22T01:11:52.714 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-22T01:11:52.970 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-22T01:11:52.985 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:53.020 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:53.069 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-test. 2024-04-22T01:11:53.085 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:53.119 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-test (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:54.234 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-22T01:11:54.299 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-22T01:11:54.300 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-22T01:11:54.787 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-joblib. 2024-04-22T01:11:54.814 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-22T01:11:54.853 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-22T01:11:55.223 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-22T01:11:55.250 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-22T01:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-volume. 2024-04-22T01:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:55.375 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-22T01:11:55.398 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-volume (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:55.678 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-sklearn. 2024-04-22T01:11:55.705 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-22T01:11:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-22T01:11:55.735 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:55.736 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-22T01:11:55.769 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephfs-mirror (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:56.260 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-colorama. 2024-04-22T01:11:56.287 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-22T01:11:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-22T01:11:56.370 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-22T01:11:56.398 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:56.436 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:56.608 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephfs-shell. 2024-04-22T01:11:56.635 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:56.674 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephfs-shell (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:56.944 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephfs-top. 2024-04-22T01:11:56.959 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:11:57.002 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephfs-top (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:57.246 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-22T01:11:57.262 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs-dev (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:11:57.340 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cachetools. 2024-04-22T01:11:57.367 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-22T01:11:57.398 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-22T01:11:57.607 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-22T01:11:57.634 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-22T01:11:57.674 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-22T01:11:57.684 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-22T01:11:57.712 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-22T01:11:57.750 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-22T01:11:58.003 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-22T01:11:58.031 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-22T01:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-22T01:11:58.070 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-22T01:11:58.098 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-22T01:11:58.137 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-22T01:11:58.416 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package nvme-cli. 2024-04-22T01:11:58.443 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-22T01:11:58.457 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rsa. 2024-04-22T01:11:58.482 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-22T01:11:58.484 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-22T01:11:58.523 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-22T01:11:58.869 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-google-auth. 2024-04-22T01:11:58.879 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package pkg-config. 2024-04-22T01:11:58.896 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-22T01:11:58.906 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-22T01:11:58.937 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-22T01:11:58.951 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-22T01:11:59.182 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-22T01:11:59.209 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-22T01:11:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-22T01:11:59.248 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-22T01:11:59.265 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-22T01:11:59.296 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-22T01:11:59.600 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-websocket. 2024-04-22T01:11:59.626 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-22T01:11:59.627 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-22T01:11:59.654 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-22T01:11:59.666 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-22T01:11:59.693 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-22T01:11:59.978 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-22T01:11:59.996 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastescript. 2024-04-22T01:12:00.006 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-22T01:12:00.023 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-22T01:12:00.044 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-22T01:12:00.062 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-22T01:12:00.343 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pluggy. 2024-04-22T01:12:00.368 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-22T01:12:00.407 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-22T01:12:00.708 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-22T01:12:00.719 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-psutil. 2024-04-22T01:12:00.735 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:12:00.746 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-22T01:12:00.774 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:00.785 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-22T01:12:01.094 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-22T01:12:01.121 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-22T01:12:01.121 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-py. 2024-04-22T01:12:01.149 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-22T01:12:01.160 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-22T01:12:01.188 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-22T01:12:01.463 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-22T01:12:01.490 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-22T01:12:01.500 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pygments. 2024-04-22T01:12:01.516 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-22T01:12:01.529 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-22T01:12:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-22T01:12:01.774 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package jq. 2024-04-22T01:12:01.801 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-22T01:12:01.840 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-22T01:12:01.995 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-22T01:12:02.023 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-22T01:12:02.062 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-22T01:12:02.093 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package socat. 2024-04-22T01:12:02.120 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-22T01:12:02.159 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-22T01:12:02.348 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-toml. 2024-04-22T01:12:02.376 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-22T01:12:02.406 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package xmlstarlet. 2024-04-22T01:12:02.478 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-22T01:12:02.521 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-22T01:12:02.693 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pytest. 2024-04-22T01:12:02.720 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-22T01:12:02.759 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-22T01:12:02.850 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-test. 2024-04-22T01:12:02.866 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:02.900 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-test (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:03.071 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplejson. 2024-04-22T01:12:03.099 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-22T01:12:03.137 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-22T01:12:03.398 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-22T01:12:03.414 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-22T01:12:03.448 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-22T01:12:04.186 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package radosgw. 2024-04-22T01:12:04.214 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../122-radosgw_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:04.252 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking radosgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:05.021 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package rbd-fuse. 2024-04-22T01:12:05.035 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:05.072 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking rbd-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:05.112 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-volume. 2024-04-22T01:12:05.128 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:12:05.162 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-volume (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:05.385 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-22T01:12:05.491 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-22T01:12:05.519 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:05.549 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking cephfs-mirror (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:05.644 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-22T01:12:05.770 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-22T01:12:05.827 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-colorama. 2024-04-22T01:12:05.855 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-22T01:12:05.894 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-22T01:12:06.033 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-22T01:12:06.154 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package cephfs-shell. 2024-04-22T01:12:06.170 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:12:06.213 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking cephfs-shell (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:06.404 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-22T01:12:06.499 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package cephfs-top. 2024-04-22T01:12:06.514 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2405-g820a2f8c-1jammy_all.deb ... 2024-04-22T01:12:06.557 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking cephfs-top (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:06.793 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-22T01:12:06.820 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:06.833 INFO:teuthology.orchestra.run.smithi098.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-22T01:12:06.851 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libcephfs-dev (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:06.905 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:07.120 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-22T01:12:07.147 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-22T01:12:07.186 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-22T01:12:07.455 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user cephadm....done 2024-04-22T01:12:07.508 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-22T01:12:07.535 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-22T01:12:07.574 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-22T01:12:07.578 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-22T01:12:07.846 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-22T01:12:07.937 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package nvme-cli. 2024-04-22T01:12:07.964 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-22T01:12:08.003 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-22T01:12:08.106 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-22T01:12:08.232 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-22T01:12:08.425 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package pkg-config. 2024-04-22T01:12:08.453 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-22T01:12:08.491 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-22T01:12:08.499 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-22T01:12:08.744 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-22T01:12:08.767 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-22T01:12:08.771 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-22T01:12:08.810 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-22T01:12:08.868 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-22T01:12:09.154 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-22T01:12:09.205 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-22T01:12:09.232 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-22T01:12:09.271 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-22T01:12:09.514 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-22T01:12:09.541 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pastescript. 2024-04-22T01:12:09.557 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-22T01:12:09.591 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-22T01:12:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-22T01:12:09.911 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pluggy. 2024-04-22T01:12:09.939 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-22T01:12:09.977 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-22T01:12:10.075 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-22T01:12:10.312 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-psutil. 2024-04-22T01:12:10.333 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-22T01:12:10.346 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-22T01:12:10.372 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-22T01:12:10.611 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-22T01:12:10.708 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-py. 2024-04-22T01:12:10.735 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-22T01:12:10.774 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-22T01:12:10.862 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-22T01:12:11.094 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pygments. 2024-04-22T01:12:11.122 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-22T01:12:11.122 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-22T01:12:11.163 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-22T01:12:11.523 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-argparse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:11.615 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-22T01:12:11.642 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-22T01:12:11.681 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-22T01:12:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-22T01:12:11.909 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-22T01:12:11.942 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-toml. 2024-04-22T01:12:11.969 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-22T01:12:12.000 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-22T01:12:12.035 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-22T01:12:12.270 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pytest. 2024-04-22T01:12:12.285 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-22T01:12:12.297 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-22T01:12:12.336 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-22T01:12:12.412 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-22T01:12:12.663 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-22T01:12:12.681 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-simplejson. 2024-04-22T01:12:12.709 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-22T01:12:12.748 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-22T01:12:13.034 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-22T01:12:13.049 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-22T01:12:13.092 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-22T01:12:13.095 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-22T01:12:13.127 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-22T01:12:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-22T01:12:13.317 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-22T01:12:13.677 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-22T01:12:13.738 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package radosgw. 2024-04-22T01:12:13.765 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../122-radosgw_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:13.803 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-22T01:12:13.804 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking radosgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:14.088 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-22T01:12:14.355 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-22T01:12:14.615 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package rbd-fuse. 2024-04-22T01:12:14.631 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2405-g820a2f8c-1jammy_amd64.deb ... 2024-04-22T01:12:14.665 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up zip (3.0-12build2) ... 2024-04-22T01:12:14.673 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking rbd-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:14.766 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-22T01:12:14.978 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-22T01:12:15.237 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-22T01:12:15.363 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-22T01:12:15.451 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-22T01:12:15.623 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-22T01:12:15.711 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-22T01:12:15.837 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-22T01:12:15.963 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-22T01:12:15.966 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-22T01:12:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-22T01:12:16.413 INFO:teuthology.orchestra.run.smithi070.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-22T01:12:16.508 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:16.674 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-22T01:12:16.917 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-22T01:12:17.285 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-22T01:12:17.372 INFO:teuthology.orchestra.run.smithi070.stdout:Adding system user cephadm....done 2024-04-22T01:12:17.569 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-22T01:12:17.603 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-22T01:12:17.879 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-22T01:12:17.940 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-22T01:12:18.174 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-22T01:12:18.222 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-22T01:12:18.301 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-22T01:12:18.498 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-22T01:12:18.551 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-22T01:12:18.608 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:18.811 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-22T01:12:18.909 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-idna (3.3-1) ... 2024-04-22T01:12:18.937 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-22T01:12:19.218 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-22T01:12:19.238 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-22T01:12:19.590 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-22T01:12:19.842 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-22T01:12:20.076 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-22T01:12:20.341 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-22T01:12:20.469 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-22T01:12:20.593 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:12:20.719 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-22T01:12:20.912 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-22T01:12:21.012 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-22T01:12:21.138 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-22T01:12:21.163 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-22T01:12:21.264 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-22T01:12:21.408 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-22T01:12:21.557 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-22T01:12:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-ceph-argparse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:21.842 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-22T01:12:22.151 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-22T01:12:22.277 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-22T01:12:22.376 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:12:22.403 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-22T01:12:22.485 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-22T01:12:22.671 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-22T01:12:22.787 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-22T01:12:22.788 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-22T01:12:23.039 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-22T01:12:23.046 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-22T01:12:23.306 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-22T01:12:23.445 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-22T01:12:23.478 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-22T01:12:23.543 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-22T01:12:23.565 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-22T01:12:23.669 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-22T01:12:23.800 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-22T01:12:24.028 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-22T01:12:24.059 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-22T01:12:24.154 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-22T01:12:24.185 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-22T01:12:24.413 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-22T01:12:24.462 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-22T01:12:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-22T01:12:24.681 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-22T01:12:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-22T01:12:24.974 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up zip (3.0-12build2) ... 2024-04-22T01:12:25.100 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-22T01:12:25.190 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-22T01:12:25.441 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-22T01:12:25.559 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-22T01:12:25.685 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-22T01:12:25.760 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-22T01:12:25.774 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-22T01:12:26.011 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-22T01:12:26.019 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-22T01:12:26.137 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-22T01:12:26.145 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-22T01:12:26.271 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-22T01:12:26.522 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-22T01:12:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-22T01:12:26.782 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-22T01:12:26.908 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-22T01:12:26.957 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-22T01:12:27.176 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:12:27.199 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-22T01:12:27.303 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-22T01:12:27.593 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-22T01:12:27.628 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-22T01:12:27.885 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-22T01:12:27.929 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-22T01:12:28.195 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-22T01:12:28.213 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-22T01:12:28.426 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-22T01:12:28.546 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-22T01:12:28.693 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-22T01:12:28.797 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-22T01:12:28.898 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:28.961 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-22T01:12:29.208 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-idna (3.3-1) ... 2024-04-22T01:12:29.238 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-22T01:12:29.517 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-22T01:12:29.615 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-22T01:12:29.900 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-22T01:12:30.235 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-22T01:12:30.361 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librados2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:30.487 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:30.614 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-22T01:12:30.672 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-22T01:12:30.895 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:12:30.960 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:31.017 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-22T01:12:31.085 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libradosstriper1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:31.202 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-22T01:12:31.276 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-22T01:12:31.402 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-22T01:12:31.478 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librbd1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:31.528 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-22T01:12:31.605 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-modules-core (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:31.731 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:31.821 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-22T01:12:31.940 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-22T01:12:31.940 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-22T01:12:32.072 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-22T01:12:32.323 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:12:32.424 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-22T01:12:32.434 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs-dev (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:32.552 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rados (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:32.717 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-22T01:12:32.720 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librgw2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:32.846 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephfs-top (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:32.968 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-22T01:12:33.227 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-22T01:12:33.331 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-22T01:12:33.478 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-22T01:12:33.721 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-22T01:12:33.972 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-22T01:12:34.098 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-22T01:12:34.374 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-22T01:12:34.492 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-22T01:12:34.527 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rbd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:34.653 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up rbd-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:34.776 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-22T01:12:34.779 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:34.905 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cephfs (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:35.031 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephfs-shell (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:35.069 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-22T01:12:35.140 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:35.320 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-22T01:12:35.446 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-22T01:12:35.572 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-22T01:12:35.661 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-22T01:12:35.666 INFO:teuthology.orchestra.run.smithi098.stdout:Adding group ceph....done 2024-04-22T01:12:35.890 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-22T01:12:35.993 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user ceph....done 2024-04-22T01:12:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:Setting system user ceph properties....done 2024-04-22T01:12:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-22T01:12:36.270 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-22T01:12:36.283 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-22T01:12:36.573 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-22T01:12:36.659 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-22T01:12:36.927 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-22T01:12:37.069 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-22T01:12:37.179 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephfs-mirror (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:37.328 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T01:12:37.385 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:12:37.385 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:12:37.454 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-22T01:12:37.756 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-22T01:12:37.822 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-test (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:37.939 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up radosgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:38.040 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-22T01:12:38.300 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-22T01:12:38.379 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:12:38.379 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:12:38.502 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-22T01:12:38.770 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-22T01:12:38.799 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-base (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:39.046 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-22T01:12:39.159 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-22T01:12:39.297 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-22T01:12:39.652 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mds (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:39.657 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-22T01:12:39.894 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-22T01:12:39.894 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-22T01:12:39.950 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-22T01:12:40.278 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-22T01:12:40.370 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:40.404 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librados2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:40.522 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:40.589 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-22T01:12:40.590 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-22T01:12:40.631 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-22T01:12:40.984 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libcephfs2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.080 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-osd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.110 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libradosstriper1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.236 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-22T01:12:41.362 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-22T01:12:41.362 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-22T01:12:41.512 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librbd1 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.638 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-modules-core (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.756 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.848 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:41.962 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-22T01:12:41.962 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-22T01:12:42.067 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:42.343 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mon (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:42.474 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libcephfs-dev (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:42.562 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-22T01:12:42.562 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-22T01:12:42.625 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rados (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:42.751 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librgw2 (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:42.877 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up cephfs-top (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:43.002 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-22T01:12:43.063 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:43.212 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:43.338 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-dashboard (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:43.490 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-volume (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:43.843 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-22T01:12:43.960 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-22T01:12:44.113 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-22T01:12:44.223 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rbd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:44.323 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up rbd-fuse (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:44.450 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:44.575 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cephfs (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:44.726 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up cephfs-shell (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:44.852 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-common (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:45.255 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-22T01:12:45.378 INFO:teuthology.orchestra.run.smithi070.stdout:Adding group ceph....done 2024-04-22T01:12:45.755 INFO:teuthology.orchestra.run.smithi070.stdout:Adding system user ceph....done 2024-04-22T01:12:45.915 INFO:teuthology.orchestra.run.smithi070.stdout:Setting system user ceph properties....done 2024-04-22T01:12:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-22T01:12:45.933 INFO:teuthology.orchestra.run.smithi070.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-22T01:12:45.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-22T01:12:46.036 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-22T01:12:46.071 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-22T01:12:46.243 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-22T01:12:46.244 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-22T01:12:46.406 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-22T01:12:46.430 INFO:teuthology.orchestra.run.smithi098.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-22T01:12:46.430 INFO:teuthology.orchestra.run.smithi098.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-22T01:12:46.430 INFO:teuthology.orchestra.run.smithi098.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-22T01:12:46.431 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:12:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:12:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:12:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:12:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T01:12:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T01:12:46.432 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:12:46.432 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-04-22T01:12:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-22T01:12:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-22T01:12:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-22T01:12:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-22T01:12:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-22T01:12:46.434 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-04-22T01:12:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-22T01:12:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-22T01:12:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-22T01:12:46.473 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-04-22T01:12:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-22T01:12:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-22T01:12:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-22T01:12:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-22T01:12:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-22T01:12:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-22T01:12:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-22T01:12:46.476 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-04-22T01:12:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-22T01:12:46.657 INFO:teuthology.orchestra.run.smithi098.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-22T01:12:46.657 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 55.7 MB of archives. 2024-04-22T01:12:46.657 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-22T01:12:46.657 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-22T01:12:46.903 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up cephfs-mirror (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:47.159 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:12:47.159 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:12:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-22T01:12:47.299 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-22T01:12:47.424 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-22T01:12:47.425 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-22T01:12:47.426 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-22T01:12:47.454 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-22T01:12:47.456 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-22T01:12:47.460 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-22T01:12:47.462 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-22T01:12:47.464 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-22T01:12:47.629 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-22T01:12:47.738 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-test (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:47.856 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up radosgw (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:47.921 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-22T01:12:47.931 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-22T01:12:47.936 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-22T01:12:48.021 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-22T01:12:48.294 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:12:48.295 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:12:48.774 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-base (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:49.109 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-22T01:12:49.114 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-22T01:12:49.115 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-22T01:12:49.121 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-22T01:12:49.122 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-22T01:12:49.122 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-22T01:12:49.132 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-22T01:12:49.304 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-22T01:12:49.567 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mds (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:49.795 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-22T01:12:49.796 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-22T01:12:49.902 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-22T01:12:49.903 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-22T01:12:50.082 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-22T01:12:50.083 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-22T01:12:50.121 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-22T01:12:50.182 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-22T01:12:50.188 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-22T01:12:50.189 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-22T01:12:50.190 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-22T01:12:50.193 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-22T01:12:50.286 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:50.399 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-22T01:12:50.504 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-22T01:12:50.504 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-22T01:12:50.810 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-22T01:12:51.020 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-osd (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:51.301 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-22T01:12:51.302 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-22T01:12:51.607 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-22T01:12:51.607 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-22T01:12:51.607 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-22T01:12:51.607 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-22T01:12:51.646 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-22T01:12:51.646 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-22T01:12:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-22T01:12:51.675 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-22T01:12:52.079 INFO:teuthology.orchestra.run.smithi098.stdout:Preconfiguring packages ... 2024-04-22T01:12:52.099 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:52.218 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:52.222 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 55.7 MB in 5s (10.7 MB/s) 2024-04-22T01:12:52.378 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mon (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:52.535 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-22T01:12:52.541 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-22T01:12:52.572 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-22T01:12:52.572 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-22T01:12:52.686 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-22T01:12:53.028 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-cephadm (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:53.171 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:53.297 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-dashboard (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:53.349 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-22T01:12:53.448 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-volume (19.0.0-2405-g820a2f8c-1jammy) ... 2024-04-22T01:12:53.784 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-22T01:12:53.893 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-22T01:12:54.029 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-22T01:12:54.308 INFO:teuthology.orchestra.run.smithi098.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-22T01:12:54.558 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package flex. 2024-04-22T01:12:54.656 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-22T01:12:54.662 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-22T01:12:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-22T01:12:55.020 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-22T01:12:55.240 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-22T01:12:55.404 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-22T01:12:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-22T01:12:55.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-22T01:12:55.726 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-04-22T01:12:55.773 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-22T01:12:55.926 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-04-22T01:12:55.927 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-04-22T01:12:56.114 INFO:teuthology.orchestra.run.smithi070.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-22T01:12:56.114 INFO:teuthology.orchestra.run.smithi070.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-22T01:12:56.114 INFO:teuthology.orchestra.run.smithi070.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-22T01:12:56.114 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T01:12:56.114 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T01:12:56.115 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T01:12:56.115 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T01:12:56.115 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T01:12:56.115 INFO:teuthology.orchestra.run.smithi070.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T01:12:56.116 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T01:12:56.116 INFO:teuthology.orchestra.run.smithi070.stdout:The following additional packages will be installed: 2024-04-22T01:12:56.116 INFO:teuthology.orchestra.run.smithi070.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-22T01:12:56.116 INFO:teuthology.orchestra.run.smithi070.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-22T01:12:56.116 INFO:teuthology.orchestra.run.smithi070.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-22T01:12:56.116 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-22T01:12:56.117 INFO:teuthology.orchestra.run.smithi070.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-22T01:12:56.117 INFO:teuthology.orchestra.run.smithi070.stdout:Suggested packages: 2024-04-22T01:12:56.117 INFO:teuthology.orchestra.run.smithi070.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-22T01:12:56.117 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-22T01:12:56.118 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-22T01:12:56.155 INFO:teuthology.orchestra.run.smithi070.stdout:The following NEW packages will be installed: 2024-04-22T01:12:56.156 INFO:teuthology.orchestra.run.smithi070.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-22T01:12:56.156 INFO:teuthology.orchestra.run.smithi070.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-22T01:12:56.157 INFO:teuthology.orchestra.run.smithi070.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-22T01:12:56.157 INFO:teuthology.orchestra.run.smithi070.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-22T01:12:56.157 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-22T01:12:56.157 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-22T01:12:56.157 INFO:teuthology.orchestra.run.smithi070.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-22T01:12:56.158 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages will be upgraded: 2024-04-22T01:12:56.159 INFO:teuthology.orchestra.run.smithi070.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-22T01:12:56.223 INFO:teuthology.orchestra.run.smithi070.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-22T01:12:56.223 INFO:teuthology.orchestra.run.smithi070.stdout:Need to get 55.7 MB of archives. 2024-04-22T01:12:56.223 INFO:teuthology.orchestra.run.smithi070.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-22T01:12:56.223 INFO:teuthology.orchestra.run.smithi070.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-22T01:12:56.301 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-22T01:12:56.306 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-22T01:12:56.384 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-22T01:12:56.393 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-22T01:12:56.433 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-22T01:12:56.433 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-22T01:12:56.434 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-22T01:12:56.445 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-22T01:12:56.445 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-22T01:12:56.446 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-22T01:12:56.448 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-22T01:12:56.449 INFO:teuthology.orchestra.run.smithi070.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-22T01:12:56.450 INFO:teuthology.orchestra.run.smithi070.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-22T01:12:56.451 INFO:teuthology.orchestra.run.smithi070.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-22T01:12:56.455 INFO:teuthology.orchestra.run.smithi070.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-22T01:12:56.473 INFO:teuthology.orchestra.run.smithi070.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-22T01:12:56.474 INFO:teuthology.orchestra.run.smithi070.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-22T01:12:56.475 INFO:teuthology.orchestra.run.smithi070.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-22T01:12:56.800 INFO:teuthology.orchestra.run.smithi070.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-22T01:12:56.801 INFO:teuthology.orchestra.run.smithi070.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-22T01:12:56.802 INFO:teuthology.orchestra.run.smithi070.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-22T01:12:56.804 INFO:teuthology.orchestra.run.smithi070.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-22T01:12:56.804 INFO:teuthology.orchestra.run.smithi070.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-22T01:12:56.808 INFO:teuthology.orchestra.run.smithi070.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-22T01:12:56.810 INFO:teuthology.orchestra.run.smithi070.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-22T01:12:56.817 INFO:teuthology.orchestra.run.smithi070.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-22T01:12:56.817 INFO:teuthology.orchestra.run.smithi070.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-22T01:12:56.837 INFO:teuthology.orchestra.run.smithi070.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-22T01:12:56.840 INFO:teuthology.orchestra.run.smithi070.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-22T01:12:56.846 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-22T01:12:56.850 INFO:teuthology.orchestra.run.smithi070.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-22T01:12:56.874 INFO:teuthology.orchestra.run.smithi070.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-22T01:12:56.877 INFO:teuthology.orchestra.run.smithi070.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-22T01:12:56.877 INFO:teuthology.orchestra.run.smithi070.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-22T01:12:56.878 INFO:teuthology.orchestra.run.smithi070.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-22T01:12:56.878 INFO:teuthology.orchestra.run.smithi070.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-22T01:12:56.888 INFO:teuthology.orchestra.run.smithi070.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-22T01:12:56.890 INFO:teuthology.orchestra.run.smithi070.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-22T01:12:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-22T01:12:57.111 INFO:teuthology.orchestra.run.smithi070.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-22T01:12:57.111 INFO:teuthology.orchestra.run.smithi070.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-22T01:12:57.111 INFO:teuthology.orchestra.run.smithi070.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-22T01:12:57.111 INFO:teuthology.orchestra.run.smithi070.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-22T01:12:57.111 INFO:teuthology.orchestra.run.smithi070.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-22T01:12:57.112 INFO:teuthology.orchestra.run.smithi070.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-22T01:12:57.123 INFO:teuthology.orchestra.run.smithi070.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-22T01:12:57.123 INFO:teuthology.orchestra.run.smithi070.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-22T01:12:57.313 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-22T01:12:57.528 INFO:teuthology.orchestra.run.smithi070.stdout:Preconfiguring packages ... 2024-04-22T01:12:57.679 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 55.7 MB in 1s (58.2 MB/s) 2024-04-22T01:12:57.720 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package bison. 2024-04-22T01:12:57.825 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-22T01:12:57.830 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-22T01:12:57.880 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-22T01:12:57.999 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-22T01:12:58.005 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-22T01:12:58.159 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-22T01:12:58.272 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-22T01:12:58.300 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-22T01:12:58.339 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-22T01:12:58.667 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package dump. 2024-04-22T01:12:58.695 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-22T01:12:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking dump (0.4b47-3) ... 2024-04-22T01:12:58.805 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-22T01:12:59.021 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package indent. 2024-04-22T01:12:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:12:59.079 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-22T01:12:59.332 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-22T01:12:59.348 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-22T01:12:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-22T01:12:59.728 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-22T01:12:59.756 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-22T01:12:59.757 INFO:teuthology.orchestra.run.smithi070.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-22T01:12:59.794 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-22T01:12:59.989 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package flex. 2024-04-22T01:13:00.047 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-22T01:13:00.063 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-22T01:13:00.084 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-22T01:13:00.090 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-22T01:13:00.097 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-22T01:13:00.138 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-22T01:13:00.426 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-22T01:13:00.454 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-22T01:13:00.492 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:00.645 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-22T01:13:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-22T01:13:00.773 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-22T01:13:00.811 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:00.818 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-22T01:13:01.064 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjson-perl. 2024-04-22T01:13:01.092 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-22T01:13:01.131 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-22T01:13:01.221 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-22T01:13:01.442 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-22T01:13:01.470 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-22T01:13:01.509 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-22T01:13:01.727 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-22T01:13:01.732 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-22T01:13:01.843 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-22T01:13:02.258 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-22T01:13:02.390 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-22T01:13:02.708 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-22T01:13:03.099 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package bison. 2024-04-22T01:13:03.200 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-22T01:13:03.206 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-22T01:13:03.212 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-22T01:13:03.240 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-22T01:13:03.250 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-22T01:13:03.279 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:03.540 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-22T01:13:03.556 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-22T01:13:03.598 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:03.910 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-22T01:13:03.937 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-22T01:13:03.960 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-22T01:13:03.976 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-22T01:13:03.988 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-22T01:13:04.026 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:04.312 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package dump. 2024-04-22T01:13:04.340 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-22T01:13:04.388 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking dump (0.4b47-3) ... 2024-04-22T01:13:04.399 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-22T01:13:04.437 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-22T01:13:04.666 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package indent. 2024-04-22T01:13:04.682 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:04.707 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-22T01:13:04.724 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-22T01:13:04.735 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-22T01:13:04.765 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-22T01:13:04.952 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-22T01:13:04.972 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-22T01:13:05.010 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-22T01:13:05.111 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-22T01:13:05.138 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:05.177 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:05.348 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-22T01:13:05.375 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-22T01:13:05.414 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-22T01:13:05.513 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-22T01:13:05.541 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:05.580 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:05.675 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-22T01:13:05.692 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-22T01:13:05.733 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-22T01:13:05.874 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-22T01:13:05.890 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:05.924 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:06.036 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-22T01:13:06.061 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-22T01:13:06.103 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:06.143 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-22T01:13:06.171 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-22T01:13:06.243 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-22T01:13:06.364 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-22T01:13:06.392 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-22T01:13:06.431 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:06.683 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libjson-perl. 2024-04-22T01:13:06.711 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-22T01:13:06.750 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-22T01:13:06.764 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-22T01:13:06.791 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-22T01:13:06.830 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-22T01:13:07.045 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-22T01:13:07.073 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-22T01:13:07.111 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-22T01:13:07.184 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package modemmanager. 2024-04-22T01:13:07.211 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-22T01:13:07.250 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:07.874 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package network-manager. 2024-04-22T01:13:07.902 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-22T01:13:07.948 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-22T01:13:08.456 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ppp. 2024-04-22T01:13:08.483 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-22T01:13:08.747 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-22T01:13:08.772 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-22T01:13:08.800 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-22T01:13:08.839 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:09.078 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package pptp-linux. 2024-04-22T01:13:09.099 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-22T01:13:09.105 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-22T01:13:09.116 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-22T01:13:09.144 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-22T01:13:09.158 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:09.397 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-22T01:13:09.413 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-22T01:13:09.455 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-22T01:13:09.494 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-22T01:13:09.521 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-22T01:13:09.560 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:09.733 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-22T01:13:09.761 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-22T01:13:09.800 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-client-common (238) ... 2024-04-22T01:13:09.872 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-22T01:13:09.899 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-22T01:13:09.938 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-22T01:13:10.087 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-22T01:13:10.115 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:10.153 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-22T01:13:10.286 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-22T01:13:10.324 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-22T01:13:10.669 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-22T01:13:10.697 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:10.735 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:10.775 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-common. 2024-04-22T01:13:10.803 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-22T01:13:10.859 INFO:teuthology.orchestra.run.smithi098.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-22T01:13:10.889 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-common (238) ... 2024-04-22T01:13:11.066 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-22T01:13:11.093 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:11.132 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-14. 2024-04-22T01:13:11.377 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:11.418 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:11.426 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-22T01:13:11.442 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:11.485 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:11.771 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-22T01:13:11.798 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-22T01:13:11.837 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-22T01:13:12.342 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-22T01:13:12.369 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-22T01:13:12.408 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-22T01:13:12.653 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql. 2024-04-22T01:13:12.682 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-22T01:13:12.719 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql (14+238) ... 2024-04-22T01:13:12.761 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package modemmanager. 2024-04-22T01:13:12.789 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-22T01:13:12.827 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:12.938 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-client. 2024-04-22T01:13:12.954 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-22T01:13:12.988 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-client (14+238) ... 2024-04-22T01:13:13.414 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-22T01:13:13.442 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-22T01:13:13.443 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package network-manager. 2024-04-22T01:13:13.471 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-22T01:13:13.517 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-22T01:13:13.791 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-22T01:13:14.159 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package tcl8.6. 2024-04-22T01:13:14.187 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-22T01:13:14.226 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-22T01:13:14.293 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ppp. 2024-04-22T01:13:14.321 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-22T01:13:14.512 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package tcl. 2024-04-22T01:13:14.540 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-22T01:13:14.583 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-22T01:13:14.617 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-22T01:13:14.865 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package wpasupplicant. 2024-04-22T01:13:14.893 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-22T01:13:14.939 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package pptp-linux. 2024-04-22T01:13:14.948 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-22T01:13:14.967 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-22T01:13:15.006 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-22T01:13:15.233 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-22T01:13:15.247 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-22T01:13:15.275 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-22T01:13:15.319 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-22T01:13:15.335 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-22T01:13:15.377 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-22T01:13:15.595 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-22T01:13:15.623 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-22T01:13:15.662 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql-client-common (238) ... 2024-04-22T01:13:15.739 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-22T01:13:15.767 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-22T01:13:15.814 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-22T01:13:15.974 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-22T01:13:16.002 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:16.040 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:16.101 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-client-common (238) ... 2024-04-22T01:13:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-22T01:13:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-22T01:13:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:16.587 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql-common. 2024-04-22T01:13:16.615 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-22T01:13:16.648 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-22T01:13:16.672 INFO:teuthology.orchestra.run.smithi070.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-22T01:13:16.702 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql-common (238) ... 2024-04-22T01:13:16.774 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-22T01:13:16.900 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:17.170 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql-14. 2024-04-22T01:13:17.197 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-22T01:13:17.239 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:17.399 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-22T01:13:17.489 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-22T01:13:17.875 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-22T01:13:17.876 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-22T01:13:18.325 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-22T01:13:18.451 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-22T01:13:18.482 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql. 2024-04-22T01:13:18.508 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-22T01:13:18.548 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql (14+238) ... 2024-04-22T01:13:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-22T01:13:18.728 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-22T01:13:18.776 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql-client. 2024-04-22T01:13:18.804 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-22T01:13:18.834 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql-client (14+238) ... 2024-04-22T01:13:18.837 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-22T01:13:18.963 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:19.044 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-22T01:13:19.061 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-22T01:13:19.103 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-22T01:13:19.430 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package tcl8.6. 2024-04-22T01:13:19.459 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-22T01:13:19.497 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-22T01:13:19.507 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-22T01:13:19.633 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-22T01:13:19.783 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package tcl. 2024-04-22T01:13:19.811 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-22T01:13:19.854 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-22T01:13:20.178 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package wpasupplicant. 2024-04-22T01:13:20.206 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-22T01:13:20.215 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-22T01:13:20.236 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-22T01:13:20.341 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up dump (0.4b47-3) ... 2024-04-22T01:13:20.434 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-22T01:13:20.492 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:20.574 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-22T01:13:20.601 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-22T01:13:20.602 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-22T01:13:20.640 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-22T01:13:20.703 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-22T01:13:20.828 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-22T01:13:20.955 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-22T01:13:21.019 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-22T01:13:21.047 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-22T01:13:21.069 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-22T01:13:21.093 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-22T01:13:21.139 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-22T01:13:21.265 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-22T01:13:21.390 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql-client-common (238) ... 2024-04-22T01:13:21.442 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-22T01:13:21.551 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:21.600 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-22T01:13:21.652 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-22T01:13:21.726 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-22T01:13:21.795 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-client (14+238) ... 2024-04-22T01:13:21.852 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:21.921 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-22T01:13:21.970 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-22T01:13:22.079 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-22T01:13:22.197 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:22.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:22.392 INFO:teuthology.orchestra.run.smithi098.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-22T01:13:22.392 INFO:teuthology.orchestra.run.smithi098.stdout:which means they are currently configured by ifupdown: 2024-04-22T01:13:22.392 INFO:teuthology.orchestra.run.smithi098.stdout:- enp3s0f1 2024-04-22T01:13:22.393 INFO:teuthology.orchestra.run.smithi098.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-22T01:13:22.393 INFO:teuthology.orchestra.run.smithi098.stdout:remove their configuration from /etc/network/interfaces. 2024-04-22T01:13:22.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-22T01:13:22.701 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-22T01:13:22.786 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-22T01:13:22.831 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-22T01:13:23.155 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-22T01:13:23.162 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-22T01:13:23.162 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-22T01:13:23.681 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-22T01:13:23.772 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:23.807 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-22T01:13:23.933 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-22T01:13:24.009 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:24.051 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-22T01:13:24.118 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-22T01:13:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-common (238) ... 2024-04-22T01:13:24.627 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-22T01:13:24.753 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:24.879 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-22T01:13:24.997 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-22T01:13:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:Adding user postgres to group ssl-cert 2024-04-22T01:13:25.364 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:25.364 INFO:teuthology.orchestra.run.smithi098.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-22T01:13:25.560 INFO:teuthology.orchestra.run.smithi098.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-22T01:13:25.560 INFO:teuthology.orchestra.run.smithi098.stdout:Removing obsolete dictionary files: 2024-04-22T01:13:25.627 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-22T01:13:25.753 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up dump (0.4b47-3) ... 2024-04-22T01:13:25.847 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-22T01:13:25.913 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:26.026 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-22T01:13:26.090 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-22T01:13:26.149 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-22T01:13:26.275 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-22T01:13:26.401 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-22T01:13:26.437 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:26.491 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-22T01:13:26.578 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-22T01:13:26.704 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-22T01:13:26.746 INFO:teuthology.orchestra.run.smithi098.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-22T01:13:26.766 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-22T01:13:26.813 INFO:teuthology.orchestra.run.smithi098.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-22T01:13:26.813 INFO:teuthology.orchestra.run.smithi098.stdout:This user must also own the server process. 2024-04-22T01:13:26.813 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:The database cluster will be initialized with locale "en_US". 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:The default text search configuration will be set to "english". 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:Data page checksums are disabled. 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:creating subdirectories ... ok 2024-04-22T01:13:26.814 INFO:teuthology.orchestra.run.smithi098.stdout:selecting dynamic shared memory implementation ... posix 2024-04-22T01:13:26.826 INFO:teuthology.orchestra.run.smithi098.stdout:selecting default max_connections ... 100 2024-04-22T01:13:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:selecting default shared_buffers ... 128MB 2024-04-22T01:13:26.853 INFO:teuthology.orchestra.run.smithi098.stdout:selecting default time zone ... Etc/UTC 2024-04-22T01:13:26.854 INFO:teuthology.orchestra.run.smithi098.stdout:creating configuration files ... ok 2024-04-22T01:13:26.880 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-22T01:13:26.961 INFO:teuthology.orchestra.run.smithi098.stdout:running bootstrap script ... ok 2024-04-22T01:13:27.031 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:27.165 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-22T01:13:27.325 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql-client (14+238) ... 2024-04-22T01:13:27.434 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-22T01:13:27.596 INFO:teuthology.orchestra.run.smithi098.stdout:performing post-bootstrap initialization ... ok 2024-04-22T01:13:27.884 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:27.884 INFO:teuthology.orchestra.run.smithi070.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-22T01:13:27.884 INFO:teuthology.orchestra.run.smithi070.stdout:which means they are currently configured by ifupdown: 2024-04-22T01:13:27.884 INFO:teuthology.orchestra.run.smithi070.stdout:- enp3s0f1 2024-04-22T01:13:27.884 INFO:teuthology.orchestra.run.smithi070.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-22T01:13:27.885 INFO:teuthology.orchestra.run.smithi070.stdout:remove their configuration from /etc/network/interfaces. 2024-04-22T01:13:27.885 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:28.009 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-22T01:13:28.301 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-22T01:13:28.315 INFO:teuthology.orchestra.run.smithi098.stdout:syncing data to disk ... ok 2024-04-22T01:13:28.635 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-22T01:13:28.700 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-22T01:13:29.129 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-22T01:13:29.280 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-22T01:13:29.415 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-22T01:13:29.541 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql-common (238) ... 2024-04-22T01:13:30.666 INFO:teuthology.orchestra.run.smithi070.stdout:Adding user postgres to group ssl-cert 2024-04-22T01:13:30.882 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:30.883 INFO:teuthology.orchestra.run.smithi070.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-22T01:13:31.078 INFO:teuthology.orchestra.run.smithi070.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-22T01:13:31.078 INFO:teuthology.orchestra.run.smithi070.stdout:Removing obsolete dictionary files: 2024-04-22T01:13:31.435 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:31.544 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-22T01:13:31.596 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-22T01:13:31.671 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql (14+238) ... 2024-04-22T01:13:31.797 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:31.923 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:31.959 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-22T01:13:32.178 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-22T01:13:32.178 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-22T01:13:32.276 INFO:teuthology.orchestra.run.smithi070.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-22T01:13:32.300 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-22T01:13:32.336 INFO:teuthology.orchestra.run.smithi070.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-22T01:13:32.336 INFO:teuthology.orchestra.run.smithi070.stdout:This user must also own the server process. 2024-04-22T01:13:32.336 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:32.336 INFO:teuthology.orchestra.run.smithi070.stdout:The database cluster will be initialized with locale "en_US". 2024-04-22T01:13:32.336 INFO:teuthology.orchestra.run.smithi070.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-22T01:13:32.336 INFO:teuthology.orchestra.run.smithi070.stdout:The default text search configuration will be set to "english". 2024-04-22T01:13:32.337 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:32.337 INFO:teuthology.orchestra.run.smithi070.stdout:Data page checksums are disabled. 2024-04-22T01:13:32.337 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:32.337 INFO:teuthology.orchestra.run.smithi070.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-22T01:13:32.337 INFO:teuthology.orchestra.run.smithi070.stdout:creating subdirectories ... ok 2024-04-22T01:13:32.337 INFO:teuthology.orchestra.run.smithi070.stdout:selecting dynamic shared memory implementation ... posix 2024-04-22T01:13:32.352 INFO:teuthology.orchestra.run.smithi070.stdout:selecting default max_connections ... 100 2024-04-22T01:13:32.372 INFO:teuthology.orchestra.run.smithi070.stdout:selecting default shared_buffers ... 128MB 2024-04-22T01:13:32.378 INFO:teuthology.orchestra.run.smithi070.stdout:selecting default time zone ... Etc/UTC 2024-04-22T01:13:32.379 INFO:teuthology.orchestra.run.smithi070.stdout:creating configuration files ... ok 2024-04-22T01:13:32.492 INFO:teuthology.orchestra.run.smithi070.stdout:running bootstrap script ... ok 2024-04-22T01:13:32.734 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-22T01:13:32.834 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-22T01:13:33.120 INFO:teuthology.orchestra.run.smithi070.stdout:performing post-bootstrap initialization ... ok 2024-04-22T01:13:33.137 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-22T01:13:33.272 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-22T01:13:33.771 INFO:teuthology.orchestra.run.smithi070.stdout:syncing data to disk ... ok 2024-04-22T01:13:34.148 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-22T01:13:34.617 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-22T01:13:34.717 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-22T01:13:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-22T01:13:35.214 DEBUG:teuthology.parallel:result is None 2024-04-22T01:13:37.108 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:37.234 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-22T01:13:37.360 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up postgresql (14+238) ... 2024-04-22T01:13:37.461 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-22T01:13:37.562 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-22T01:13:37.813 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-22T01:13:37.813 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-22T01:13:38.348 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-22T01:13:38.465 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-22T01:13:38.776 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-22T01:13:38.920 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-22T01:13:40.249 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-22T01:13:40.349 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-22T01:13:40.847 INFO:teuthology.orchestra.run.smithi070.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-22T01:13:40.852 DEBUG:teuthology.parallel:result is None 2024-04-22T01:13:40.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:13:41.068 DEBUG:teuthology.orchestra.run.smithi070:> dpkg-query -W -f '${Version}' ceph 2024-04-22T01:13:41.087 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:13:41.088 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:13:41.088 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405-g820a2f8c-1jammy is installed. 2024-04-22T01:13:41.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:13:41.305 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-query -W -f '${Version}' ceph 2024-04-22T01:13:41.325 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:13:41.325 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405-g820a2f8c-1jammy 2024-04-22T01:13:41.325 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405-g820a2f8c-1jammy is installed. 2024-04-22T01:13:41.327 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T01:13:41.327 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:41.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:13:41.341 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:41.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:13:41.382 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T01:13:41.383 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:41.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:13:41.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:13:41.451 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:41.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:13:41.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:13:41.516 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T01:13:41.517 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:41.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:13:41.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:13:41.582 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:41.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:13:41.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:13:41.652 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T01:13:41.653 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:41.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:13:41.667 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:13:41.722 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:41.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:13:41.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:13:41.789 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T01:13:41.875 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T01:13:41.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-04-22T01:13:41.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-04-22T01:13:41.891 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T01:13:41.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T01:13:41.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T01:13:41.945 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T01:13:41.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T01:13:41.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T01:13:42.008 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-22T01:13:42.008 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-22T01:13:42.008 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-22T01:13:42.008 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:42.008 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-22T01:13:42.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-22T01:13:42.118 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:42.118 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-22T01:13:42.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-22T01:13:42.203 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T01:13:42.203 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T01:13:42.203 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725', 'branch': 'squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quota}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7667725', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '430e09df97c8fc7dc2b2ae424f68ed11366c540f'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 676, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'sleep_before_teardown': 0, 'subset': '46/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '430e09df97c8fc7dc2b2ae424f68ed11366c540f', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBT/4mDsuiWTW/0UQNTgYB632NwPv23SHEKXswKvSLs3w6WQsINlxKnxQ+USx0MW9l9A7O6oXbDltROIEBWszbE=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBVkidFhzskkhvOaYJ2UFXFdVXUNhYyYHmYfy3KlfDCrPJLBxoM8mqDNyR85QobvOMariv66l2yHH/XOKE5kW0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_quota']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-21_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T01:13:42.204 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T01:13:42.206 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T01:13:42.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T01:13:42.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T01:13:42.302 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:42.302 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:13:42.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:13:42.311 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-22 01:07:36.219517806 +0000 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-22 01:07:36.215517842 +0000 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-22 01:07:36.215517842 +0000 2024-04-22T01:13:42.360 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-04-22T01:13:42.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:13:42.414 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-22T01:13:42.414 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-22T01:13:42.414 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000208101 s, 2.5 MB/s 2024-04-22T01:13:42.415 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:13:42.465 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-04-22T01:13:42.515 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:13:42.515 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:42.515 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-04-22T01:13:42.516 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:42.516 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-22 01:07:36.919511553 +0000 2024-04-22T01:13:42.516 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-22 01:07:36.675513733 +0000 2024-04-22T01:13:42.516 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-22 01:07:36.675513733 +0000 2024-04-22T01:13:42.516 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-04-22T01:13:42.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:13:42.570 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-22T01:13:42.570 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-22T01:13:42.570 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.0003091 s, 1.7 MB/s 2024-04-22T01:13:42.571 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:13:42.621 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-22 01:07:37.391507337 +0000 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-22 01:07:37.151509481 +0000 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-22 01:07:37.151509481 +0000 2024-04-22T01:13:42.668 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-04-22T01:13:42.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:13:42.721 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-22T01:13:42.721 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-22T01:13:42.722 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000192757 s, 2.7 MB/s 2024-04-22T01:13:42.723 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:13:42.773 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-04-22T01:13:42.820 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:13:42.820 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:42.821 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-04-22T01:13:42.821 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:42.821 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-22 01:07:37.627505229 +0000 2024-04-22T01:13:42.821 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-22 01:07:37.623505265 +0000 2024-04-22T01:13:42.821 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-22 01:07:37.623505265 +0000 2024-04-22T01:13:42.821 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-04-22T01:13:42.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:13:42.874 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-22T01:13:42.874 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-22T01:13:42.874 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000314538 s, 1.6 MB/s 2024-04-22T01:13:42.876 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:13:42.925 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:42.925 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:42.925 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:13:42.931 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:13:42.931 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-04-22T01:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-04-22T01:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 01:07:36.575568904 +0000 2024-04-22T01:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 01:07:36.343570976 +0000 2024-04-22T01:13:42.979 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 01:07:36.343570976 +0000 2024-04-22T01:13:42.979 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-22T01:13:42.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:13:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T01:13:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T01:13:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000450271 s, 1.1 MB/s 2024-04-22T01:13:43.034 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:13:43.083 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-04-22T01:13:43.129 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 01:07:37.035564795 +0000 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 01:07:36.783567046 +0000 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 01:07:36.783567046 +0000 2024-04-22T01:13:43.130 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-22T01:13:43.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:13:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T01:13:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T01:13:43.184 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000418487 s, 1.2 MB/s 2024-04-22T01:13:43.185 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:13:43.235 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-04-22T01:13:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 01:07:37.575559972 +0000 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 01:07:37.303562401 +0000 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 01:07:37.303562401 +0000 2024-04-22T01:13:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-22T01:13:43.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:13:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T01:13:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T01:13:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.0004306 s, 1.2 MB/s 2024-04-22T01:13:43.337 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:13:43.387 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 01:07:38.095555327 +0000 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 01:07:37.823557757 +0000 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 01:07:37.823557757 +0000 2024-04-22T01:13:43.435 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-22T01:13:43.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:13:43.488 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T01:13:43.488 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T01:13:43.488 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000322732 s, 1.6 MB/s 2024-04-22T01:13:43.490 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:13:43.539 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:43.539 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-22T01:13:43.539 INFO:tasks.ceph:Generating config... 2024-04-22T01:13:43.540 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-22T01:13:43.540 INFO:tasks.ceph:[client] debug client = 20 2024-04-22T01:13:43.540 INFO:tasks.ceph:[client] debug ms = 1 2024-04-22T01:13:43.540 INFO:tasks.ceph:[client] pid file = 2024-04-22T01:13:43.540 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-22T01:13:43.540 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-22T01:13:43.540 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-22T01:13:43.541 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-22T01:13:43.542 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-22T01:13:43.542 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-22T01:13:43.542 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T01:13:43.542 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T01:13:43.543 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T01:13:43.543 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-22T01:13:43.543 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T01:13:43.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T01:13:43.577 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T01:13:43.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T01:13:43.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:13:43.718 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.70'), ('mon.b', '172.21.15.98'), ('mon.c', '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]')] 2024-04-22T01:13:43.718 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.70,172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-22T01:13:43.719 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:43.719 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T01:13:43.763 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.70 --add b 172.21.15.98 --addv c '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:43.849 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: generated fsid b126cadd-d35d-4fad-9fb6-c51f861f8653 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 0 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:fsid b126cadd-d35d-4fad-9fb6-c51f861f8653 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:last_changed 2024-04-22T01:13:43.854434+0000 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-04-22T01:13:43.854434+0000 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:min_mon_release 17 (quincy) 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:election_strategy: 1 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-22T01:13:43.850 INFO:teuthology.orchestra.run.smithi070.stdout:1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-22T01:13:43.851 INFO:teuthology.orchestra.run.smithi070.stdout:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-04-22T01:13:43.851 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-22T01:13:43.852 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T01:13:43.900 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b126cadd-d35d-4fad-9fb6-c51f861f8653... 2024-04-22T01:13:43.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T01:13:43.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-04-22T01:13:43.973 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-04-22T01:13:43.974 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.975 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = "172.21.15.70,172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790]" 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = b126cadd-d35d-4fad-9fb6-c51f861f8653 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 600 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-04-22T01:13:43.976 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-04-22T01:13:43.977 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore block size = 96636764160 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore fsck on mount = True 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluefs = 20 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluestore = 20 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: debug rocksdb = 10 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T01:13:43.978 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd full ratio = 0.9 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd nearfull ratio = 0.8 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: osd failsafe full ratio = 0.95 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = bluestore 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: osd op complaint time = 180 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: debug client = 20 2024-04-22T01:13:43.979 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T01:13:43.980 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: mon down mkfs grace = 300 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: mon op complaint time = 120 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-04-22T01:13:43.981 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: client mount timeout = 600 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: debug client = 20 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = "" 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: rados mon op timeout = 900 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout: rados osd op timeout = 900 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.b] 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.c] 2024-04-22T01:13:43.982 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.a] 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.c] 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.b] 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.d] 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout:[mds] 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: debug mds = 20 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: debug mds balancer = 20 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: mds debug frag = True 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: mds debug scatterstat = True 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: mds op complaint time = 180 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: mds verify scatter = True 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: osd op complaint time = 180 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: rados mon op timeout = 900 2024-04-22T01:13:43.983 INFO:teuthology.orchestra.run.smithi070.stdout: rados osd op timeout = 900 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-04-22T01:13:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.985 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T01:13:43.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.70,172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790]" 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = b126cadd-d35d-4fad-9fb6-c51f861f8653 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-04-22T01:13:43.987 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-04-22T01:13:43.988 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-04-22T01:13:43.989 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:13:43.990 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-04-22T01:13:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: mon down mkfs grace = 300 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: mon op complaint time = 120 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: client mount timeout = 600 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = "" 2024-04-22T01:13:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout:[mds] 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds = 20 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds balancer = 20 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug frag = True 2024-04-22T01:13:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug scatterstat = True 2024-04-22T01:13:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mds op complaint time = 180 2024-04-22T01:13:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mds verify scatter = True 2024-04-22T01:13:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2024-04-22T01:13:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2024-04-22T01:13:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2024-04-22T01:13:43.996 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T01:13:43.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T01:13:44.086 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T01:13:44.086 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:44.086 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T01:13:44.093 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:44.093 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T01:13:44.139 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:13:44.140 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:44.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T01:13:44.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:13:44.204 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:13:44.204 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:44.257 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:13:44.257 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:44.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T01:13:44.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:13:44.280 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:13:44.281 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:44.330 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T01:13:44.330 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T01:13:44.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-22T01:13:44.374 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-22T01:13:44.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T01:13:44.419 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T01:13:44.439 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T01:13:44.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-22T01:13:44.483 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-22T01:13:44.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-22T01:13:44.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-22T01:13:44.627 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-22T01:13:44.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-22T01:13:44.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-22T01:13:44.735 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-22T01:13:44.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-22T01:13:44.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-22T01:13:44.828 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-22T01:13:44.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-22T01:13:44.885 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T01:13:44.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T01:13:44.920 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T01:13:44.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-22T01:13:45.008 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-22T01:13:45.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T01:13:45.055 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-22T01:13:45.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-22T01:13:45.142 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-22T01:13:45.153 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T01:13:45.153 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-22T01:13:45.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T01:13:45.167 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:45.167 INFO:tasks.ceph:role: osd.0 2024-04-22T01:13:45.167 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:13:45.167 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:13:45.226 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:13:45.227 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:13:45.227 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:13:45.227 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:13:45.251 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-22T01:13:45.253 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-22T01:13:45.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T01:13:45.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T01:13:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:13:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:13:45.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T01:13:45.419 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:45.419 INFO:tasks.ceph:role: osd.1 2024-04-22T01:13:45.419 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:13:45.419 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T01:13:45.475 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:13:45.475 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:13:45.475 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:13:45.476 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:13:45.500 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-22T01:13:45.502 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-22T01:13:45.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T01:13:45.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T01:13:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:13:45.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:13:45.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T01:13:45.665 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:45.666 INFO:tasks.ceph:role: osd.2 2024-04-22T01:13:45.666 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:13:45.666 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:13:45.719 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:13:45.720 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:13:45.720 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:13:45.720 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:13:45.743 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-22T01:13:45.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-22T01:13:45.745 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T01:13:45.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T01:13:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:13:45.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:13:45.861 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-22T01:13:45.914 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:45.914 INFO:tasks.ceph:role: osd.3 2024-04-22T01:13:45.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:13:45.914 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-22T01:13:45.967 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:13:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:13:45.969 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:13:45.969 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:13:45.991 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-22T01:13:45.993 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-22T01:13:45.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-22T01:13:46.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-22T01:13:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:13:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:13:46.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:46.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:46.218+0000 7f7f968667c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-22T01:13:46.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:46.218+0000 7f7f968667c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-22T01:13:46.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:46.218+0000 7f7f968667c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T01:13:46.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:46.218+0000 7f7f968667c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T01:13:46.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:46.218+0000 7f7f968667c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T01:13:46.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:46.218+0000 7f7f968667c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-22T01:13:49.196 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T01:13:49.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:49.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:49.290+0000 7f38bb7b77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-22T01:13:49.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:49.290+0000 7f38bb7b77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-22T01:13:49.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:49.290+0000 7f38bb7b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T01:13:49.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:49.290+0000 7f38bb7b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T01:13:49.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:49.290+0000 7f38bb7b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T01:13:49.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:49.290+0000 7f38bb7b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-22T01:13:52.247 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T01:13:52.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:52.390+0000 7fa5f54bf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-22T01:13:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:52.390+0000 7fa5f54bf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-22T01:13:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:52.390+0000 7fa5f54bf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T01:13:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:52.390+0000 7fa5f54bf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T01:13:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:52.390+0000 7fa5f54bf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T01:13:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:52.390+0000 7fa5f54bf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-22T01:13:55.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T01:13:55.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:13:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:55.482+0000 7f206c01e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-22T01:13:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:55.482+0000 7f206c01e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-22T01:13:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:55.482+0000 7f206c01e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T01:13:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:55.482+0000 7f206c01e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T01:13:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:55.482+0000 7f206c01e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T01:13:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:13:55.482+0000 7f206c01e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-22T01:13:58.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T01:13:58.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-22T01:13:58.534 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:58.534 INFO:tasks.ceph:role: osd.4 2024-04-22T01:13:58.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:13:58.535 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T01:13:58.598 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:13:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:13:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:13:58.598 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:13:58.599 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:13:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:13:58.599 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:13:58.599 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:13:58.599 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:13:58.599 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:13:59.078 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-04-22T01:13:59.080 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-04-22T01:13:59.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-22T01:13:59.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-22T01:13:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:13:59.244 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:13:59.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-22T01:13:59.301 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:59.301 INFO:tasks.ceph:role: osd.5 2024-04-22T01:13:59.301 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:13:59.301 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T01:13:59.364 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:13:59.365 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:13:59.809 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-04-22T01:13:59.811 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-04-22T01:13:59.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-22T01:13:59.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-22T01:13:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:13:59.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:13:59.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-22T01:13:59.971 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T01:13:59.971 INFO:tasks.ceph:role: osd.6 2024-04-22T01:13:59.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:13:59.972 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:14:00.032 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:14:00.494 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-04-22T01:14:00.496 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-04-22T01:14:00.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-22T01:14:00.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-22T01:14:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:14:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:14:00.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-22T01:14:00.644 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T01:14:00.644 INFO:tasks.ceph:role: osd.7 2024-04-22T01:14:00.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:14:00.645 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:14:00.710 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T01:14:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:14:00.711 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:14:01.203 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-04-22T01:14:01.205 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-04-22T01:14:01.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-22T01:14:01.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-22T01:14:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:14:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T01:14:01.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:14:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:01.483+0000 7f9bf6ac97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-22T01:14:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:01.487+0000 7f9bf6ac97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-22T01:14:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:01.487+0000 7f9bf6ac97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T01:14:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:01.487+0000 7f9bf6ac97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T01:14:01.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:01.487+0000 7f9bf6ac97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T01:14:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:01.499+0000 7f9bf6ac97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-22T01:14:04.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T01:14:04.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:14:04.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:04.587+0000 7fb641a287c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-22T01:14:04.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:04.587+0000 7fb641a287c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-22T01:14:04.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:04.587+0000 7fb641a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T01:14:04.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:04.587+0000 7fb641a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T01:14:04.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:04.587+0000 7fb641a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T01:14:04.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:04.587+0000 7fb641a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-22T01:14:07.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T01:14:07.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:14:07.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:07.703+0000 7f0b66db17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-22T01:14:07.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:07.703+0000 7f0b66db17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-22T01:14:07.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:07.703+0000 7f0b66db17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-22T01:14:07.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:07.703+0000 7f0b66db17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-22T01:14:07.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:07.703+0000 7f0b66db17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-22T01:14:07.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:07.703+0000 7f0b66db17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-22T01:14:10.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-22T01:14:10.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:14:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:10.803+0000 7f4263b017c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-22T01:14:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:10.803+0000 7f4263b017c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-22T01:14:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:10.803+0000 7f4263b017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-22T01:14:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:10.803+0000 7f4263b017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-22T01:14:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:10.803+0000 7f4263b017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-22T01:14:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:10.803+0000 7f4263b017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-22T01:14:12.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:14:12.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:14:13.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-22T01:14:13.802 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T01:14:13.802 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:13.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-22T01:14:13.816 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:13.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-22T01:14:13.870 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:13.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-22T01:14:13.926 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:13.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T01:14:13.977 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:13.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T01:14:14.033 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:14.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T01:14:14.087 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:14.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-22T01:14:14.142 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T01:14:14.155 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-22T01:14:14.208 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-22T01:14:14.264 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-22T01:14:14.320 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-22T01:14:14.376 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-22T01:14:14.428 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-22T01:14:14.480 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:14.480 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T01:14:14.486 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:14.486 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-22T01:14:14.535 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.535 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-22T01:14:14.540 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:14.541 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-22T01:14:14.589 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T01:14:14.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T01:14:14.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T01:14:14.635 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.y] 2024-04-22T01:14:14.635 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg== 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.a] 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw== 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.c] 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ== 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA== 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA== 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg== 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-04-22T01:14:14.636 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.b] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.d] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.4] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.5] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.6] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.7] 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ== 2024-04-22T01:14:14.637 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout:[client.2] 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA== 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout:[client.1] 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ== 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout:[client.3] 2024-04-22T01:14:14.638 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw== 2024-04-22T01:14:14.643 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-04-22T01:14:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg== 2024-04-22T01:14:14.643 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2024-04-22T01:14:14.643 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw== 2024-04-22T01:14:14.643 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w== 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2024-04-22T01:14:14.644 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[client.2] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA== 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-04-22T01:14:14.645 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ== 2024-04-22T01:14:14.646 INFO:teuthology.orchestra.run.smithi098.stdout:[client.3] 2024-04-22T01:14:14.646 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw== 2024-04-22T01:14:14.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T01:14:14.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T01:14:14.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:14.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:14.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:14.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:14.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:14.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:14.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.215 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T01:14:15.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T01:14:15.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:15.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:15.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:15.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-22T01:14:15.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:14:15.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:15.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:15.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:15.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:15.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:15.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:16.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:16.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:14:16.124 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T01:14:16.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T01:14:16.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T01:14:16.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T01:14:16.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-22T01:14:16.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T01:14:16.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-22T01:14:16.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-22T01:14:16.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T01:14:16.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-22T01:14:16.584 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:14:16.586 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:14:16.629 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T01:14:16.629 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T01:14:16.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T01:14:16.636 INFO:tasks.ceph.mon.a:Started 2024-04-22T01:14:16.636 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-22T01:14:16.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-22T01:14:16.678 INFO:tasks.ceph.mon.b:Started 2024-04-22T01:14:16.678 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T01:14:16.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T01:14:16.680 INFO:tasks.ceph.mon.c:Started 2024-04-22T01:14:16.680 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T01:14:16.680 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-22T01:14:16.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-22T01:14:16.683 INFO:tasks.ceph.mgr.y:Started 2024-04-22T01:14:16.683 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T01:14:16.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T01:14:16.685 INFO:tasks.ceph.mgr.x:Started 2024-04-22T01:14:16.685 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T01:14:16.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T01:14:16.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 Processor -- start 2024-04-22T01:14:16.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 -- start start 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda00cf3c0 con 0x7fcda00b6db0 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcda00b75e0 con 0x7fcda015de20 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda6f47640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcda00b78c0 con 0x7fcda00cea10 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcd9ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcda00b6db0 0x7fcda00d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46848/0 (socket says 172.21.15.70:46848) 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 learned_addr learned my addr 172.21.15.70:0/952170127 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.802+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:16.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T01:14:16.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:16.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:16.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:14:16.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:16.806+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:14:17.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:17.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:17.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.010+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:14:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T01:14:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T01:14:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:17.414+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T01:14:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T01:14:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:18.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda4ce3640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:18.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:18.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T01:14:18.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:18.218+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 msgr2=0x7fcda00cee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcda00cea10 0x7fcda00cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 msgr2=0x7fcda00b98a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.98:3300/0 conn(0x7fcda015de20 0x7fcda00b98a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 -- 172.21.15.70:0/952170127 >> v1:172.21.15.70:6789/0 conn(0x7fcda00b6db0 legacy=0x7fcda00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 -- 172.21.15.70:0/952170127 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda00b7ba0 con 0x7fcd8c0036d0 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.802+0000 7fcd9e7fc640 1 -- 172.21.15.70:0/952170127 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcda00b75e0 con 0x7fcd8c0073f0 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.806+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.806+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.806+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.806+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:19.806+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:14:20.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.010+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:20.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.010+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:20.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.010+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:20.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.010+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:20.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.010+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:14:20.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.414+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:20.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.414+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:20.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.414+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:20.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.414+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:20.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:20.414+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T01:14:21.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:21.218+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:21.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:21.218+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T01:14:21.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:21.218+0000 7fcd9ffff640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T01:14:21.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:21.218+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:14:21.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:21.218+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T01:14:22.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.058+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866458064 0 0) 0x7fcda00b7ba0 con 0x7fcd8c0036d0 2024-04-22T01:14:22.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.058+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd88003530 con 0x7fcd8c0036d0 2024-04-22T01:14:22.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3777625607 0 0) 0x7fcd88003530 con 0x7fcd8c0036d0 2024-04-22T01:14:22.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 msgr2=0x7fcd8c0098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T01:14:22.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 --2- 172.21.15.70:0/952170127 >> v2:172.21.15.70:3300/0 conn(0x7fcd8c0073f0 0x7fcd8c0098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8c00b010 con 0x7fcd8c0036d0 2024-04-22T01:14:22.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fcd90009a00 con 0x7fcd8c0036d0 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 >> v1:172.21.15.98:6789/0 conn(0x7fcd8c0036d0 legacy=0x7fcd8c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 reap_dead start 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880039c0 0x7fcd88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcd88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880052f0 0x7fcd8800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcda00b75e0 con 0x7fcd880052f0 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8800c240 con 0x7fcd880039c0 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd8800c440 con 0x7fcd88004810 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880039c0 0x7fcd88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcd88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880052f0 0x7fcd8800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 msgr2=0x7fcd88004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcd88004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880039c0 msgr2=0x7fcd88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880039c0 0x7fcd88003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.062+0000 7fcda54e4640 1 -- 172.21.15.70:0/952170127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd88003530 con 0x7fcd880052f0 2024-04-22T01:14:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.066+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880039c0 0x7fcd88003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.070+0000 7fcda54e4640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880052f0 0x7fcd8800b9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcda0161900 tx=0x7fcd9801ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e081c3d34f997c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.070+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd980ab070 con 0x7fcd880052f0 2024-04-22T01:14:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.070+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd980a5d30 con 0x7fcd880052f0 2024-04-22T01:14:22.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.070+0000 7fcd9f7fe640 1 -- 172.21.15.70:0/952170127 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9800f040 con 0x7fcd880052f0 2024-04-22T01:14:22.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 -- 172.21.15.70:0/952170127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880052f0 msgr2=0x7fcd8800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880052f0 0x7fcd8800b9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcda0161900 tx=0x7fcd9801ce40 comp rx=0 tx=0).stop 2024-04-22T01:14:22.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 -- 172.21.15.70:0/952170127 shutdown_connections 2024-04-22T01:14:22.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880052f0 0x7fcd8800b9d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcd88004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 --2- 172.21.15.70:0/952170127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880039c0 0x7fcd88003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 -- 172.21.15.70:0/952170127 >> 172.21.15.70:0/952170127 conn(0x7fcda0072d20 msgr2=0x7fcda0073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 -- 172.21.15.70:0/952170127 shutdown_connections 2024-04-22T01:14:22.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.074+0000 7fcda6f47640 1 -- 172.21.15.70:0/952170127 wait complete. 2024-04-22T01:14:22.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 Processor -- start 2024-04-22T01:14:22.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 -- start start 2024-04-22T01:14:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 0x7fcda0160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcda01f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880052f0 0x7fcda01f7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcda015e200 con 0x7fcd880039c0 2024-04-22T01:14:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcda00b8160 con 0x7fcd880052f0 2024-04-22T01:14:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcda00b71f0 con 0x7fcd88004810 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda4ce3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 0x7fcda0160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880052f0 0x7fcda01f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda4ce3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 0x7fcda0160110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35826/0 (socket says 172.21.15.70:35826) 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda4ce3640 1 -- 172.21.15.70:0/2282106774 learned_addr learned my addr 172.21.15.70:0/2282106774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcd9ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcda01f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 -- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 msgr2=0x7fcda01f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcda01f6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 -- 172.21.15.70:0/2282106774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 msgr2=0x7fcda0160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 0x7fcda0160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 -- 172.21.15.70:0/2282106774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda0160b10 con 0x7fcd880052f0 2024-04-22T01:14:22.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda4ce3640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 0x7fcda0160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcd9ffff640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcda01f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda54e4640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880052f0 0x7fcda01f7130 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd980ad000 tx=0x7fcd98019c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd980ab040 con 0x7fcd880052f0 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd980afd30 con 0x7fcd880052f0 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98004260 con 0x7fcd880052f0 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda0160df0 con 0x7fcd880052f0 2024-04-22T01:14:22.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.078+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda01613b0 con 0x7fcd880052f0 2024-04-22T01:14:22.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.082+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda01499c0 con 0x7fcd880052f0 2024-04-22T01:14:22.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.082+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcd980c1020 con 0x7fcd880052f0 2024-04-22T01:14:22.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.082+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcd98003c10 con 0x7fcd880052f0 2024-04-22T01:14:22.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.082+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcda01499c0 con 0x7fcd880052f0 2024-04-22T01:14:22.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.150+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fcda00b2c80 con 0x7fcd880052f0 2024-04-22T01:14:22.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcd9d7fa640 1 -- 172.21.15.70:0/2282106774 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fcd98016be0 con 0x7fcd880052f0 2024-04-22T01:14:22.157 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-04-22T01:14:22.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880052f0 msgr2=0x7fcda01f7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880052f0 0x7fcda01f7130 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd980ad000 tx=0x7fcd98019c30 comp rx=0 tx=0).stop 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 shutdown_connections 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd880052f0 0x7fcda01f7130 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd88004810 0x7fcda01f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 --2- 172.21.15.70:0/2282106774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd880039c0 0x7fcda0160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 >> 172.21.15.70:0/2282106774 conn(0x7fcda0072d20 msgr2=0x7fcda00b8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 shutdown_connections 2024-04-22T01:14:22.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:22.158+0000 7fcda6f47640 1 -- 172.21.15.70:0/2282106774 wait complete. 2024-04-22T01:14:22.176 INFO:tasks.ceph:check_enable_crimson: False 2024-04-22T01:14:22.176 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T01:14:22.177 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:22.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T01:14:22.187 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:22.187 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T01:14:22.240 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:22.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T01:14:22.283 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:22.286+0000 7f84bc87e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:14:22.292 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:22.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-22T01:14:22.321 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:22.323+0000 7fb2046cf180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:14:22.344 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:22.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-22T01:14:22.358 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:22.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-22T01:14:22.412 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:22.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-22T01:14:22.464 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:14:22.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-22T01:14:22.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 50e61bd7-7279-4366-9afa-06822339ded6 0 2024-04-22T01:14:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 Processor -- start 2024-04-22T01:14:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 -- start start 2024-04-22T01:14:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd87815f450 0x7fd8780cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd87809a440 0x7fd8780c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87809a920 con 0x7fd87815f830 2024-04-22T01:14:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd8780c5720 con 0x7fd87815f450 2024-04-22T01:14:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd87dcf2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd87809abc0 con 0x7fd87809a440 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd876ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd87815f830 0x7fd878099d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36116/0 (socket says 172.21.15.98:36116) 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd877fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd87809a440 0x7fd8780c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd876ffd640 1 -- 172.21.15.98:0/1405887414 learned_addr learned my addr 172.21.15.98:0/1405887414 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.679+0000 7fd8777fe640 1 --2- 172.21.15.98:0/1405887414 >> v2:172.21.15.70:3300/0 conn(0x7fd87815f450 0x7fd8780cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd8767fc640 1 -- 172.21.15.98:0/1405887414 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 389924558 0 0) 0x7fd87809a920 con 0x7fd87815f830 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd8767fc640 1 -- 172.21.15.98:0/1405887414 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd864003530 con 0x7fd87815f830 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd877fff640 1 -- 172.21.15.98:0/1405887414 >> v2:172.21.15.70:3300/0 conn(0x7fd87815f450 msgr2=0x7fd8780cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd877fff640 1 --2- 172.21.15.98:0/1405887414 >> v2:172.21.15.70:3300/0 conn(0x7fd87815f450 0x7fd8780cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd877fff640 1 -- 172.21.15.98:0/1405887414 >> v1:172.21.15.98:6789/0 conn(0x7fd87815f830 legacy=0x7fd878099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd877fff640 1 -- 172.21.15.98:0/1405887414 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87809aea0 con 0x7fd87809a440 2024-04-22T01:14:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd8777fe640 1 --2- 172.21.15.98:0/1405887414 >> v2:172.21.15.70:3300/0 conn(0x7fd87815f450 0x7fd8780cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd877fff640 1 --2- 172.21.15.98:0/1405887414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd8780c5bb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd87806e5a0 tx=0x7fd87001c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dc831860d3891c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd8767fc640 1 -- 172.21.15.98:0/1405887414 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700b2070 con 0x7fd87809a440 2024-04-22T01:14:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd8767fc640 1 -- 172.21.15.98:0/1405887414 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8700a59e0 con 0x7fd87809a440 2024-04-22T01:14:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd8767fc640 1 -- 172.21.15.98:0/1405887414 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700a5db0 con 0x7fd87809a440 2024-04-22T01:14:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 -- 172.21.15.98:0/1405887414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 msgr2=0x7fd8780c5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/1405887414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd8780c5bb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd87806e5a0 tx=0x7fd87001c900 comp rx=0 tx=0).stop 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 -- 172.21.15.98:0/1405887414 shutdown_connections 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/1405887414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd8780c5bb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/1405887414 >> v2:172.21.15.70:3300/0 conn(0x7fd87815f450 0x7fd8780cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 -- 172.21.15.98:0/1405887414 >> 172.21.15.98:0/1405887414 conn(0x7fd8780b36c0 msgr2=0x7fd8780b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 -- 172.21.15.98:0/1405887414 shutdown_connections 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.683+0000 7fd87dcf2640 1 -- 172.21.15.98:0/1405887414 wait complete. 2024-04-22T01:14:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 Processor -- start 2024-04-22T01:14:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- start start 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd878155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd87815f450 0x7fd8780b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd87815f830 0x7fd8780cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8780c6c90 con 0x7fd87815f450 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8780c6b10 con 0x7fd87809a440 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd8780c6e10 con 0x7fd87815f830 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd878155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd878155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54892/0 (socket says 172.21.15.98:54892) 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 -- 172.21.15.98:0/2488066183 learned_addr learned my addr 172.21.15.98:0/2488066183 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd877fff640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd87815f830 0x7fd8780cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd876ffd640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd87815f450 0x7fd8780b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 -- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd87815f830 msgr2=0x7fd8780cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd87815f830 0x7fd8780cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 -- 172.21.15.98:0/2488066183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd87815f450 msgr2=0x7fd8780b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd87815f450 0x7fd8780b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 -- 172.21.15.98:0/2488066183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8780cb740 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd8777fe640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd878155200 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd86000e870 tx=0x7fd86000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd860017070 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86000c8d0 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86000f040 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8780b5f20 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8780b64e0 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd86001a070 con 0x7fd87809a440 2024-04-22T01:14:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fd860013750 con 0x7fd87809a440 2024-04-22T01:14:22.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.687+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd878068920 con 0x7fd87809a440 2024-04-22T01:14:22.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.691+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd86001c6d0 con 0x7fd87809a440 2024-04-22T01:14:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.759+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "50e61bd7-7279-4366-9afa-06822339ded6", "id": 0} v 0) -- 0x7fd878071c30 con 0x7fd87809a440 2024-04-22T01:14:22.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.767+0000 7fd86ffff640 1 -- 172.21.15.98:0/2488066183 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "50e61bd7-7279-4366-9afa-06822339ded6", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fd86000cca0 con 0x7fd87809a440 2024-04-22T01:14:22.769 INFO:teuthology.orchestra.run.smithi098.stdout:0 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 msgr2=0x7fd878155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd878155200 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd86000e870 tx=0x7fd86000ed30 comp rx=0 tx=0).stop 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 shutdown_connections 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd87815f830 0x7fd8780cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd87815f450 0x7fd8780b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 --2- 172.21.15.98:0/2488066183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd87809a440 0x7fd878155200 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 >> 172.21.15.98:0/2488066183 conn(0x7fd8780b36c0 msgr2=0x7fd878156160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 shutdown_connections 2024-04-22T01:14:22.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.771+0000 7fd87dcf2640 1 -- 172.21.15.98:0/2488066183 wait complete. 2024-04-22T01:14:22.771 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:22.773+0000 7f84bc87e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:14:22.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 21575948-910b-45c7-903d-7cb907f90dc0 1 2024-04-22T01:14:22.835 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:22.835+0000 7fb2046cf180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:14:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4982c6b640 1 Processor -- start 2024-04-22T01:14:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4982c6b640 1 -- start start 2024-04-22T01:14:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4982c6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c06e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4982c6b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f497c06c5a0 con 0x7f497c15b100 2024-04-22T01:14:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4982c6b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f497c06c880 con 0x7f497c0ae280 2024-04-22T01:14:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4982c6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f497c0b60f0 con 0x7f497c15ad20 2024-04-22T01:14:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4980a07640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c06e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.907+0000 7f4981208640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f497c0ae280 0x7f497c06b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36120/0 (socket says 172.21.15.98:36120) 2024-04-22T01:14:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4981208640 1 -- 172.21.15.98:0/3853646936 learned_addr learned my addr 172.21.15.98:0/3853646936 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-04-22T01:14:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4980a07640 1 -- 172.21.15.98:0/3853646936 >> v1:172.21.15.98:6789/0 conn(0x7f497c0ae280 legacy=0x7f497c06b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4980a07640 1 -- 172.21.15.98:0/3853646936 >> v1:172.21.15.70:6789/0 conn(0x7f497c15b100 legacy=0x7f497c06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4980a07640 1 -- 172.21.15.98:0/3853646936 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c0b5aa0 con 0x7f497c15ad20 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4980a07640 1 --2- 172.21.15.98:0/3853646936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c06e4d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4974009dc0 tx=0x7f497401c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=367531663628764a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f497b7fe640 1 -- 172.21.15.98:0/3853646936 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740b2070 con 0x7f497c15ad20 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f497b7fe640 1 -- 172.21.15.98:0/3853646936 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49740a5a00 con 0x7f497c15ad20 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f497b7fe640 1 -- 172.21.15.98:0/3853646936 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740a5dd0 con 0x7f497c15ad20 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 -- 172.21.15.98:0/3853646936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 msgr2=0x7f497c06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 --2- 172.21.15.98:0/3853646936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c06e4d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4974009dc0 tx=0x7f497401c920 comp rx=0 tx=0).stop 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 -- 172.21.15.98:0/3853646936 shutdown_connections 2024-04-22T01:14:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 --2- 172.21.15.98:0/3853646936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c06e4d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 -- 172.21.15.98:0/3853646936 >> 172.21.15.98:0/3853646936 conn(0x7f497c099a60 msgr2=0x7f497c0b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 -- 172.21.15.98:0/3853646936 shutdown_connections 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.911+0000 7f4982c6b640 1 -- 172.21.15.98:0/3853646936 wait complete. 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 Processor -- start 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- start start 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 0x7f497c0afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c15b100 0x7f497c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f497c06ede0 con 0x7f497c15b100 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f497c06ec60 con 0x7f497c0ae280 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f497c169270 con 0x7f497c15ad20 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4980a07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 0x7f497c0afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4980a07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 0x7f497c0afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54902/0 (socket says 172.21.15.98:54902) 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59408/0 (socket says 172.21.15.98:59408) 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4980a07640 1 -- 172.21.15.98:0/2557592780 learned_addr learned my addr 172.21.15.98:0/2557592780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4981208640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c15b100 0x7f497c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 -- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 msgr2=0x7f497c0afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 0x7f497c0afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 -- 172.21.15.98:0/2557592780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c15b100 msgr2=0x7f497c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c15b100 0x7f497c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 -- 172.21.15.98:0/2557592780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c149e60 con 0x7f497c15ad20 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4980a07640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 0x7f497c0afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f497bfff640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c148340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f497000b440 tx=0x7f497000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970011830 con 0x7f497c15ad20 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4970016070 con 0x7f497c15ad20 2024-04-22T01:14:22.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970011c10 con 0x7f497c15ad20 2024-04-22T01:14:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c1498a0 con 0x7f497c15ad20 2024-04-22T01:14:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c15c7c0 con 0x7f497c15ad20 2024-04-22T01:14:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4970004020 con 0x7f497c15ad20 2024-04-22T01:14:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f497001b460 con 0x7f497c15ad20 2024-04-22T01:14:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.915+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c068920 con 0x7f497c15ad20 2024-04-22T01:14:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.927+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4970020020 con 0x7f497c15ad20 2024-04-22T01:14:22.941 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:22.941+0000 7f84bc87e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:14:22.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:22.995+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "21575948-910b-45c7-903d-7cb907f90dc0", "id": 1} v 0) -- 0x7f497c06d410 con 0x7f497c15ad20 2024-04-22T01:14:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.003+0000 7f49797fa640 1 -- 172.21.15.98:0/2557592780 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21575948-910b-45c7-903d-7cb907f90dc0", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f4970011db0 con 0x7f497c15ad20 2024-04-22T01:14:23.004 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:14:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 msgr2=0x7f497c148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c148340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f497000b440 tx=0x7f497000b900 comp rx=0 tx=0).stop 2024-04-22T01:14:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 shutdown_connections 2024-04-22T01:14:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c15b100 0x7f497c0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15ad20 0x7f497c148340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 --2- 172.21.15.98:0/2557592780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c0ae280 0x7f497c0afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 >> 172.21.15.98:0/2557592780 conn(0x7f497c099a60 msgr2=0x7f497c0cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 shutdown_connections 2024-04-22T01:14:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.007+0000 7f4982c6b640 1 -- 172.21.15.98:0/2557592780 wait complete. 2024-04-22T01:14:23.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 78c44d13-1d12-40d9-88b3-6df647d83dcc 2 2024-04-22T01:14:23.032 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.031+0000 7fb2046cf180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:14:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 Processor -- start 2024-04-22T01:14:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 -- start start 2024-04-22T01:14:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f40b015f380 0x7f40b00d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b00cc710 0x7f40b00cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40b0099da0 0x7f40b009a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f40b015f760 con 0x7f40b015f380 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f40b009a710 con 0x7f40b0099da0 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.135+0000 7f40b6089640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f40b00d0c00 con 0x7f40b00cc710 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40b0099da0 0x7f40b009a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40a6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b00cc710 0x7f40b00cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40b0099da0 0x7f40b009a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54912/0 (socket says 172.21.15.98:54912) 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 -- 172.21.15.98:0/4240719762 learned_addr learned my addr 172.21.15.98:0/4240719762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40af7fe640 1 --2- 172.21.15.98:0/4240719762 >> v2:172.21.15.70:3300/0 conn(0x7f40b015f380 0x7f40b00d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 -- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b00cc710 msgr2=0x7f40b00cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 --2- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b00cc710 0x7f40b00cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 -- 172.21.15.98:0/4240719762 >> v2:172.21.15.70:3300/0 conn(0x7f40b015f380 msgr2=0x7f40b00d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:23.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 --2- 172.21.15.98:0/4240719762 >> v2:172.21.15.70:3300/0 conn(0x7f40b015f380 0x7f40b00d06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 -- 172.21.15.98:0/4240719762 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b00ce190 con 0x7f40b0099da0 2024-04-22T01:14:23.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40a6ffd640 1 --2- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b00cc710 0x7f40b00cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:23.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40affff640 1 --2- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b0099da0 0x7f40b009a1d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f40b0160a10 tx=0x7f40a801c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35d3dc0f78da457e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40aeffd640 1 -- 172.21.15.98:0/4240719762 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80a55e0 con 0x7f40b0099da0 2024-04-22T01:14:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40aeffd640 1 -- 172.21.15.98:0/4240719762 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a80a5780 con 0x7f40b0099da0 2024-04-22T01:14:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40aeffd640 1 -- 172.21.15.98:0/4240719762 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80a5b50 con 0x7f40b0099da0 2024-04-22T01:14:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 -- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b0099da0 msgr2=0x7f40b009a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 --2- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b0099da0 0x7f40b009a1d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f40b0160a10 tx=0x7f40a801c960 comp rx=0 tx=0).stop 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 -- 172.21.15.98:0/4240719762 shutdown_connections 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 --2- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b0099da0 0x7f40b009a1d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 --2- 172.21.15.98:0/4240719762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b00cc710 0x7f40b00cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 --2- 172.21.15.98:0/4240719762 >> v2:172.21.15.70:3300/0 conn(0x7f40b015f380 0x7f40b00d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 -- 172.21.15.98:0/4240719762 >> 172.21.15.98:0/4240719762 conn(0x7f40b00b36f0 msgr2=0x7f40b00b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 -- 172.21.15.98:0/4240719762 shutdown_connections 2024-04-22T01:14:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.139+0000 7f40b6089640 1 -- 172.21.15.98:0/4240719762 wait complete. 2024-04-22T01:14:23.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 Processor -- start 2024-04-22T01:14:23.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- start start 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40b0099da0 0x7f40b0152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b00cc710 0x7f40b00b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b015f380 0x7f40b00b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40b00cf920 con 0x7f40b0099da0 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f40b00cd450 con 0x7f40b00cc710 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f40b00cfaa0 con 0x7f40b015f380 2024-04-22T01:14:23.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b00cc710 0x7f40b00b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b015f380 0x7f40b00b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b00cc710 0x7f40b00b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54916/0 (socket says 172.21.15.98:54916) 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40a6ffd640 1 -- 172.21.15.98:0/2680968395 learned_addr learned my addr 172.21.15.98:0/2680968395 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40af7fe640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40b0099da0 0x7f40b0152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 -- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b00cc710 msgr2=0x7f40b00b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b00cc710 0x7f40b00b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 -- 172.21.15.98:0/2680968395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40b0099da0 msgr2=0x7f40b0152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40b0099da0 0x7f40b0152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 -- 172.21.15.98:0/2680968395 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b0159d70 con 0x7f40b015f380 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40affff640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b015f380 0x7f40b00b5bf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f40b0072d80 tx=0x7f40a8019700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80bc070 con 0x7f40b015f380 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a80b8690 con 0x7f40b015f380 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a8009510 con 0x7f40b015f380 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b0152640 con 0x7f40b015f380 2024-04-22T01:14:23.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b00b5770 con 0x7f40b015f380 2024-04-22T01:14:23.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f40a80b8a80 con 0x7f40b015f380 2024-04-22T01:14:23.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f40a80c0c80 con 0x7f40b015f380 2024-04-22T01:14:23.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.143+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b0068920 con 0x7f40b015f380 2024-04-22T01:14:23.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.147+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f40a80165e0 con 0x7f40b015f380 2024-04-22T01:14:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.215+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "78c44d13-1d12-40d9-88b3-6df647d83dcc", "id": 2} v 0) -- 0x7f40b01597b0 con 0x7f40b015f380 2024-04-22T01:14:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.219+0000 7f40acff9640 1 -- 172.21.15.98:0/2680968395 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78c44d13-1d12-40d9-88b3-6df647d83dcc", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f40a80c0400 con 0x7f40b015f380 2024-04-22T01:14:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:2 2024-04-22T01:14:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.219+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b015f380 msgr2=0x7f40b00b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.219+0000 7f40b6089640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b015f380 0x7f40b00b5bf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f40b0072d80 tx=0x7f40a8019700 comp rx=0 tx=0).stop 2024-04-22T01:14:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 shutdown_connections 2024-04-22T01:14:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40b015f380 0x7f40b00b5bf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40b00cc710 0x7f40b00b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 --2- 172.21.15.98:0/2680968395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40b0099da0 0x7f40b0152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 >> 172.21.15.98:0/2680968395 conn(0x7f40b00b36f0 msgr2=0x7f40b0070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 shutdown_connections 2024-04-22T01:14:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.223+0000 7f40b6089640 1 -- 172.21.15.98:0/2680968395 wait complete. 2024-04-22T01:14:23.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e 3 2024-04-22T01:14:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 Processor -- start 2024-04-22T01:14:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 -- start start 2024-04-22T01:14:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a900cd530 0x7f3a9006d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9006c200 0x7f3a9015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a9006c7e0 con 0x7f3a900cd910 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9006cac0 con 0x7f3a900cd530 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.391+0000 7f3a949b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9006cda0 con 0x7f3a9006c200 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9006c200 0x7f3a9015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a900cd530 0x7f3a9006d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a87fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3a900cd910 0x7f3a900d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36132/0 (socket says 172.21.15.98:36132) 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a87fff640 1 -- 172.21.15.98:0/448892001 learned_addr learned my addr 172.21.15.98:0/448892001 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8e59c640 1 -- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9006c200 msgr2=0x7f3a9015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8e59c640 1 --2- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9006c200 0x7f3a9015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8e59c640 1 -- 172.21.15.98:0/448892001 >> v1:172.21.15.98:6789/0 conn(0x7f3a900cd910 legacy=0x7f3a900d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8e59c640 1 -- 172.21.15.98:0/448892001 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9006d080 con 0x7f3a900cd530 2024-04-22T01:14:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8dd9b640 1 -- 172.21.15.98:0/448892001 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180735322 0 0) 0x7f3a9006c7e0 con 0x7f3a900cd910 2024-04-22T01:14:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8e59c640 1 --2- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a9006d880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3a780097c0 tx=0x7f3a7801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=770f724cd1f633a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8dd9b640 1 -- 172.21.15.98:0/448892001 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a780b2070 con 0x7f3a900cd530 2024-04-22T01:14:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8dd9b640 1 -- 172.21.15.98:0/448892001 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a780ad680 con 0x7f3a900cd530 2024-04-22T01:14:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.395+0000 7f3a8dd9b640 1 -- 172.21.15.98:0/448892001 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a780ada50 con 0x7f3a900cd530 2024-04-22T01:14:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 msgr2=0x7f3a9006d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 --2- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a9006d880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3a780097c0 tx=0x7f3a7801c470 comp rx=0 tx=0).stop 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/448892001 shutdown_connections 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 --2- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9006c200 0x7f3a9015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 --2- 172.21.15.98:0/448892001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a9006d880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/448892001 >> 172.21.15.98:0/448892001 conn(0x7f3a900d0ae0 msgr2=0x7f3a900d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/448892001 shutdown_connections 2024-04-22T01:14:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/448892001 wait complete. 2024-04-22T01:14:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 Processor -- start 2024-04-22T01:14:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- start start 2024-04-22T01:14:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a9006c200 0x7f3a9013c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a900b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 0x7f3a9015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90168b10 con 0x7f3a9006c200 2024-04-22T01:14:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90168990 con 0x7f3a900cd530 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a90168c90 con 0x7f3a900cd910 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a900b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a8ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 0x7f3a9015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a900b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54934/0 (socket says 172.21.15.98:54934) 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a8ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 0x7f3a9015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59440/0 (socket says 172.21.15.98:59440) 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 -- 172.21.15.98:0/3773431407 learned_addr learned my addr 172.21.15.98:0/3773431407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a8e59c640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a9006c200 0x7f3a9013c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 -- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 msgr2=0x7f3a9015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 0x7f3a9015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 -- 172.21.15.98:0/3773431407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a9006c200 msgr2=0x7f3a9013c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a9006c200 0x7f3a9013c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 -- 172.21.15.98:0/3773431407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a900b4c20 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a8ed9d640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 0x7f3a9015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a8e59c640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a9006c200 0x7f3a9013c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a87fff640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a900b5670 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000c4b0 tx=0x7f3a8000c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80018070 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a80007500 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9015a090 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80004a30 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9015a650 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a8001b070 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.399+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f3a80004bd0 con 0x7f3a900cd530 2024-04-22T01:14:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.403+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a900703d0 con 0x7f3a900cd530 2024-04-22T01:14:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.403+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3a8001d440 con 0x7f3a900cd530 2024-04-22T01:14:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.471+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e", "id": 3} v 0) -- 0x7f3a9004f5f0 con 0x7f3a900cd530 2024-04-22T01:14:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a86ffd640 1 -- 172.21.15.98:0/3773431407 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f3a8001d700 con 0x7f3a900cd530 2024-04-22T01:14:23.474 INFO:teuthology.orchestra.run.smithi098.stdout:3 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 msgr2=0x7f3a900b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a900b5670 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000c4b0 tx=0x7f3a8000c970 comp rx=0 tx=0).stop 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 shutdown_connections 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a900cd910 0x7f3a9015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a900cd530 0x7f3a900b5670 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 --2- 172.21.15.98:0/3773431407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a9006c200 0x7f3a9013c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 >> 172.21.15.98:0/3773431407 conn(0x7f3a900d0ae0 msgr2=0x7f3a90142d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 shutdown_connections 2024-04-22T01:14:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.475+0000 7f3a949b3640 1 -- 172.21.15.98:0/3773431407 wait complete. 2024-04-22T01:14:23.484 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.485+0000 7f84bc87e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:14:23.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 310bb8df-5bcd-4966-9e02-98bcd61e596c 4 2024-04-22T01:14:23.537 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.537+0000 7f84bc87e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:14:23.589 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.589+0000 7f84bc87e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:14:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 Processor -- start 2024-04-22T01:14:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 -- start start 2024-04-22T01:14:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdefc15f450 0x7fdefc0cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdefc15f830 0x7fdefc099d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdefc09a920 con 0x7fdefc0b2b90 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdefc0c5720 con 0x7fdefc15f830 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.599+0000 7fdf023db640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdefc09abc0 con 0x7fdefc15f450 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdefc15f450 0x7fdefc0cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf00978640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdefc0b2b90 0x7fdefc0cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36134/0 (socket says 172.21.15.98:36134) 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf00978640 1 -- 172.21.15.98:0/719235707 learned_addr learned my addr 172.21.15.98:0/719235707 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefb7fe640 1 --2- 172.21.15.98:0/719235707 >> v2:172.21.15.70:3300/0 conn(0x7fdefc15f830 0x7fdefc099d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefbfff640 1 -- 172.21.15.98:0/719235707 >> v2:172.21.15.70:3300/0 conn(0x7fdefc15f830 msgr2=0x7fdefc099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefbfff640 1 --2- 172.21.15.98:0/719235707 >> v2:172.21.15.70:3300/0 conn(0x7fdefc15f830 0x7fdefc099d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefbfff640 1 -- 172.21.15.98:0/719235707 >> v1:172.21.15.98:6789/0 conn(0x7fdefc0b2b90 legacy=0x7fdefc0cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefbfff640 1 -- 172.21.15.98:0/719235707 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc09aea0 con 0x7fdefc15f450 2024-04-22T01:14:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefaffd640 1 -- 172.21.15.98:0/719235707 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1636576757 0 0) 0x7fdefc09a920 con 0x7fdefc0b2b90 2024-04-22T01:14:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefbfff640 1 --2- 172.21.15.98:0/719235707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0cf9b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdefc073450 tx=0x7fdeec01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87ce8d3ff3feeb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefaffd640 1 -- 172.21.15.98:0/719235707 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0ab680 con 0x7fdefc15f450 2024-04-22T01:14:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefaffd640 1 -- 172.21.15.98:0/719235707 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeec0b0070 con 0x7fdefc15f450 2024-04-22T01:14:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdefaffd640 1 -- 172.21.15.98:0/719235707 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0aba40 con 0x7fdefc15f450 2024-04-22T01:14:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 -- 172.21.15.98:0/719235707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 msgr2=0x7fdefc0cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 --2- 172.21.15.98:0/719235707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0cf9b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdefc073450 tx=0x7fdeec01c480 comp rx=0 tx=0).stop 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 -- 172.21.15.98:0/719235707 shutdown_connections 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 --2- 172.21.15.98:0/719235707 >> v2:172.21.15.70:3300/0 conn(0x7fdefc15f830 0x7fdefc099d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 --2- 172.21.15.98:0/719235707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0cf9b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 -- 172.21.15.98:0/719235707 >> 172.21.15.98:0/719235707 conn(0x7fdefc0b36c0 msgr2=0x7fdefc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 -- 172.21.15.98:0/719235707 shutdown_connections 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.603+0000 7fdf023db640 1 -- 172.21.15.98:0/719235707 wait complete. 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 Processor -- start 2024-04-22T01:14:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- start start 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdefc0b2b90 0x7fdefc154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdefc15f830 0x7fdefc15c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc0cd690 con 0x7fdefc15f830 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc09a680 con 0x7fdefc15f450 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdefc168f20 con 0x7fdefc0b2b90 2024-04-22T01:14:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdefc0b2b90 0x7fdefc154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54954/0 (socket says 172.21.15.98:54954) 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 -- 172.21.15.98:0/2837812928 learned_addr learned my addr 172.21.15.98:0/2837812928 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf00978640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdefc15f830 0x7fdefc15c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 -- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdefc0b2b90 msgr2=0x7fdefc154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdefc0b2b90 0x7fdefc154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 -- 172.21.15.98:0/2837812928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdefc15f830 msgr2=0x7fdefc15c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdefc15f830 0x7fdefc15c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 -- 172.21.15.98:0/2837812928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc15b580 con 0x7fdefc15f450 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf00978640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdefc15f830 0x7fdefc15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdefb7fe640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0c50a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdef000d800 tx=0x7fdef000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef000b6c0 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0019070 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc15b820 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef000ba60 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc15bde0 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdef000bc00 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fdef00149e0 con 0x7fdefc15f450 2024-04-22T01:14:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.607+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdefc068920 con 0x7fdefc15f450 2024-04-22T01:14:23.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.611+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdef0014d20 con 0x7fdefc15f450 2024-04-22T01:14:23.621 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.623+0000 7fb2046cf180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:14:23.665 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.665+0000 7f84bc87e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:14:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.675+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "310bb8df-5bcd-4966-9e02-98bcd61e596c", "id": 4} v 0) -- 0x7fdefc15ec60 con 0x7fdefc15f450 2024-04-22T01:14:23.675 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.675+0000 7fb2046cf180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:14:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdef8ff9640 1 -- 172.21.15.98:0/2837812928 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "310bb8df-5bcd-4966-9e02-98bcd61e596c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fdef00145a0 con 0x7fdefc15f450 2024-04-22T01:14:23.679 INFO:teuthology.orchestra.run.smithi098.stdout:4 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 msgr2=0x7fdefc0c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0c50a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdef000d800 tx=0x7fdef000dcc0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 shutdown_connections 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdefc15f830 0x7fdefc15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdefc15f450 0x7fdefc0c50a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 --2- 172.21.15.98:0/2837812928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdefc0b2b90 0x7fdefc154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 >> 172.21.15.98:0/2837812928 conn(0x7fdefc0b36c0 msgr2=0x7fdefc0cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 shutdown_connections 2024-04-22T01:14:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.679+0000 7fdf023db640 1 -- 172.21.15.98:0/2837812928 wait complete. 2024-04-22T01:14:23.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new d9619c39-d326-4500-a754-a2fbe3a2070f 5 2024-04-22T01:14:23.718 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.717+0000 7f84bc87e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:14:23.731 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.731+0000 7fb2046cf180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:14:23.794 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.793+0000 7f84bc87e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:14:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 Processor -- start 2024-04-22T01:14:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 -- start start 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f80b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83f80b5e30 0x7f83f80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f80b0530 con 0x7f83f80afd50 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f83f80b15b0 con 0x7f83f80b5e30 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fe971640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83f80afaf0 con 0x7f83f80b0750 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f80b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83fcf0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83f80b5e30 0x7f83f80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f80b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59462/0 (socket says 172.21.15.98:59462) 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83f7fff640 1 -- 172.21.15.98:0/3989437754 learned_addr learned my addr 172.21.15.98:0/3989437754 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.807+0000 7f83f77fe640 1 --1- 172.21.15.98:0/3989437754 >> v1:172.21.15.98:6789/0 conn(0x7f83f80afd50 0x7f83f806b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36140/0 (socket says 172.21.15.98:36140) 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fcf0e640 1 -- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 msgr2=0x7f83f80b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fcf0e640 1 --2- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f80b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fcf0e640 1 -- 172.21.15.98:0/3989437754 >> v1:172.21.15.98:6789/0 conn(0x7f83f80afd50 legacy=0x7f83f806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fcf0e640 1 -- 172.21.15.98:0/3989437754 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f80aed30 con 0x7f83f80b5e30 2024-04-22T01:14:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83f6ffd640 1 -- 172.21.15.98:0/3989437754 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936417323 0 0) 0x7f83f80b0530 con 0x7f83f80afd50 2024-04-22T01:14:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83f7fff640 1 --2- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f80b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:23.812 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.811+0000 7fb2046cf180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fcf0e640 1 --2- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f80b1070 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f83f80b0700 tx=0x7f83f001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9496e14829210db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83f6ffd640 1 -- 172.21.15.98:0/3989437754 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00b2070 con 0x7f83f80b5e30 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83f6ffd640 1 -- 172.21.15.98:0/3989437754 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f00a58f0 con 0x7f83f80b5e30 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83f6ffd640 1 -- 172.21.15.98:0/3989437754 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00a5cc0 con 0x7f83f80b5e30 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fe971640 1 -- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 msgr2=0x7f83f80b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.811+0000 7f83fe971640 1 --2- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f80b1070 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f83f80b0700 tx=0x7f83f001c830 comp rx=0 tx=0).stop 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/3989437754 shutdown_connections 2024-04-22T01:14:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 --2- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f80b1070 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 --2- 172.21.15.98:0/3989437754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f80b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/3989437754 >> 172.21.15.98:0/3989437754 conn(0x7f83f8099a60 msgr2=0x7f83f80b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/3989437754 shutdown_connections 2024-04-22T01:14:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/3989437754 wait complete. 2024-04-22T01:14:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 Processor -- start 2024-04-22T01:14:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- start start 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83f80afd50 0x7f83f813bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f81387b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f8159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83f80b0200 con 0x7f83f80afd50 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83f815b0b0 con 0x7f83f80b5e30 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83f8167890 con 0x7f83f80b0750 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f8159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f77fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f81387b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f8159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54966/0 (socket says 172.21.15.98:54966) 2024-04-22T01:14:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 -- 172.21.15.98:0/2574752275 learned_addr learned my addr 172.21.15.98:0/2574752275 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f7fff640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83f80afd50 0x7f83f813bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 -- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 msgr2=0x7f83f81387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f81387b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 -- 172.21.15.98:0/2574752275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83f80afd50 msgr2=0x7f83f813bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83f80afd50 0x7f83f813bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 -- 172.21.15.98:0/2574752275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f81538e0 con 0x7f83f80b5e30 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f77fe640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f81387b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f7fff640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83f80afd50 0x7f83f813bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fcf0e640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f8159880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83f00b0040 tx=0x7f83f00195d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00b2050 con 0x7f83f80b5e30 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f00be070 con 0x7f83f80b5e30 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f8153bc0 con 0x7f83f80b5e30 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00ada70 con 0x7f83f80b5e30 2024-04-22T01:14:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f8159360 con 0x7f83f80b5e30 2024-04-22T01:14:23.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f83f00adc10 con 0x7f83f80b5e30 2024-04-22T01:14:23.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f83f00c3c80 con 0x7f83f80b5e30 2024-04-22T01:14:23.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.815+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f8068920 con 0x7f83f80b5e30 2024-04-22T01:14:23.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.819+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f83f0016540 con 0x7f83f80b5e30 2024-04-22T01:14:23.847 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.849+0000 7f84bc87e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:14:23.866 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.867+0000 7fb2046cf180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:14:23.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.887+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d9619c39-d326-4500-a754-a2fbe3a2070f", "id": 5} v 0) -- 0x7f83f806ebb0 con 0x7f83f80b5e30 2024-04-22T01:14:23.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83f4ff9640 1 -- 172.21.15.98:0/2574752275 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9619c39-d326-4500-a754-a2fbe3a2070f", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f83f00a57f0 con 0x7f83f80b5e30 2024-04-22T01:14:23.891 INFO:teuthology.orchestra.run.smithi098.stdout:5 2024-04-22T01:14:23.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 msgr2=0x7f83f8159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:23.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f8159880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83f00b0040 tx=0x7f83f00195d0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 shutdown_connections 2024-04-22T01:14:23.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f80b5e30 0x7f83f8159880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83f80b0750 0x7f83f81387b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 --2- 172.21.15.98:0/2574752275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83f80afd50 0x7f83f813bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:23.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 >> 172.21.15.98:0/2574752275 conn(0x7f83f8099a60 msgr2=0x7f83f80b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:23.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 shutdown_connections 2024-04-22T01:14:23.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:23.891+0000 7f83fe971640 1 -- 172.21.15.98:0/2574752275 wait complete. 2024-04-22T01:14:23.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new e8096831-1495-4589-aefc-91205b3cb8c8 6 2024-04-22T01:14:23.945 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.943+0000 7fb2046cf180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:14:23.996 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:23.997+0000 7f84bc87e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:14:24.000 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:23.999+0000 7fb2046cf180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:14:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 Processor -- start 2024-04-22T01:14:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 -- start start 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e280b2750 0x7f8e280b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e2806c230 0x7f8e2806c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2815efe0 con 0x7f8e2815a4b0 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8e280b5530 con 0x7f8e2806c230 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e2dded640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8e280b2be0 con 0x7f8e280b2750 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.051+0000 7f8e26ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e280b2750 0x7f8e280b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e280b2750 0x7f8e280b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54978/0 (socket says 172.21.15.98:54978) 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 -- 172.21.15.98:0/2610062590 learned_addr learned my addr 172.21.15.98:0/2610062590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e27fff640 1 --2- 172.21.15.98:0/2610062590 >> v2:172.21.15.70:3300/0 conn(0x7f8e2806c230 0x7f8e2806c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 -- 172.21.15.98:0/2610062590 >> v2:172.21.15.70:3300/0 conn(0x7f8e2806c230 msgr2=0x7f8e2806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 --2- 172.21.15.98:0/2610062590 >> v2:172.21.15.70:3300/0 conn(0x7f8e2806c230 0x7f8e2806c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 -- 172.21.15.98:0/2610062590 >> v1:172.21.15.70:6789/0 conn(0x7f8e2815a4b0 legacy=0x7f8e2815e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 -- 172.21.15.98:0/2610062590 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e280b5760 con 0x7f8e280b2750 2024-04-22T01:14:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e26ffd640 1 --2- 172.21.15.98:0/2610062590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e280b4ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e28071820 tx=0x7f8e1c01c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b219454869c4e455 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e267fc640 1 -- 172.21.15.98:0/2610062590 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0b0070 con 0x7f8e280b2750 2024-04-22T01:14:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e267fc640 1 -- 172.21.15.98:0/2610062590 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e1c0ab600 con 0x7f8e280b2750 2024-04-22T01:14:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e267fc640 1 -- 172.21.15.98:0/2610062590 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0ab9d0 con 0x7f8e280b2750 2024-04-22T01:14:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 -- 172.21.15.98:0/2610062590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 msgr2=0x7f8e280b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 --2- 172.21.15.98:0/2610062590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e280b4ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e28071820 tx=0x7f8e1c01c270 comp rx=0 tx=0).stop 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 -- 172.21.15.98:0/2610062590 shutdown_connections 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 --2- 172.21.15.98:0/2610062590 >> v2:172.21.15.70:3300/0 conn(0x7f8e2806c230 0x7f8e2806c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 --2- 172.21.15.98:0/2610062590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e280b4ff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 -- 172.21.15.98:0/2610062590 >> 172.21.15.98:0/2610062590 conn(0x7f8e28099a60 msgr2=0x7f8e280b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 -- 172.21.15.98:0/2610062590 shutdown_connections 2024-04-22T01:14:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.055+0000 7f8e2dded640 1 -- 172.21.15.98:0/2610062590 wait complete. 2024-04-22T01:14:24.058 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:24.057+0000 7f84bc87e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 Processor -- start 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- start start 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e2806c230 0x7f8e281557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e281582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e2815a4b0 0x7f8e28158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2815aa10 con 0x7f8e2806c230 2024-04-22T01:14:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2815a890 con 0x7f8e280b2750 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e280cca60 con 0x7f8e2815a4b0 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e281582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e2815a4b0 0x7f8e28158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e281582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54990/0 (socket says 172.21.15.98:54990) 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 -- 172.21.15.98:0/4245394051 learned_addr learned my addr 172.21.15.98:0/4245394051 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e277fe640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e2806c230 0x7f8e281557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 -- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e2815a4b0 msgr2=0x7f8e28158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e2815a4b0 0x7f8e28158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 -- 172.21.15.98:0/4245394051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e2806c230 msgr2=0x7f8e281557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e2806c230 0x7f8e281557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 -- 172.21.15.98:0/4245394051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2813d730 con 0x7f8e280b2750 2024-04-22T01:14:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e26ffd640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e281582c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c019ea0 tx=0x7f8e1c019010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0b9070 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e1c0b4400 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2813da10 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0b0040 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e2813b3c0 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e1c0bc070 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e28068920 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.059+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f8e1c0bd9a0 con 0x7f8e280b2750 2024-04-22T01:14:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.063+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8e2815efe0 con 0x7f8e280b2750 2024-04-22T01:14:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.127+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e8096831-1495-4589-aefc-91205b3cb8c8", "id": 6} v 0) -- 0x7f8e280af650 con 0x7f8e280b2750 2024-04-22T01:14:24.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e0ffff640 1 -- 172.21.15.98:0/4245394051 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8096831-1495-4589-aefc-91205b3cb8c8", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f8e1c0bdbf0 con 0x7f8e280b2750 2024-04-22T01:14:24.135 INFO:teuthology.orchestra.run.smithi098.stdout:6 2024-04-22T01:14:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 msgr2=0x7f8e281582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e281582c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c019ea0 tx=0x7f8e1c019010 comp rx=0 tx=0).stop 2024-04-22T01:14:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 shutdown_connections 2024-04-22T01:14:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e2815a4b0 0x7f8e28158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e280b2750 0x7f8e281582c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 --2- 172.21.15.98:0/4245394051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e2806c230 0x7f8e281557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 >> 172.21.15.98:0/4245394051 conn(0x7f8e28099a60 msgr2=0x7f8e280ae380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:24.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 shutdown_connections 2024-04-22T01:14:24.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.135+0000 7f8e2dded640 1 -- 172.21.15.98:0/4245394051 wait complete. 2024-04-22T01:14:24.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 620872a8-a49b-46ea-853a-b83de44daa0c 7 2024-04-22T01:14:24.157 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:24.159+0000 7fb2046cf180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:14:24.221 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:24.219+0000 7fb2046cf180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:14:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 Processor -- start 2024-04-22T01:14:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 -- start start 2024-04-22T01:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e5015bef0 0x7f8e5009a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e500cfd20 0x7f8e500d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e500d0ba0 con 0x7f8e500d0780 2024-04-22T01:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8e5009acf0 con 0x7f8e500cfd20 2024-04-22T01:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e56f88640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8e5009afc0 con 0x7f8e5015bef0 2024-04-22T01:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e54d24640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e5015bef0 0x7f8e5009a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e55525640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8e500d0780 0x7f8e500cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36142/0 (socket says 172.21.15.98:36142) 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.259+0000 7f8e55525640 1 -- 172.21.15.98:0/3806987971 learned_addr learned my addr 172.21.15.98:0/3806987971 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e4ffff640 1 --2- 172.21.15.98:0/3806987971 >> v2:172.21.15.70:3300/0 conn(0x7f8e500cfd20 0x7f8e500d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e54d24640 1 -- 172.21.15.98:0/3806987971 >> v2:172.21.15.70:3300/0 conn(0x7f8e500cfd20 msgr2=0x7f8e500d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e54d24640 1 --2- 172.21.15.98:0/3806987971 >> v2:172.21.15.70:3300/0 conn(0x7f8e500cfd20 0x7f8e500d0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e54d24640 1 -- 172.21.15.98:0/3806987971 >> v1:172.21.15.98:6789/0 conn(0x7f8e500d0780 legacy=0x7f8e500cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e54d24640 1 -- 172.21.15.98:0/3806987971 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5009b2a0 con 0x7f8e5015bef0 2024-04-22T01:14:24.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e54d24640 1 --2- 172.21.15.98:0/3806987971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e5015bef0 0x7f8e5009a7b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e500727f0 tx=0x7f8e3801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca21043952ea067 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:24.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e4f7fe640 1 -- 172.21.15.98:0/3806987971 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e380a7070 con 0x7f8e5015bef0 2024-04-22T01:14:24.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e4f7fe640 1 -- 172.21.15.98:0/3806987971 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e380b0070 con 0x7f8e5015bef0 2024-04-22T01:14:24.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e4f7fe640 1 -- 172.21.15.98:0/3806987971 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e380ab890 con 0x7f8e5015bef0 2024-04-22T01:14:24.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 -- 172.21.15.98:0/3806987971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e5015bef0 msgr2=0x7f8e5009a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 --2- 172.21.15.98:0/3806987971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e5015bef0 0x7f8e5009a7b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e500727f0 tx=0x7f8e3801c450 comp rx=0 tx=0).stop 2024-04-22T01:14:24.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 -- 172.21.15.98:0/3806987971 shutdown_connections 2024-04-22T01:14:24.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 --2- 172.21.15.98:0/3806987971 >> v2:172.21.15.70:3300/0 conn(0x7f8e500cfd20 0x7f8e500d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 --2- 172.21.15.98:0/3806987971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e5015bef0 0x7f8e5009a7b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 -- 172.21.15.98:0/3806987971 >> 172.21.15.98:0/3806987971 conn(0x7f8e500b3770 msgr2=0x7f8e500b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 -- 172.21.15.98:0/3806987971 shutdown_connections 2024-04-22T01:14:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.263+0000 7f8e56f88640 1 -- 172.21.15.98:0/3806987971 wait complete. 2024-04-22T01:14:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 Processor -- start 2024-04-22T01:14:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- start start 2024-04-22T01:14:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e500cfd20 0x7f8e50153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e500d0780 0x7f8e501663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e5015bef0 0x7f8e500ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e50168ab0 con 0x7f8e500d0780 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5015c2d0 con 0x7f8e500cfd20 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e50168c30 con 0x7f8e5015bef0 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e54d24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e500cfd20 0x7f8e50153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e5015bef0 0x7f8e500ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e54d24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e500cfd20 0x7f8e50153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55012/0 (socket says 172.21.15.98:55012) 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e54d24640 1 -- 172.21.15.98:0/2856981444 learned_addr learned my addr 172.21.15.98:0/2856981444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e4ffff640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e500d0780 0x7f8e501663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 -- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e500cfd20 msgr2=0x7f8e50153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e500cfd20 0x7f8e50153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 -- 172.21.15.98:0/2856981444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e500d0780 msgr2=0x7f8e501663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e500d0780 0x7f8e501663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 -- 172.21.15.98:0/2856981444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e50159ff0 con 0x7f8e5015bef0 2024-04-22T01:14:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e55525640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e5015bef0 0x7f8e500ceab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800cad0 tx=0x7f8e4800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48017070 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e4800ac30 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5015a2d0 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48012400 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5015a890 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e4801a070 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f8e4801cd70 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.267+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e50068920 con 0x7f8e5015bef0 2024-04-22T01:14:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.271+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8e4801c710 con 0x7f8e5015bef0 2024-04-22T01:14:24.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.339+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "620872a8-a49b-46ea-853a-b83de44daa0c", "id": 7} v 0) -- 0x7f8e500ce590 con 0x7f8e5015bef0 2024-04-22T01:14:24.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e4d7fa640 1 -- 172.21.15.98:0/2856981444 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "620872a8-a49b-46ea-853a-b83de44daa0c", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f8e4801c9c0 con 0x7f8e5015bef0 2024-04-22T01:14:24.346 INFO:teuthology.orchestra.run.smithi098.stdout:7 2024-04-22T01:14:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e5015bef0 msgr2=0x7f8e500ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e5015bef0 0x7f8e500ceab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800cad0 tx=0x7f8e4800cf90 comp rx=0 tx=0).stop 2024-04-22T01:14:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 shutdown_connections 2024-04-22T01:14:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e5015bef0 0x7f8e500ceab0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e500d0780 0x7f8e501663b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 --2- 172.21.15.98:0/2856981444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e500cfd20 0x7f8e50153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 >> 172.21.15.98:0/2856981444 conn(0x7f8e500b3770 msgr2=0x7f8e500aef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:24.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 shutdown_connections 2024-04-22T01:14:24.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:14:24.347+0000 7f8e56f88640 1 -- 172.21.15.98:0/2856981444 wait complete. 2024-04-22T01:14:24.361 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:14:24.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-22T01:14:24.364 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:14:24.364 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T01:14:24.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-22T01:14:24.367 INFO:tasks.ceph.osd.1:Started 2024-04-22T01:14:24.367 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T01:14:24.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-22T01:14:24.369 INFO:tasks.ceph.osd.2:Started 2024-04-22T01:14:24.369 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T01:14:24.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-22T01:14:24.372 INFO:tasks.ceph.osd.3:Started 2024-04-22T01:14:24.372 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-22T01:14:24.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-22T01:14:24.406 INFO:tasks.ceph.osd.4:Started 2024-04-22T01:14:24.407 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-22T01:14:24.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-22T01:14:24.409 INFO:tasks.ceph.osd.5:Started 2024-04-22T01:14:24.409 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-22T01:14:24.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-22T01:14:24.410 INFO:tasks.ceph.osd.6:Started 2024-04-22T01:14:24.411 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-22T01:14:24.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-22T01:14:24.415 INFO:tasks.ceph.osd.7:Started 2024-04-22T01:14:24.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f85a6e15640 1 Processor -- start 2024-04-22T01:14:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f85a6e15640 1 -- start start 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f85a6e15640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f85a00b55f0 0x7f85a00b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f85a6e15640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a00b3dc0 con 0x7f85a015c3b0 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f85a6e15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a00b40a0 con 0x7f85a01445b0 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f85a6e15640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f85a00b4e60 con 0x7f85a00b55f0 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f8597fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f85a00b55f0 0x7f85a00b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f8597fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f85a00b55f0 0x7f85a00b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35862/0 (socket says 172.21.15.70:35862) 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f8597fff640 1 -- 172.21.15.70:0/1359663746 learned_addr learned my addr 172.21.15.70:0/1359663746 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f8597fff640 1 -- 172.21.15.70:0/1359663746 >> v1:172.21.15.98:6789/0 conn(0x7f85a01445b0 legacy=0x7f85a015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f8597fff640 1 -- 172.21.15.70:0/1359663746 >> v1:172.21.15.70:6789/0 conn(0x7f85a015c3b0 legacy=0x7f85a015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.561+0000 7f8597fff640 1 -- 172.21.15.70:0/1359663746 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a00b4380 con 0x7f85a00b55f0 2024-04-22T01:14:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f8597fff640 1 --2- 172.21.15.70:0/1359663746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a00b5a60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f859000bb10 tx=0x7f859001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27fcf49800852a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85977fe640 1 -- 172.21.15.70:0/1359663746 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85900b4070 con 0x7f85a00b55f0 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85977fe640 1 -- 172.21.15.70:0/1359663746 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85900a7b10 con 0x7f85a00b55f0 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85977fe640 1 -- 172.21.15.70:0/1359663746 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85900af440 con 0x7f85a00b55f0 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 -- 172.21.15.70:0/1359663746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 msgr2=0x7f85a00b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 --2- 172.21.15.70:0/1359663746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a00b5a60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f859000bb10 tx=0x7f859001ec30 comp rx=0 tx=0).stop 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 -- 172.21.15.70:0/1359663746 shutdown_connections 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 --2- 172.21.15.70:0/1359663746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a00b5a60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 -- 172.21.15.70:0/1359663746 >> 172.21.15.70:0/1359663746 conn(0x7f85a0072e50 msgr2=0x7f85a0073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 -- 172.21.15.70:0/1359663746 shutdown_connections 2024-04-22T01:14:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 -- 172.21.15.70:0/1359663746 wait complete. 2024-04-22T01:14:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 Processor -- start 2024-04-22T01:14:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 -- start start 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.565+0000 7f85a6e15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a0150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85a01445b0 0x7f85a0159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a015c3b0 0x7f85a015a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85a006f4a0 con 0x7f85a00b55f0 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85a00b5fa0 con 0x7f85a015c3b0 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f85a006f620 con 0x7f85a01445b0 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a0150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a0150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35876/0 (socket says 172.21.15.70:35876) 2024-04-22T01:14:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 -- 172.21.15.70:0/2876777445 learned_addr learned my addr 172.21.15.70:0/2876777445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a53b2640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a015c3b0 0x7f85a015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f8597fff640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85a01445b0 0x7f85a0159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 -- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85a01445b0 msgr2=0x7f85a0159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85a01445b0 0x7f85a0159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 -- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a015c3b0 msgr2=0x7f85a015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a015c3b0 0x7f85a015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 -- 172.21.15.70:0/2876777445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a01f7ad0 con 0x7f85a00b55f0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a4bb1640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a0150f40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f859c00c500 tx=0x7f859c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c016020 con 0x7f85a00b55f0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f859c005220 con 0x7f85a00b55f0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a01f7db0 con 0x7f85a00b55f0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c007030 con 0x7f85a00b55f0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a01f8370 con 0x7f85a00b55f0 2024-04-22T01:14:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f859c0067e0 con 0x7f85a00b55f0 2024-04-22T01:14:24.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f859c007580 con 0x7f85a00b55f0 2024-04-22T01:14:24.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a0068210 con 0x7f85a00b55f0 2024-04-22T01:14:24.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.569+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f859c020020 con 0x7f85a00b55f0 2024-04-22T01:14:24.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85a006de40 con 0x7f85a00b55f0 2024-04-22T01:14:24.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85957fa640 1 -- 172.21.15.70:0/2876777445 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f859c012d20 con 0x7f85a00b55f0 2024-04-22T01:14:24.628 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:24.628 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":10,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:24.343579+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 msgr2=0x7f85a0150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a0150f40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f859c00c500 tx=0x7f859c00c9c0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 shutdown_connections 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a015c3b0 0x7f85a015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85a01445b0 0x7f85a0159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 --2- 172.21.15.70:0/2876777445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a00b55f0 0x7f85a0150f40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 >> 172.21.15.70:0/2876777445 conn(0x7f85a0072e50 msgr2=0x7f85a00cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 shutdown_connections 2024-04-22T01:14:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:24.629+0000 7f85a6e15640 1 -- 172.21.15.70:0/2876777445 wait complete. 2024-04-22T01:14:24.643 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T01:14:24.643 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T01:14:24.719 INFO:tasks.ceph.mgr.y.smithi070.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-22T01:14:24.719 INFO:tasks.ceph.mgr.y.smithi070.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-22T01:14:24.719 INFO:tasks.ceph.mgr.y.smithi070.stderr: from numpy import show_config as show_numpy_config 2024-04-22T01:14:24.760 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:24.761+0000 7f84bc87e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:14:24.883 INFO:tasks.ceph.mgr.x.smithi098.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-22T01:14:24.883 INFO:tasks.ceph.mgr.x.smithi098.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-22T01:14:24.883 INFO:tasks.ceph.mgr.x.smithi098.stderr: from numpy import show_config as show_numpy_config 2024-04-22T01:14:24.891 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:24.889+0000 7f84bc87e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:14:24.958 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:24.959+0000 7fb2046cf180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:14:25.004 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-22T01:14:25.005+0000 7f68128577c0 -1 Falling back to public interface 2024-04-22T01:14:25.032 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-22T01:14:25.033+0000 7f14852397c0 -1 Falling back to public interface 2024-04-22T01:14:25.032 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-22T01:14:25.033+0000 7fddd282d7c0 -1 Falling back to public interface 2024-04-22T01:14:25.034 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-04-22T01:14:25.035+0000 7f9dfef5f7c0 -1 Falling back to public interface 2024-04-22T01:14:25.035 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-04-22T01:14:25.035+0000 7f824d4d17c0 -1 Falling back to public interface 2024-04-22T01:14:25.036 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-04-22T01:14:25.035+0000 7f9dfdf1f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6810/0: (98) Address already in use 2024-04-22T01:14:25.038 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:25.037+0000 7f84bc87e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:14:25.038 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-04-22T01:14:25.039+0000 7f522c41c7c0 -1 Falling back to public interface 2024-04-22T01:14:25.042 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-04-22T01:14:25.043+0000 7fabea49b7c0 -1 Falling back to public interface 2024-04-22T01:14:25.059 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-22T01:14:25.061+0000 7f0c5b6257c0 -1 Falling back to public interface 2024-04-22T01:14:25.102 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:25.101+0000 7f84bc87e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:14:25.146 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:25.147+0000 7fb2046cf180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:14:25.234 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:25.233+0000 7f84bc87e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:14:25.315 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:25.315+0000 7fb2046cf180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:14:25.386 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:25.387+0000 7fb2046cf180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:14:25.447 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:25.532 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:25.531+0000 7fb2046cf180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:14:25.550 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.549+0000 7f4121b09640 1 Processor -- start 2024-04-22T01:14:25.551 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.549+0000 7f4121b09640 1 -- start start 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4121b09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4121b09640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f411c0cdb40 0x7f411c15ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4121b09640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411c15c360 con 0x7f411c0cd6c0 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4121b09640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f411c0cdfd0 con 0x7f411c0cdb40 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4121b09640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f411c15cc50 con 0x7f411c06fb00 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f411c0cdb40 0x7f411c15ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4113fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f411c0cd6c0 0x7f411c15f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46872/0 (socket says 172.21.15.70:46872) 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f4113fff640 1 -- 172.21.15.70:0/2326730911 learned_addr learned my addr 172.21.15.70:0/2326730911 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411b7fe640 1 --2- 172.21.15.70:0/2326730911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:25.553 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411bfff640 1 -- 172.21.15.70:0/2326730911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 msgr2=0x7f411c06fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:25.554 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411bfff640 1 --2- 172.21.15.70:0/2326730911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c06fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.554 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411bfff640 1 -- 172.21.15.70:0/2326730911 >> v1:172.21.15.70:6789/0 conn(0x7f411c0cd6c0 legacy=0x7f411c15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:25.554 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411affd640 1 -- 172.21.15.70:0/2326730911 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106727570 0 0) 0x7f411c15c360 con 0x7f411c0cd6c0 2024-04-22T01:14:25.554 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411bfff640 1 -- 172.21.15.70:0/2326730911 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c070420 con 0x7f411c0cdb40 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411bfff640 1 --2- 172.21.15.70:0/2326730911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cdb40 0x7f411c15ff50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4114009d90 tx=0x7f411401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06a78517a5c868 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411affd640 1 -- 172.21.15.70:0/2326730911 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41140b2070 con 0x7f411c0cdb40 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411affd640 1 -- 172.21.15.70:0/2326730911 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41140a5920 con 0x7f411c0cdb40 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.553+0000 7f411affd640 1 -- 172.21.15.70:0/2326730911 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41140a5cf0 con 0x7f411c0cdb40 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/2326730911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cdb40 msgr2=0x7f411c15ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 --2- 172.21.15.70:0/2326730911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cdb40 0x7f411c15ff50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4114009d90 tx=0x7f411401c970 comp rx=0 tx=0).stop 2024-04-22T01:14:25.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/2326730911 shutdown_connections 2024-04-22T01:14:25.556 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 --2- 172.21.15.70:0/2326730911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cdb40 0x7f411c15ff50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.556 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 --2- 172.21.15.70:0/2326730911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c06fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.556 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/2326730911 >> 172.21.15.70:0/2326730911 conn(0x7f411c06e5c0 msgr2=0x7f411c06e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:25.556 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/2326730911 shutdown_connections 2024-04-22T01:14:25.556 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/2326730911 wait complete. 2024-04-22T01:14:25.558 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 Processor -- start 2024-04-22T01:14:25.558 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- start start 2024-04-22T01:14:25.558 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c0b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 0x7f411c140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f411c0cdb40 0x7f411c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0ce370 con 0x7f411c0cd6c0 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0ce1f0 con 0x7f411c0cdb40 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f411c15d0c0 con 0x7f411c06fb00 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 0x7f411c140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 0x7f411c140ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35908/0 (socket says 172.21.15.70:35908) 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 -- 172.21.15.70:0/763730309 learned_addr learned my addr 172.21.15.70:0/763730309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f411b7fe640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c0b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f411bfff640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f411c0cdb40 0x7f411c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 -- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 msgr2=0x7f411c0b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c0b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 -- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f411c0cdb40 msgr2=0x7f411c15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f411c0cdb40 0x7f411c15a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 -- 172.21.15.70:0/763730309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c06b340 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4113fff640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 0x7f411c140ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41040011f0 tx=0x7f410400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104018070 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41040074e0 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104004a10 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411c06b5e0 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411c06bba0 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f410401b070 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4104012ad0 con 0x7f411c0cd6c0 2024-04-22T01:14:25.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.557+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f411c04f510 con 0x7f411c0cd6c0 2024-04-22T01:14:25.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.561+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f410401d440 con 0x7f411c0cd6c0 2024-04-22T01:14:25.617 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.617+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f411c072670 con 0x7f411c0cd6c0 2024-04-22T01:14:25.618 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.617+0000 7f4118ff9640 1 -- 172.21.15.70:0/763730309 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4104004bb0 con 0x7f411c0cd6c0 2024-04-22T01:14:25.618 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-22T01:14:25.618 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":10,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:24.343579+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 msgr2=0x7f411c140ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 0x7f411c140ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41040011f0 tx=0x7f410400c9b0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 shutdown_connections 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f411c0cdb40 0x7f411c15a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f411c0cd6c0 0x7f411c140ae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 --2- 172.21.15.70:0/763730309 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f411c06fb00 0x7f411c0b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 >> 172.21.15.70:0/763730309 conn(0x7f411c06e5c0 msgr2=0x7f411c072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 shutdown_connections 2024-04-22T01:14:25.620 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:25.621+0000 7f4121b09640 1 -- 172.21.15.70:0/763730309 wait complete. 2024-04-22T01:14:25.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-22T01:14:25.867 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:25.869+0000 7f84bc87e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:14:25.941 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:25.941+0000 7f84bc87e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:14:26.123 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:26.125+0000 7f84bc87e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:14:26.182 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:26.183+0000 7fb2046cf180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:14:26.196 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:26.197+0000 7f84bc87e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:14:26.257 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:26.255+0000 7fb2046cf180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:14:26.438 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:26.439+0000 7fb2046cf180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:14:26.486 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:26.485+0000 7f84bc87e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:14:26.511 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:26.511+0000 7fb2046cf180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:14:26.547 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:26.549+0000 7f84bc87e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:14:26.807 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:26.807+0000 7fb2046cf180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:14:26.868 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-04-22T01:14:26.867+0000 7fb2046cf180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:14:27.591 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:27.593+0000 7f8463cc9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-22T01:14:27.592 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-22T01:14:27.593+0000 7f8463cc9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-22T01:14:28.762 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-04-22T01:14:28.763+0000 7f824d4d17c0 -1 osd.4 0 log_to_monitors true 2024-04-22T01:14:28.840 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-22T01:14:28.841+0000 7fddd282d7c0 -1 osd.3 0 log_to_monitors true 2024-04-22T01:14:28.852 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-04-22T01:14:28.851+0000 7f522c41c7c0 -1 osd.5 0 log_to_monitors true 2024-04-22T01:14:28.857 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-22T01:14:28.857+0000 7f68128577c0 -1 osd.1 0 log_to_monitors true 2024-04-22T01:14:28.932 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-04-22T01:14:28.931+0000 7fabea49b7c0 -1 osd.7 0 log_to_monitors true 2024-04-22T01:14:28.991 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-22T01:14:28.993+0000 7f14852397c0 -1 osd.0 0 log_to_monitors true 2024-04-22T01:14:28.995 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-22T01:14:28.997+0000 7f0c5b6257c0 -1 osd.2 0 log_to_monitors true 2024-04-22T01:14:28.996 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-04-22T01:14:28.995+0000 7f9dfef5f7c0 -1 osd.6 0 log_to_monitors true 2024-04-22T01:14:32.120 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-22T01:14:32.121+0000 7f680e800640 -1 osd.1 0 waiting for initial osdmap 2024-04-22T01:14:32.148 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-22T01:14:32.149+0000 7f6806635640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.175 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-22T01:14:32.177+0000 7f14819f4640 -1 osd.0 0 waiting for initial osdmap 2024-04-22T01:14:32.215 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-22T01:14:32.213+0000 7f0c575ce640 -1 osd.2 0 waiting for initial osdmap 2024-04-22T01:14:32.219 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-22T01:14:32.221+0000 7f1479017640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.227 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-22T01:14:32.229+0000 7f0c4f403640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.250 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-22T01:14:32.249+0000 7fddce7d6640 -1 osd.3 0 waiting for initial osdmap 2024-04-22T01:14:32.275 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-22T01:14:32.277+0000 7fddc660b640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.441 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:32.555 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 Processor -- start 2024-04-22T01:14:32.556 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 -- start start 2024-04-22T01:14:32.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21d4145730 0x7f21d40b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 0x7f21d40b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21d4145b10 con 0x7f21d40b0b20 2024-04-22T01:14:32.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f21d40b2ff0 con 0x7f21d4145730 2024-04-22T01:14:32.559 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21dbd2f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f21d40b23e0 con 0x7f21d415d500 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f21d40b0b20 0x7f21d406d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35668/0 (socket says 172.21.15.70:35668) 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 -- 172.21.15.70:0/739287353 learned_addr learned my addr 172.21.15.70:0/739287353 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d9acb640 1 --2- 172.21.15.70:0/739287353 >> v2:172.21.15.98:3300/0 conn(0x7f21d4145730 0x7f21d40b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21da2cc640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 0x7f21d40b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810391262 0 0) 0x7f21d4145b10 con 0x7f21d40b0b20 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21c0003530 con 0x7f21d40b0b20 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1193338402 0 0) 0x7f21c0003530 con 0x7f21d40b0b20 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 msgr2=0x7f21d40b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 0x7f21d40b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 >> v2:172.21.15.98:3300/0 conn(0x7f21d4145730 msgr2=0x7f21d40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 --2- 172.21.15.70:0/739287353 >> v2:172.21.15.98:3300/0 conn(0x7f21d4145730 0x7f21d40b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.560 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d40b26c0 con 0x7f21d40b0b20 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f21c4001680 con 0x7f21d40b0b20 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d9acb640 1 --2- 172.21.15.70:0/739287353 >> v2:172.21.15.98:3300/0 conn(0x7f21d4145730 0x7f21d40b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 >> v1:172.21.15.70:6789/0 conn(0x7f21d40b0b20 legacy=0x7f21d406d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 0x7f21c00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c0003f40 0x7f21c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c0004d50 0x7f21c00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21d40b2ff0 con 0x7f21c0003f40 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21d40b23e0 con 0x7f21c0004d50 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f21c000ba20 con 0x7f21d415d500 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c0003f40 0x7f21c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.561 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d9acb640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 0x7f21c00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.562 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21da2cc640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c0004d50 0x7f21c00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.562 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 -- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 msgr2=0x7f21c00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:32.562 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d415d500 0x7f21c00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.562 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 -- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c0004d50 msgr2=0x7f21c00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:32.562 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c0004d50 0x7f21c00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.562 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.556+0000 7f21d92ca640 1 -- 172.21.15.70:0/739287353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c0003530 con 0x7f21c0003f40 2024-04-22T01:14:32.566 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c0003f40 0x7f21c0004810 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f21c400c270 tx=0x7f21c401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd2e92df84c5cad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:32.566 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4007d60 con 0x7f21c0003f40 2024-04-22T01:14:32.566 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c40a9020 con 0x7f21c0003f40 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d8ac9640 1 -- 172.21.15.70:0/739287353 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4005ae0 con 0x7f21c0003f40 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- 172.21.15.70:0/739287353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c0003f40 msgr2=0x7f21c0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/739287353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c0003f40 0x7f21c0004810 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f21c400c270 tx=0x7f21c401ccc0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21da2cc640 1 -- 172.21.15.70:0/739287353 reap_dead start 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- 172.21.15.70:0/739287353 shutdown_connections 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- 172.21.15.70:0/739287353 >> 172.21.15.70:0/739287353 conn(0x7f21d40b64d0 msgr2=0x7f21d4099ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- 172.21.15.70:0/739287353 shutdown_connections 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- 172.21.15.70:0/739287353 wait complete. 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 Processor -- start 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- start start 2024-04-22T01:14:32.567 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c0003f40 0x7f21d41f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 0x7f21d4135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d409e240 0x7f21d409e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21d40b2000 con 0x7f21d415b440 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21d40b1350 con 0x7f21d409e240 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21dbd2f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f21d40b2180 con 0x7f21c0003f40 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 0x7f21d4135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 0x7f21d4135610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46284/0 (socket says 172.21.15.70:46284) 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 -- 172.21.15.70:0/101076950 learned_addr learned my addr 172.21.15.70:0/101076950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21da2cc640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d409e240 0x7f21d409e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d9acb640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c0003f40 0x7f21d41f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.568 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 -- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c0003f40 msgr2=0x7f21d41f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c0003f40 0x7f21d41f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 -- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d409e240 msgr2=0x7f21d409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d409e240 0x7f21d409e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d92ca640 1 -- 172.21.15.70:0/101076950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d41f6300 con 0x7f21d415b440 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21da2cc640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d409e240 0x7f21d409e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.560+0000 7f21d9acb640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c0003f40 0x7f21d41f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21d92ca640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 0x7f21d4135610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21c400c270 tx=0x7f21c0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c40a5a90 con 0x7f21d415b440 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c40a5c30 con 0x7f21d415b440 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d41f5740 con 0x7f21d415b440 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c40047f0 con 0x7f21d415b440 2024-04-22T01:14:32.569 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d41f5d00 con 0x7f21d415b440 2024-04-22T01:14:32.570 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d404f510 con 0x7f21d415b440 2024-04-22T01:14:32.570 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 87389+0+0 (secure 0 0 0) 0x7f21c4004990 con 0x7f21d415b440 2024-04-22T01:14:32.570 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21ca7fc640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21b8067830 0x7f21b8069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:32.570 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21d9acb640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21b8067830 0x7f21b8069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:32.570 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f21c411eaf0 con 0x7f21d415b440 2024-04-22T01:14:32.570 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.564+0000 7f21d9acb640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21b8067830 0x7f21b8069cf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009d70 tx=0x7f21d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:32.655 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.652+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f21c40169d0 con 0x7f21d415b440 2024-04-22T01:14:32.794 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-04-22T01:14:32.794+0000 7f9dfaf08640 -1 osd.6 0 waiting for initial osdmap 2024-04-22T01:14:32.812 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.812+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21d406e400 con 0x7f21d415b440 2024-04-22T01:14:32.814 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.812+0000 7f21ca7fc640 1 -- 172.21.15.70:0/101076950 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f21c40f04a0 con 0x7f21d415b440 2024-04-22T01:14:32.814 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-22T01:14:32.814 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":12,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:30.597260+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:32.816 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21b8067830 msgr2=0x7f21b8069cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:32.816 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21b8067830 0x7f21b8069cf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009d70 tx=0x7f21d0008040 comp rx=0 tx=0).stop 2024-04-22T01:14:32.816 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 msgr2=0x7f21d4135610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:32.816 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 0x7f21d4135610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21c400c270 tx=0x7f21c0005840 comp rx=0 tx=0).stop 2024-04-22T01:14:32.816 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 shutdown_connections 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21b8067830 0x7f21b8069cf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d409e240 0x7f21d409e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21d415b440 0x7f21d4135610 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 --2- 172.21.15.70:0/101076950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c0003f40 0x7f21d41f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 >> 172.21.15.70:0/101076950 conn(0x7f21d40b64d0 msgr2=0x7f21d4144250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 shutdown_connections 2024-04-22T01:14:32.817 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:32.816+0000 7f21dbd2f640 1 -- 172.21.15.70:0/101076950 wait complete. 2024-04-22T01:14:32.822 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-04-22T01:14:32.822+0000 7f9df2d3d640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-22T01:14:32.832 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-04-22T01:14:32.830+0000 7f5228bd7640 -1 osd.5 0 waiting for initial osdmap 2024-04-22T01:14:32.858 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-04-22T01:14:32.858+0000 7f52201fa640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.872 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-04-22T01:14:32.870+0000 7f824947a640 -1 osd.4 0 waiting for initial osdmap 2024-04-22T01:14:32.902 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-04-22T01:14:32.902+0000 7f82412af640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:32.912 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-04-22T01:14:32.910+0000 7fabe6444640 -1 osd.7 0 waiting for initial osdmap 2024-04-22T01:14:32.934 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-04-22T01:14:32.934+0000 7fabde279640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:14:39.636 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:39.756 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff978f6d640 1 Processor -- start 2024-04-22T01:14:39.756 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff978f6d640 1 -- start start 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff978f6d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9740b58f0 0x7ff9740b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff978f6d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9740ce7a0 con 0x7ff9740b3e00 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff978f6d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9740cea80 con 0x7ff9740b34e0 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff978f6d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff9740ce560 con 0x7ff9740b58f0 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff9740b3e00 0x7ff97416be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43302/0 (socket says 172.21.15.70:43302) 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 -- 172.21.15.70:0/1826846386 learned_addr learned my addr 172.21.15.70:0/1826846386 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:39.759 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff971d9b640 1 --2- 172.21.15.70:0/1826846386 >> v2:172.21.15.98:3300/0 conn(0x7ff9740b58f0 0x7ff9740b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 874819077 0 0) 0x7ff9740ce7a0 con 0x7ff9740b3e00 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff95c003530 con 0x7ff9740b3e00 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1702523421 0 0) 0x7ff95c003530 con 0x7ff9740b3e00 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 >> v2:172.21.15.98:3300/0 conn(0x7ff9740b58f0 msgr2=0x7ff9740b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 --2- 172.21.15.70:0/1826846386 >> v2:172.21.15.98:3300/0 conn(0x7ff9740b58f0 0x7ff9740b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 >> v1:172.21.15.98:6789/0 conn(0x7ff9740b34e0 legacy=0x7ff9740cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9740ced60 con 0x7ff9740b3e00 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7ff96c001a00 con 0x7ff9740b3e00 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 >> v1:172.21.15.70:6789/0 conn(0x7ff9740b3e00 legacy=0x7ff97416be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c0039c0 0x7ff95c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff95c004810 0x7ff95c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff95c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9740ce560 con 0x7ff95c0051d0 2024-04-22T01:14:39.760 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff95c00c070 con 0x7ff95c0039c0 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff95c00c270 con 0x7ff95c004810 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff95c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff97259c640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c0039c0 0x7ff95c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff971d9b640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff95c004810 0x7ff95c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 -- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff95c004810 msgr2=0x7ff95c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff95c004810 0x7ff95c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 -- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c0039c0 msgr2=0x7ff95c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c0039c0 0x7ff95c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.761 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.756+0000 7ff972d9d640 1 -- 172.21.15.70:0/1826846386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95c003530 con 0x7ff95c0051d0 2024-04-22T01:14:39.762 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff972d9d640 1 -- 172.21.15.70:0/1826846386 reap_dead start 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff972d9d640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff95c00b9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00a840 tx=0x7ff96c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97573112dd534f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c00a460 con 0x7ff95c0051d0 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff96c0ac640 con 0x7ff95c0051d0 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff97159a640 1 -- 172.21.15.70:0/1826846386 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0aca30 con 0x7ff95c0051d0 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 -- 172.21.15.70:0/1826846386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 msgr2=0x7ff95c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff95c00b9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00a840 tx=0x7ff96c01d360 comp rx=0 tx=0).stop 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 -- 172.21.15.70:0/1826846386 shutdown_connections 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 --2- 172.21.15.70:0/1826846386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff95c00b9d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 -- 172.21.15.70:0/1826846386 >> 172.21.15.70:0/1826846386 conn(0x7ff974072aa0 msgr2=0x7ff97406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 -- 172.21.15.70:0/1826846386 shutdown_connections 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 -- 172.21.15.70:0/1826846386 wait complete. 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 Processor -- start 2024-04-22T01:14:39.763 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.760+0000 7ff978f6d640 1 -- start start 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff974146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97415f760 0x7ff9741f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9741f8fa0 0x7ff97414a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9740b4250 con 0x7ff95c0051d0 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9740cf320 con 0x7ff97415f760 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff9740b38c0 con 0x7ff9741f8fa0 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff974146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff974146700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44674/0 (socket says 172.21.15.70:44674) 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 -- 172.21.15.70:0/3828326609 learned_addr learned my addr 172.21.15.70:0/3828326609 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff971d9b640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97415f760 0x7ff9741f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff972d9d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9741f8fa0 0x7ff97414a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 -- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9741f8fa0 msgr2=0x7ff97414a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.766 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9741f8fa0 0x7ff97414a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 -- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97415f760 msgr2=0x7ff9741f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97415f760 0x7ff9741f7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 -- 172.21.15.70:0/3828326609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9741f8720 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff972d9d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9741f8fa0 0x7ff97414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff97259c640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff974146700 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff96800e4a0 tx=0x7ff96800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9680052e0 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff968018070 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9741f8a00 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff968003820 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff974160fb0 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9741613f0 con 0x7ff95c0051d0 2024-04-22T01:14:39.767 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff9680039c0 con 0x7ff95c0051d0 2024-04-22T01:14:39.768 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff962ffd640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff9540678c0 0x7ff954069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:39.768 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff968085420 con 0x7ff95c0051d0 2024-04-22T01:14:39.768 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff971d9b640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff9540678c0 0x7ff954069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:39.768 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.764+0000 7ff971d9b640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff9540678c0 0x7ff954069d80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9741fb330 tx=0x7ff964008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:39.771 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.768+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff968056340 con 0x7ff95c0051d0 2024-04-22T01:14:39.931 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.928+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff97415f540 con 0x7ff95c0051d0 2024-04-22T01:14:39.933 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff962ffd640 1 -- 172.21.15.70:0/3828326609 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7ff968005480 con 0x7ff95c0051d0 2024-04-22T01:14:39.933 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-22T01:14:39.933 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":16,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:36.814010+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:39.935 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff9540678c0 msgr2=0x7ff954069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.935 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff9540678c0 0x7ff954069d80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9741fb330 tx=0x7ff964008040 comp rx=0 tx=0).stop 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 msgr2=0x7ff974146700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff974146700 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff96800e4a0 tx=0x7ff96800e960 comp rx=0 tx=0).stop 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 shutdown_connections 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff9540678c0 0x7ff954069d80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9741f8fa0 0x7ff97414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97415f760 0x7ff9741f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 --2- 172.21.15.70:0/3828326609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff95c0051d0 0x7ff974146700 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 >> 172.21.15.70:0/3828326609 conn(0x7ff974072aa0 msgr2=0x7ff97406e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 shutdown_connections 2024-04-22T01:14:39.936 INFO:teuthology.misc.health.smithi070.stderr:2024-04-22T01:14:39.932+0000 7ff978f6d640 1 -- 172.21.15.70:0/3828326609 wait complete. 2024-04-22T01:14:39.950 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-22T01:14:39.950 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T01:14:39.950 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:14:39.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:14:39.952 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:14:39.952 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:14:39.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:14:39.955 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:14:39.955 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:14:39.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:14:39.958 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:14:39.958 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:14:39.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:14:39.961 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:14:39.961 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-22T01:14:39.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:40.024 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:14:40.024 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:14:40.052 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:14:40.052 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:14:40.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 Processor -- start 2024-04-22T01:14:40.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 -- start start 2024-04-22T01:14:40.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a6015ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a6015efb0 0x7f0a6006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a6006efc0 con 0x7f0a6006b730 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0a6006fbd0 con 0x7f0a6015efb0 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a66bc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a6006f2a0 con 0x7f0a6015e690 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a6015efb0 0x7f0a6006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a6015efb0 0x7f0a6006b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44728/0 (socket says 172.21.15.70:44728) 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a6515f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0a6006b730 0x7f0a600af7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43320/0 (socket says 172.21.15.70:43320) 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 -- 172.21.15.70:0/2935936867 learned_addr learned my addr 172.21.15.70:0/2935936867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a6495e640 1 --2- 172.21.15.70:0/2935936867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a6015ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 -- 172.21.15.70:0/2935936867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 msgr2=0x7f0a6015ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 --2- 172.21.15.70:0/2935936867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a6015ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 -- 172.21.15.70:0/2935936867 >> v1:172.21.15.70:6789/0 conn(0x7f0a6006b730 legacy=0x7f0a600af7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a577fe640 1 -- 172.21.15.70:0/2935936867 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917822010 0 0) 0x7f0a6006efc0 con 0x7f0a6006b730 2024-04-22T01:14:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 -- 172.21.15.70:0/2935936867 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6006f580 con 0x7f0a6015efb0 2024-04-22T01:14:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a57fff640 1 --2- 172.21.15.70:0/2935936867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a6006b1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009870 tx=0x7f0a5c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d0d2cce83e2e24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a577fe640 1 -- 172.21.15.70:0/2935936867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0b2070 con 0x7f0a6015efb0 2024-04-22T01:14:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.068+0000 7f0a577fe640 1 -- 172.21.15.70:0/2935936867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0a9070 con 0x7f0a6015efb0 2024-04-22T01:14:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a577fe640 1 -- 172.21.15.70:0/2935936867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0ad880 con 0x7f0a6015efb0 2024-04-22T01:14:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2935936867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 msgr2=0x7f0a6006b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2935936867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a6006b1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009870 tx=0x7f0a5c01c560 comp rx=0 tx=0).stop 2024-04-22T01:14:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2935936867 shutdown_connections 2024-04-22T01:14:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2935936867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a6006b1f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2935936867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a6015ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2935936867 >> 172.21.15.70:0/2935936867 conn(0x7f0a60099a60 msgr2=0x7f0a60099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2935936867 shutdown_connections 2024-04-22T01:14:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2935936867 wait complete. 2024-04-22T01:14:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 Processor -- start 2024-04-22T01:14:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- start start 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6006b730 0x7f0a60148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a600b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a60157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a600d0390 con 0x7f0a6015efb0 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6006bb80 con 0x7f0a6006b730 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a600d0510 con 0x7f0a6015e690 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a60157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a60157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44730/0 (socket says 172.21.15.70:44730) 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 -- 172.21.15.70:0/2291355285 learned_addr learned my addr 172.21.15.70:0/2291355285 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6495e640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6006b730 0x7f0a60148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a57fff640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a600b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 -- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 msgr2=0x7f0a600b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a600b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 -- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6006b730 msgr2=0x7f0a60148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6006b730 0x7f0a60148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 -- 172.21.15.70:0/2291355285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a60149290 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a6515f640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a60157ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800c7a0 tx=0x7f0a5800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58018070 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a5800a640 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a600b27c0 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5800aab0 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.072+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a600b2d80 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.076+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a60068920 con 0x7f0a6015efb0 2024-04-22T01:14:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.076+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a5801b070 con 0x7f0a6015efb0 2024-04-22T01:14:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.076+0000 7f0a557fa640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a4c0678c0 0x7f0a4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.076+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f0a58084ef0 con 0x7f0a6015efb0 2024-04-22T01:14:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.076+0000 7f0a6495e640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a4c0678c0 0x7f0a4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.076+0000 7f0a6495e640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a4c0678c0 0x7f0a4c069d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009b60 tx=0x7f0a50009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.080+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0a580556a0 con 0x7f0a6015efb0 2024-04-22T01:14:40.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a600c42b0 con 0x7f0a6015efb0 2024-04-22T01:14:40.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a557fa640 1 -- 172.21.15.70:0/2291355285 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7f0a580599a0 con 0x7f0a6015efb0 2024-04-22T01:14:40.237 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:40.237 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":16,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:36.814010+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a4c0678c0 msgr2=0x7f0a4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a4c0678c0 0x7f0a4c069d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009b60 tx=0x7f0a50009190 comp rx=0 tx=0).stop 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 msgr2=0x7f0a60157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a60157ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800c7a0 tx=0x7f0a5800cc60 comp rx=0 tx=0).stop 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 shutdown_connections 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a4c0678c0 0x7f0a4c069d80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a6015efb0 0x7f0a60157ad0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a6015e690 0x7f0a600b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 --2- 172.21.15.70:0/2291355285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a6006b730 0x7f0a60148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 >> 172.21.15.70:0/2291355285 conn(0x7f0a60099a60 msgr2=0x7f0a600ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:40.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 shutdown_connections 2024-04-22T01:14:40.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.236+0000 7f0a66bc2640 1 -- 172.21.15.70:0/2291355285 wait complete. 2024-04-22T01:14:40.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:14:40.254 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:14:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 Processor -- start 2024-04-22T01:14:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 -- start start 2024-04-22T01:14:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b7c15de20 0x7f8b7c0b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0b6db0 0x7f8b7c0d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b7c0cea10 0x7f8b7c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8b7c0cf3e0 con 0x7f8b7c0cea10 2024-04-22T01:14:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8b7c0b75e0 con 0x7f8b7c15de20 2024-04-22T01:14:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b84321640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7c0b78c0 con 0x7f8b7c0b6db0 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b7c0cea10 0x7f8b7c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b7c0cea10 0x7f8b7c0ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44738/0 (socket says 172.21.15.70:44738) 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 -- 172.21.15.70:0/2884813919 learned_addr learned my addr 172.21.15.70:0/2884813919 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b820bd640 1 --2- 172.21.15.70:0/2884813919 >> v2:172.21.15.98:3300/0 conn(0x7f8b7c15de20 0x7f8b7c0b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b818bc640 1 --2- 172.21.15.70:0/2884813919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0b6db0 0x7f8b7c0d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 -- 172.21.15.70:0/2884813919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0b6db0 msgr2=0x7f8b7c0d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 --2- 172.21.15.70:0/2884813919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0b6db0 0x7f8b7c0d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 -- 172.21.15.70:0/2884813919 >> v2:172.21.15.98:3300/0 conn(0x7f8b7c15de20 msgr2=0x7f8b7c0b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 --2- 172.21.15.70:0/2884813919 >> v2:172.21.15.98:3300/0 conn(0x7f8b7c15de20 0x7f8b7c0b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 -- 172.21.15.70:0/2884813919 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7c0b7ba0 con 0x7f8b7c0cea10 2024-04-22T01:14:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b820bd640 1 --2- 172.21.15.70:0/2884813919 >> v2:172.21.15.98:3300/0 conn(0x7f8b7c15de20 0x7f8b7c0b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:40.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.364+0000 7f8b828be640 1 --2- 172.21.15.70:0/2884813919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c0cea10 0x7f8b7c0ceea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009e50 tx=0x7f8b7401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6d44d9add2e8c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b810bb640 1 -- 172.21.15.70:0/2884813919 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b740b4070 con 0x7f8b7c0cea10 2024-04-22T01:14:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b810bb640 1 -- 172.21.15.70:0/2884813919 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b740af400 con 0x7f8b7c0cea10 2024-04-22T01:14:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b810bb640 1 -- 172.21.15.70:0/2884813919 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b740af7d0 con 0x7f8b7c0cea10 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2884813919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c0cea10 msgr2=0x7f8b7c0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- 172.21.15.70:0/2884813919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c0cea10 0x7f8b7c0ceea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009e50 tx=0x7f8b7401f1e0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2884813919 shutdown_connections 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- 172.21.15.70:0/2884813919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c0cea10 0x7f8b7c0ceea0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- 172.21.15.70:0/2884813919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0b6db0 0x7f8b7c0d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- 172.21.15.70:0/2884813919 >> v2:172.21.15.98:3300/0 conn(0x7f8b7c15de20 0x7f8b7c0b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2884813919 >> 172.21.15.70:0/2884813919 conn(0x7f8b7c072d20 msgr2=0x7f8b7c073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2884813919 shutdown_connections 2024-04-22T01:14:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2884813919 wait complete. 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 Processor -- start 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- start start 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7c0b6db0 0x7f8b7c158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0cea10 0x7f8b7c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 0x7f8b7c0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c15e200 con 0x7f8b7c15de20 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c0b8160 con 0x7f8b7c0b6db0 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7c0d22c0 con 0x7f8b7c0cea10 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 0x7f8b7c0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 0x7f8b7c0cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44750/0 (socket says 172.21.15.70:44750) 2024-04-22T01:14:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 -- 172.21.15.70:0/2605678705 learned_addr learned my addr 172.21.15.70:0/2605678705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b820bd640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7c0b6db0 0x7f8b7c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 -- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0cea10 msgr2=0x7f8b7c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b818bc640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0cea10 0x7f8b7c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0cea10 0x7f8b7c0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 -- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7c0b6db0 msgr2=0x7f8b7c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7c0b6db0 0x7f8b7c158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 -- 172.21.15.70:0/2605678705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7c0cc630 con 0x7f8b7c15de20 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b828be640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 0x7f8b7c0cd0b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8b7401d890 tx=0x7f8b7401bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b740b4040 con 0x7f8b7c15de20 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b740afdb0 con 0x7f8b7c15de20 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7c0c75e0 con 0x7f8b7c15de20 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74009510 con 0x7f8b7c15de20 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.368+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7c158e80 con 0x7f8b7c15de20 2024-04-22T01:14:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.372+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7c0c73d0 con 0x7f8b7c15de20 2024-04-22T01:14:40.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.372+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b740c2020 con 0x7f8b7c15de20 2024-04-22T01:14:40.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.372+0000 7f8b6effd640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b600678f0 0x7f8b60069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.372+0000 7f8b820bd640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b600678f0 0x7f8b60069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.372+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8b74123440 con 0x7f8b7c15de20 2024-04-22T01:14:40.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.372+0000 7f8b820bd640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b600678f0 0x7f8b60069db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8b70006fd0 tx=0x7f8b70008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.376+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8b740c9020 con 0x7f8b7c15de20 2024-04-22T01:14:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.536+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b7c06b9b0 con 0x7f8b7c15de20 2024-04-22T01:14:40.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.536+0000 7f8b6effd640 1 -- 172.21.15.70:0/2605678705 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f8b74018d40 con 0x7f8b7c15de20 2024-04-22T01:14:40.539 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:14:40.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.536+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b600678f0 msgr2=0x7f8b60069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.536+0000 7f8b84321640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b600678f0 0x7f8b60069db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8b70006fd0 tx=0x7f8b70008040 comp rx=0 tx=0).stop 2024-04-22T01:14:40.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.536+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 msgr2=0x7f8b7c0cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.536+0000 7f8b84321640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 0x7f8b7c0cd0b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8b7401d890 tx=0x7f8b7401bf80 comp rx=0 tx=0).stop 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 shutdown_connections 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b600678f0 0x7f8b60069db0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b7c15de20 0x7f8b7c0cd0b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b7c0cea10 0x7f8b7c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 --2- 172.21.15.70:0/2605678705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b7c0b6db0 0x7f8b7c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 >> 172.21.15.70:0/2605678705 conn(0x7f8b7c072d20 msgr2=0x7f8b7c0744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 shutdown_connections 2024-04-22T01:14:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.540+0000 7f8b84321640 1 -- 172.21.15.70:0/2605678705 wait complete. 2024-04-22T01:14:40.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:14:40.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 Processor -- start 2024-04-22T01:14:40.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 -- start start 2024-04-22T01:14:40.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97600b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97600aee40 0x7f97600b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976006dbd0 con 0x7f976015bb50 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f976006de70 con 0x7f97600aee40 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f97660c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f976006e150 con 0x7f976015aa30 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97600aee40 0x7f97600b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97600b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97600aee40 0x7f97600b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:45658/0 (socket says 172.21.15.70:45658) 2024-04-22T01:14:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 -- 172.21.15.70:0/938571090 learned_addr learned my addr 172.21.15.70:0/938571090 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 -- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 msgr2=0x7f97600b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 --2- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97600b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 -- 172.21.15.70:0/938571090 >> v1:172.21.15.98:6789/0 conn(0x7f976015bb50 legacy=0x7f97600ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.672+0000 7f975ffff640 1 -- 172.21.15.70:0/938571090 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976006e430 con 0x7f97600aee40 2024-04-22T01:14:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f975ffff640 1 --2- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b28f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f975800d150 tx=0x7f975801ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56804318a36a4de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f975e7fc640 1 -- 172.21.15.70:0/938571090 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97580a7ce0 con 0x7f97600aee40 2024-04-22T01:14:40.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f975e7fc640 1 -- 172.21.15.70:0/938571090 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975800b0d0 con 0x7f97600aee40 2024-04-22T01:14:40.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f975e7fc640 1 -- 172.21.15.70:0/938571090 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758009600 con 0x7f97600aee40 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 msgr2=0x7f97600b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 --2- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b28f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f975800d150 tx=0x7f975801ec30 comp rx=0 tx=0).stop 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- 172.21.15.70:0/938571090 shutdown_connections 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 --2- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b28f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 --2- 172.21.15.70:0/938571090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97600b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- 172.21.15.70:0/938571090 >> 172.21.15.70:0/938571090 conn(0x7f976006b1f0 msgr2=0x7f976006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- 172.21.15.70:0/938571090 shutdown_connections 2024-04-22T01:14:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- 172.21.15.70:0/938571090 wait complete. 2024-04-22T01:14:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 Processor -- start 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- start start 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97601f9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 0x7f97601f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f976006ead0 con 0x7f976015bb50 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f976006e950 con 0x7f97600aee40 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.676+0000 7f97660c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9760071b20 con 0x7f976015aa30 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 0x7f97601f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 0x7f97601f9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44758/0 (socket says 172.21.15.70:44758) 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 -- 172.21.15.70:0/229325936 learned_addr learned my addr 172.21.15.70:0/229325936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975f7fe640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975effd640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97601f9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 -- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 msgr2=0x7f97601f9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97601f9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 -- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 msgr2=0x7f97600b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 -- 172.21.15.70:0/229325936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976015e0f0 con 0x7f976015bb50 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975effd640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97601f9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975ffff640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 0x7f97601f9fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f975801d2e0 tx=0x7f975801b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758009800 con 0x7f976015bb50 2024-04-22T01:14:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97580be070 con 0x7f976015bb50 2024-04-22T01:14:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758009c20 con 0x7f976015bb50 2024-04-22T01:14:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97600a8560 con 0x7f976015bb50 2024-04-22T01:14:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97600b00b0 con 0x7f976015bb50 2024-04-22T01:14:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97600a8a40 con 0x7f976015bb50 2024-04-22T01:14:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f97580b6070 con 0x7f976015bb50 2024-04-22T01:14:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f9747fff640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9758120ae0 con 0x7f976015bb50 2024-04-22T01:14:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975f7fe640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.680+0000 7f975f7fe640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9750009af0 tx=0x7f9750008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:40.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.684+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9758018820 con 0x7f976015bb50 2024-04-22T01:14:40.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97600afea0 con 0x7f976015bb50 2024-04-22T01:14:40.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f9747fff640 1 -- 172.21.15.70:0/229325936 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f97580f1b60 con 0x7f976015bb50 2024-04-22T01:14:40.865 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:40.865 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T01:14:40.866 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 2 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 msgr2=0x7f974c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9750009af0 tx=0x7f9750008040 comp rx=0 tx=0).stop 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 msgr2=0x7f97601f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 0x7f97601f9fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f975801d2e0 tx=0x7f975801b9d0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 shutdown_connections 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976015bb50 0x7f97601f9fe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976015aa30 0x7f97601f9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 --2- 172.21.15.70:0/229325936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97600aee40 0x7f97600b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:40.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 >> 172.21.15.70:0/229325936 conn(0x7f976006b1f0 msgr2=0x7f976015d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:40.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 shutdown_connections 2024-04-22T01:14:40.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:40.864+0000 7f97660c2640 1 -- 172.21.15.70:0/229325936 wait complete. 2024-04-22T01:14:42.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:14:42.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:14:42.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:14:43.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.012+0000 7f56fb7c8640 1 Processor -- start 2024-04-22T01:14:43.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.012+0000 7f56fb7c8640 1 -- start start 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56fb7c8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56f415bb40 0x7f56f4072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56fb7c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f56f406ed40 0x7f56f40b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56fb7c8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f4161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56fb7c8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f56f406e8b0 con 0x7f56f406ed40 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56fb7c8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f56f415bf20 con 0x7f56f415bb40 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56fb7c8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40aec30 con 0x7f56f40b3b00 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f56f406ed40 0x7f56f40b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f56f406ed40 0x7f56f40b35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44772/0 (socket says 172.21.15.70:44772) 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 -- 172.21.15.70:0/3068925665 learned_addr learned my addr 172.21.15.70:0/3068925665 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f9564640 1 --2- 172.21.15.70:0/3068925665 >> v2:172.21.15.98:3300/0 conn(0x7f56f415bb40 0x7f56f4072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f9d65640 1 --2- 172.21.15.70:0/3068925665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f4161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 -- 172.21.15.70:0/3068925665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 msgr2=0x7f56f4161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 --2- 172.21.15.70:0/3068925665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f4161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 -- 172.21.15.70:0/3068925665 >> v2:172.21.15.98:3300/0 conn(0x7f56f415bb40 msgr2=0x7f56f4072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 --2- 172.21.15.70:0/3068925665 >> v2:172.21.15.98:3300/0 conn(0x7f56f415bb40 0x7f56f4072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 -- 172.21.15.70:0/3068925665 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40aef10 con 0x7f56f406ed40 2024-04-22T01:14:43.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f9564640 1 --2- 172.21.15.70:0/3068925665 >> v2:172.21.15.98:3300/0 conn(0x7f56f415bb40 0x7f56f4072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:43.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56f8d63640 1 --2- 172.21.15.70:0/3068925665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f40b35c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56f406fce0 tx=0x7f56e401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5540184672d756 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56ebfff640 1 -- 172.21.15.70:0/3068925665 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e40b1070 con 0x7f56f406ed40 2024-04-22T01:14:43.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56ebfff640 1 -- 172.21.15.70:0/3068925665 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e40ac630 con 0x7f56f406ed40 2024-04-22T01:14:43.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.016+0000 7f56ebfff640 1 -- 172.21.15.70:0/3068925665 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e40a8050 con 0x7f56f406ed40 2024-04-22T01:14:43.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- 172.21.15.70:0/3068925665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 msgr2=0x7f56f40b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/3068925665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f40b35c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56f406fce0 tx=0x7f56e401c520 comp rx=0 tx=0).stop 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- 172.21.15.70:0/3068925665 shutdown_connections 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/3068925665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f4161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/3068925665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f40b35c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/3068925665 >> v2:172.21.15.98:3300/0 conn(0x7f56f415bb40 0x7f56f4072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- 172.21.15.70:0/3068925665 >> 172.21.15.70:0/3068925665 conn(0x7f56f406b1f0 msgr2=0x7f56f406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- 172.21.15.70:0/3068925665 shutdown_connections 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- 172.21.15.70:0/3068925665 wait complete. 2024-04-22T01:14:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 Processor -- start 2024-04-22T01:14:43.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- start start 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f414e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f415bb40 0x7f56f41548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56f406f740 con 0x7f56f406ed40 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56f406f5c0 con 0x7f56f415bb40 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40cf600 con 0x7f56f40b3b00 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f414e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f414e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44774/0 (socket says 172.21.15.70:44774) 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 -- 172.21.15.70:0/1873701654 learned_addr learned my addr 172.21.15.70:0/1873701654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 -- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 msgr2=0x7f56f414c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 -- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f415bb40 msgr2=0x7f56f41548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9d65640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f415bb40 0x7f56f41548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f415bb40 0x7f56f41548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 -- 172.21.15.70:0/1873701654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f415c220 con 0x7f56f406ed40 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56f9564640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f414e470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00ce30 tx=0x7f56ec00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00f890 con 0x7f56f406ed40 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec00fa30 con 0x7f56f406ed40 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.020+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f415f770 con 0x7f56f406ed40 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00fe00 con 0x7f56f406ed40 2024-04-22T01:14:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f415fa50 con 0x7f56f406ed40 2024-04-22T01:14:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f4068210 con 0x7f56f406ed40 2024-04-22T01:14:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f56ec018400 con 0x7f56f406ed40 2024-04-22T01:14:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56e9ffb640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f56d80678c0 0x7f56d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f56ec0866f0 con 0x7f56f406ed40 2024-04-22T01:14:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56f8d63640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f56d80678c0 0x7f56d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.024+0000 7f56f8d63640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f56d80678c0 0x7f56d8069d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f56e40ae000 tx=0x7f56e4004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.028+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f56ec014030 con 0x7f56f406ed40 2024-04-22T01:14:43.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.220+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56f406e040 con 0x7f56f406ed40 2024-04-22T01:14:43.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.220+0000 7f56e9ffb640 1 -- 172.21.15.70:0/1873701654 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3864 (secure 0 0 0) 0x7f56ec00ae20 con 0x7f56f406ed40 2024-04-22T01:14:43.224 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:43.224 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T01:14:43.224 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 3 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f56d80678c0 msgr2=0x7f56d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f56d80678c0 0x7f56d8069d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f56e40ae000 tx=0x7f56e4004740 comp rx=0 tx=0).stop 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 msgr2=0x7f56f414e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f414e470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00ce30 tx=0x7f56ec00a650 comp rx=0 tx=0).stop 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 shutdown_connections 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f56d80678c0 0x7f56d8069d80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f415bb40 0x7f56f41548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56f40b3b00 0x7f56f414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 --2- 172.21.15.70:0/1873701654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56f406ed40 0x7f56f414e470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 >> 172.21.15.70:0/1873701654 conn(0x7f56f406b1f0 msgr2=0x7f56f4072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 shutdown_connections 2024-04-22T01:14:43.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.224+0000 7f56fb7c8640 1 -- 172.21.15.70:0/1873701654 wait complete. 2024-04-22T01:14:43.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 Processor -- start 2024-04-22T01:14:43.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 -- start start 2024-04-22T01:14:43.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e541455b0 0x7f2e540cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e540cc150 0x7f2e54071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e54072060 0x7f2e5415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2e540724d0 con 0x7f2e541455b0 2024-04-22T01:14:43.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2e540d0250 con 0x7f2e54072060 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.356+0000 7f2e5c469640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e540cd7b0 con 0x7f2e540cc150 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e541455b0 0x7f2e540cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e541455b0 0x7f2e540cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44784/0 (socket says 172.21.15.70:44784) 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 -- 172.21.15.70:0/616362820 learned_addr learned my addr 172.21.15.70:0/616362820 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e59a04640 1 --2- 172.21.15.70:0/616362820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e540cc150 0x7f2e54071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5aa06640 1 --2- 172.21.15.70:0/616362820 >> v2:172.21.15.98:3300/0 conn(0x7f2e54072060 0x7f2e5415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 -- 172.21.15.70:0/616362820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e540cc150 msgr2=0x7f2e54071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 --2- 172.21.15.70:0/616362820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e540cc150 0x7f2e54071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 -- 172.21.15.70:0/616362820 >> v2:172.21.15.98:3300/0 conn(0x7f2e54072060 msgr2=0x7f2e5415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 --2- 172.21.15.70:0/616362820 >> v2:172.21.15.98:3300/0 conn(0x7f2e54072060 0x7f2e5415f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 -- 172.21.15.70:0/616362820 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e540cc710 con 0x7f2e541455b0 2024-04-22T01:14:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5aa06640 1 --2- 172.21.15.70:0/616362820 >> v2:172.21.15.98:3300/0 conn(0x7f2e54072060 0x7f2e5415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5a205640 1 --2- 172.21.15.70:0/616362820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e541455b0 0x7f2e540cfd10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e540b1580 tx=0x7f2e5001c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adf8bab89187579 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e59203640 1 -- 172.21.15.70:0/616362820 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500b1070 con 0x7f2e541455b0 2024-04-22T01:14:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e59203640 1 -- 172.21.15.70:0/616362820 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e500ac440 con 0x7f2e541455b0 2024-04-22T01:14:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e59203640 1 -- 172.21.15.70:0/616362820 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500ac810 con 0x7f2e541455b0 2024-04-22T01:14:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 -- 172.21.15.70:0/616362820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e541455b0 msgr2=0x7f2e540cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 --2- 172.21.15.70:0/616362820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e541455b0 0x7f2e540cfd10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e540b1580 tx=0x7f2e5001c140 comp rx=0 tx=0).stop 2024-04-22T01:14:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 -- 172.21.15.70:0/616362820 shutdown_connections 2024-04-22T01:14:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 --2- 172.21.15.70:0/616362820 >> v2:172.21.15.98:3300/0 conn(0x7f2e54072060 0x7f2e5415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 --2- 172.21.15.70:0/616362820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e540cc150 0x7f2e54071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 --2- 172.21.15.70:0/616362820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e541455b0 0x7f2e540cfd10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 -- 172.21.15.70:0/616362820 >> 172.21.15.70:0/616362820 conn(0x7f2e5406b1f0 msgr2=0x7f2e5406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 -- 172.21.15.70:0/616362820 shutdown_connections 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.360+0000 7f2e5c469640 1 -- 172.21.15.70:0/616362820 wait complete. 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 Processor -- start 2024-04-22T01:14:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- start start 2024-04-22T01:14:43.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e54072060 0x7f2e541f8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 0x7f2e54141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e541455b0 0x7f2e54143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5406e380 con 0x7f2e540cc150 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5406e200 con 0x7f2e541455b0 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5406e500 con 0x7f2e54072060 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 0x7f2e54141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 0x7f2e54141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44798/0 (socket says 172.21.15.70:44798) 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 -- 172.21.15.70:0/850903472 learned_addr learned my addr 172.21.15.70:0/850903472 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5a205640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e54072060 0x7f2e541f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5aa06640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e541455b0 0x7f2e54143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 -- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e54072060 msgr2=0x7f2e541f8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e54072060 0x7f2e541f8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 -- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e541455b0 msgr2=0x7f2e54143d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e541455b0 0x7f2e54143d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 -- 172.21.15.70:0/850903472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e54157930 con 0x7f2e540cc150 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e59a04640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 0x7f2e54141280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400d800 tx=0x7f2e4400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400b6c0 con 0x7f2e540cc150 2024-04-22T01:14:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e44019070 con 0x7f2e540cc150 2024-04-22T01:14:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541404c0 con 0x7f2e540cc150 2024-04-22T01:14:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400ba60 con 0x7f2e540cc150 2024-04-22T01:14:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e54140a80 con 0x7f2e540cc150 2024-04-22T01:14:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e54068210 con 0x7f2e540cc150 2024-04-22T01:14:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.364+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e4400bc00 con 0x7f2e540cc150 2024-04-22T01:14:43.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.368+0000 7f2e4affd640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e380678c0 0x7f2e38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.368+0000 7f2e5a205640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e380678c0 0x7f2e38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.368+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f2e44085ec0 con 0x7f2e540cc150 2024-04-22T01:14:43.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.368+0000 7f2e5a205640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e380678c0 0x7f2e38069d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2e500ae000 tx=0x7f2e500a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.372+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2e54157930 con 0x7f2e540cc150 2024-04-22T01:14:43.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.520+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e5415f510 con 0x7f2e540cc150 2024-04-22T01:14:43.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.520+0000 7f2e4affd640 1 -- 172.21.15.70:0/850903472 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7f2e44056cb0 con 0x7f2e540cc150 2024-04-22T01:14:43.522 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:43.522 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":16,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:36.814010+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e380678c0 msgr2=0x7f2e38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e380678c0 0x7f2e38069d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2e500ae000 tx=0x7f2e500a6070 comp rx=0 tx=0).stop 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 msgr2=0x7f2e54141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 0x7f2e54141280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400d800 tx=0x7f2e4400dcc0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 shutdown_connections 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e380678c0 0x7f2e38069d80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e541455b0 0x7f2e54143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e540cc150 0x7f2e54141280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 --2- 172.21.15.70:0/850903472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e54072060 0x7f2e541f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 >> 172.21.15.70:0/850903472 conn(0x7f2e5406b1f0 msgr2=0x7f2e5415b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 shutdown_connections 2024-04-22T01:14:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.524+0000 7f2e5c469640 1 -- 172.21.15.70:0/850903472 wait complete. 2024-04-22T01:14:43.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:14:43.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:14:43.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef96459640 1 Processor -- start 2024-04-22T01:14:43.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef96459640 1 -- start start 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef96459640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 0x7fef90071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef96459640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9015c290 con 0x7fef900720b0 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef96459640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9015c570 con 0x7fef900cd220 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef96459640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fef90072510 con 0x7fef900b4450 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fef900720b0 0x7fef900ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43324/0 (socket says 172.21.15.70:43324) 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 -- 172.21.15.70:0/1400603450 learned_addr learned my addr 172.21.15.70:0/1400603450 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8ffff640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 0x7fef90071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834261983 0 0) 0x7fef9015c290 con 0x7fef900720b0 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef78003530 con 0x7fef900720b0 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 902426172 0 0) 0x7fef78003530 con 0x7fef900720b0 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 msgr2=0x7fef90071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 0x7fef90071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 >> v1:172.21.15.98:6789/0 conn(0x7fef900cd220 legacy=0x7fef900b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9015c850 con 0x7fef900720b0 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fef84001680 con 0x7fef900720b0 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 >> v1:172.21.15.70:6789/0 conn(0x7fef900720b0 legacy=0x7fef900ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef780039f0 0x7fef78003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78004810 0x7fef78004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 0x7fef78005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef7800bad0 con 0x7fef78004810 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef7800bc80 con 0x7fef780039f0 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fef7800be80 con 0x7fef900b4450 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78004810 0x7fef78004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8ffff640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef780039f0 0x7fef78003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef949f6640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 0x7fef78005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 -- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 msgr2=0x7fef78005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900b4450 0x7fef78005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 -- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef780039f0 msgr2=0x7fef78003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef780039f0 0x7fef78003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.660+0000 7fef8f7fe640 1 -- 172.21.15.70:0/1400603450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef78003530 con 0x7fef78004810 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef8f7fe640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78004810 0x7fef78004c90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef8400c1c0 tx=0x7fef8401f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb94bfb93750dbc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84007d00 con 0x7fef78004810 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef84005ae0 con 0x7fef78004810 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef8effd640 1 -- 172.21.15.70:0/1400603450 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84007070 con 0x7fef78004810 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef96459640 1 -- 172.21.15.70:0/1400603450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78004810 msgr2=0x7fef78004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef96459640 1 --2- 172.21.15.70:0/1400603450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef78004810 0x7fef78004c90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef8400c1c0 tx=0x7fef8401f3d0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef949f6640 1 -- 172.21.15.70:0/1400603450 reap_dead start 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef96459640 1 -- 172.21.15.70:0/1400603450 shutdown_connections 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef96459640 1 -- 172.21.15.70:0/1400603450 >> 172.21.15.70:0/1400603450 conn(0x7fef900b1d40 msgr2=0x7fef900cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef96459640 1 -- 172.21.15.70:0/1400603450 shutdown_connections 2024-04-22T01:14:43.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.664+0000 7fef96459640 1 -- 172.21.15.70:0/1400603450 wait complete. 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 Processor -- start 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- start start 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78004810 0x7fef90152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef9014c060 0x7fef90152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 0x7fef9014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef900b4830 con 0x7fef90150d40 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef9015d050 con 0x7fef78004810 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fef900b49b0 con 0x7fef9014c060 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 0x7fef9014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 0x7fef9014ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44810/0 (socket says 172.21.15.70:44810) 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 -- 172.21.15.70:0/3277598220 learned_addr learned my addr 172.21.15.70:0/3277598220 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 -- 172.21.15.70:0/3277598220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef9014c060 msgr2=0x7fef90152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef9014c060 0x7fef90152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 -- 172.21.15.70:0/3277598220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78004810 msgr2=0x7fef90152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78004810 0x7fef90152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 -- 172.21.15.70:0/3277598220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef901531e0 con 0x7fef90150d40 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef949f6640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 0x7fef9014ef40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef901567f0 tx=0x7fef8800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88003980 con 0x7fef90150d40 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef88018070 con 0x7fef90150d40 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88003d60 con 0x7fef90150d40 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef901534c0 con 0x7fef90150d40 2024-04-22T01:14:43.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef90150820 con 0x7fef90150d40 2024-04-22T01:14:43.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef90068210 con 0x7fef90150d40 2024-04-22T01:14:43.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef880050d0 con 0x7fef90150d40 2024-04-22T01:14:43.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8cff9640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef7c0678c0 0x7fef7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8ffff640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef7c0678c0 0x7fef7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.668+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fef880855f0 con 0x7fef90150d40 2024-04-22T01:14:43.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.672+0000 7fef8ffff640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef7c0678c0 0x7fef7c069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef8000b8e0 tx=0x7fef8000b2c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.676+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fef88056510 con 0x7fef90150d40 2024-04-22T01:14:43.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.832+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fef900d01e0 con 0x7fef90150d40 2024-04-22T01:14:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef8cff9640 1 -- 172.21.15.70:0/3277598220 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fef88055e10 con 0x7fef90150d40 2024-04-22T01:14:43.836 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef7c0678c0 msgr2=0x7fef7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef7c0678c0 0x7fef7c069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef8000b8e0 tx=0x7fef8000b2c0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 msgr2=0x7fef9014ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 0x7fef9014ef40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef901567f0 tx=0x7fef8800ec10 comp rx=0 tx=0).stop 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 shutdown_connections 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef7c0678c0 0x7fef7c069d80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef90150d40 0x7fef9014ef40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef9014c060 0x7fef90152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 --2- 172.21.15.70:0/3277598220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78004810 0x7fef90152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 >> 172.21.15.70:0/3277598220 conn(0x7fef900b1d40 msgr2=0x7fef9015d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 shutdown_connections 2024-04-22T01:14:43.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.836+0000 7fef96459640 1 -- 172.21.15.70:0/3277598220 wait complete. 2024-04-22T01:14:43.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:14:43.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.972+0000 7f1b65195640 1 Processor -- start 2024-04-22T01:14:43.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.972+0000 7f1b65195640 1 -- start start 2024-04-22T01:14:43.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b600aee40 0x7f1b600b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b6006dbd0 con 0x7f1b6015aa30 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b6006de70 con 0x7f1b6015bb50 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1b6006e150 con 0x7f1b600aee40 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b600aee40 0x7f1b600b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b600aee40 0x7f1b600b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44826/0 (socket says 172.21.15.70:44826) 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 -- 172.21.15.70:0/436690623 learned_addr learned my addr 172.21.15.70:0/436690623 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 -- 172.21.15.70:0/436690623 >> v1:172.21.15.98:6789/0 conn(0x7f1b6015bb50 legacy=0x7f1b600b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 -- 172.21.15.70:0/436690623 >> v1:172.21.15.70:6789/0 conn(0x7f1b6015aa30 legacy=0x7f1b600ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 -- 172.21.15.70:0/436690623 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6006e430 con 0x7f1b600aee40 2024-04-22T01:14:43.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5659c640 1 --2- 172.21.15.70:0/436690623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b600aee40 0x7f1b600b4720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009870 tx=0x7f1b4801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562fbb79d6bcfe45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5e59c640 1 -- 172.21.15.70:0/436690623 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480a9400 con 0x7f1b600aee40 2024-04-22T01:14:43.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5e59c640 1 -- 172.21.15.70:0/436690623 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b480a95a0 con 0x7f1b600aee40 2024-04-22T01:14:43.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b5e59c640 1 -- 172.21.15.70:0/436690623 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480a9970 con 0x7f1b600aee40 2024-04-22T01:14:43.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- 172.21.15.70:0/436690623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b600aee40 msgr2=0x7f1b600b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 --2- 172.21.15.70:0/436690623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b600aee40 0x7f1b600b4720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009870 tx=0x7f1b4801c520 comp rx=0 tx=0).stop 2024-04-22T01:14:43.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- 172.21.15.70:0/436690623 shutdown_connections 2024-04-22T01:14:43.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 --2- 172.21.15.70:0/436690623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b600aee40 0x7f1b600b4720 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- 172.21.15.70:0/436690623 >> 172.21.15.70:0/436690623 conn(0x7f1b6006b1f0 msgr2=0x7f1b6006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:43.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- 172.21.15.70:0/436690623 shutdown_connections 2024-04-22T01:14:43.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.976+0000 7f1b65195640 1 -- 172.21.15.70:0/436690623 wait complete. 2024-04-22T01:14:43.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 Processor -- start 2024-04-22T01:14:43.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- start start 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b600aee40 0x7f1b6014e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 0x7f1b60149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6015bb50 0x7f1b6014f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6006ead0 con 0x7f1b6015aa30 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6006e950 con 0x7f1b6015bb50 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1b60168aa0 con 0x7f1b600aee40 2024-04-22T01:14:43.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 0x7f1b60149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 0x7f1b60149b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44830/0 (socket says 172.21.15.70:44830) 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 -- 172.21.15.70:0/720290854 learned_addr learned my addr 172.21.15.70:0/720290854 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 -- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b600aee40 msgr2=0x7f1b6014e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5ed9d640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b600aee40 0x7f1b6014e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b600aee40 0x7f1b6014e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 -- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6015bb50 msgr2=0x7f1b6014f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6015bb50 0x7f1b6014f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 -- 172.21.15.70:0/720290854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b60156cc0 con 0x7f1b6015aa30 2024-04-22T01:14:43.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b5659c640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 0x7f1b60149b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1b4801a150 tx=0x7f1b480192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b1800 con 0x7f1b6015aa30 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b480b19a0 con 0x7f1b6015aa30 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b1d70 con 0x7f1b6015aa30 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6014a370 con 0x7f1b6015aa30 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6014f220 con 0x7f1b6015aa30 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.980+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b60068210 con 0x7f1b6015aa30 2024-04-22T01:14:43.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.984+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b480c0060 con 0x7f1b6015aa30 2024-04-22T01:14:43.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.984+0000 7f1b57fff640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b400678c0 0x7f1b40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:43.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.984+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f1b4811fbc0 con 0x7f1b6015aa30 2024-04-22T01:14:43.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.984+0000 7f1b5ed9d640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b400678c0 0x7f1b40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:43.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.984+0000 7f1b5ed9d640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b400678c0 0x7f1b40069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:43.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:43.988+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1b60156cc0 con 0x7f1b6015aa30 2024-04-22T01:14:44.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.148+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1b600b2de0 con 0x7f1b6015aa30 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b57fff640 1 -- 172.21.15.70:0/720290854 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f1b480ad020 con 0x7f1b6015aa30 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:14:44.152 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:14:44.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b400678c0 msgr2=0x7f1b40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b400678c0 0x7f1b40069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50009040 comp rx=0 tx=0).stop 2024-04-22T01:14:44.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 msgr2=0x7f1b60149b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 0x7f1b60149b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1b4801a150 tx=0x7f1b480192c0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 shutdown_connections 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b400678c0 0x7f1b40069d80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6015bb50 0x7f1b6014f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b6015aa30 0x7f1b60149b00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 --2- 172.21.15.70:0/720290854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b600aee40 0x7f1b6014e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 >> 172.21.15.70:0/720290854 conn(0x7f1b6006b1f0 msgr2=0x7f1b6015c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 shutdown_connections 2024-04-22T01:14:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.152+0000 7f1b65195640 1 -- 172.21.15.70:0/720290854 wait complete. 2024-04-22T01:14:44.171 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:14:44.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:14:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 Processor -- start 2024-04-22T01:14:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 -- start start 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f701815de20 0x7f70180b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70180b6db0 0x7f70180d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70180b75e0 con 0x7f70180cea10 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f70180b78c0 con 0x7f701815de20 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f7020782640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f70180b7ba0 con 0x7f70180b6db0 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f701e51e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f701815de20 0x7f70180b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f701ed1f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f70180cea10 0x7f70180cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43344/0 (socket says 172.21.15.70:43344) 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.292+0000 7f701ed1f640 1 -- 172.21.15.70:0/2228819184 learned_addr learned my addr 172.21.15.70:0/2228819184 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701dd1d640 1 --2- 172.21.15.70:0/2228819184 >> v2:172.21.15.98:3300/0 conn(0x7f70180b6db0 0x7f70180d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701e51e640 1 -- 172.21.15.70:0/2228819184 >> v2:172.21.15.98:3300/0 conn(0x7f70180b6db0 msgr2=0x7f70180d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701e51e640 1 --2- 172.21.15.70:0/2228819184 >> v2:172.21.15.98:3300/0 conn(0x7f70180b6db0 0x7f70180d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701e51e640 1 -- 172.21.15.70:0/2228819184 >> v1:172.21.15.70:6789/0 conn(0x7f70180cea10 legacy=0x7f70180cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701e51e640 1 -- 172.21.15.70:0/2228819184 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70180b7e80 con 0x7f701815de20 2024-04-22T01:14:44.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701d51c640 1 -- 172.21.15.70:0/2228819184 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4049477496 0 0) 0x7f70180b75e0 con 0x7f70180cea10 2024-04-22T01:14:44.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701e51e640 1 --2- 172.21.15.70:0/2228819184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f701815de20 0x7f70180b98a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70180d4e10 tx=0x7f701401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edda2e2466b1ca2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:44.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701d51c640 1 -- 172.21.15.70:0/2228819184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140b1070 con 0x7f701815de20 2024-04-22T01:14:44.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701d51c640 1 -- 172.21.15.70:0/2228819184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70140a8050 con 0x7f701815de20 2024-04-22T01:14:44.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f701d51c640 1 -- 172.21.15.70:0/2228819184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140ac860 con 0x7f701815de20 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 -- 172.21.15.70:0/2228819184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f701815de20 msgr2=0x7f70180b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 --2- 172.21.15.70:0/2228819184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f701815de20 0x7f70180b98a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70180d4e10 tx=0x7f701401c450 comp rx=0 tx=0).stop 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 -- 172.21.15.70:0/2228819184 shutdown_connections 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 --2- 172.21.15.70:0/2228819184 >> v2:172.21.15.98:3300/0 conn(0x7f70180b6db0 0x7f70180d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 --2- 172.21.15.70:0/2228819184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f701815de20 0x7f70180b98a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 -- 172.21.15.70:0/2228819184 >> 172.21.15.70:0/2228819184 conn(0x7f7018072d20 msgr2=0x7f7018073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 -- 172.21.15.70:0/2228819184 shutdown_connections 2024-04-22T01:14:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.296+0000 7f7020782640 1 -- 172.21.15.70:0/2228819184 wait complete. 2024-04-22T01:14:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 Processor -- start 2024-04-22T01:14:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- start start 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 0x7f70181f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70180cea10 0x7f701815d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f701815de20 0x7f70180cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70180b7310 con 0x7f70180b6db0 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70180b7190 con 0x7f70180cea10 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f70180b8e90 con 0x7f701815de20 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 0x7f70181f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 0x7f70181f67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44844/0 (socket says 172.21.15.70:44844) 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 -- 172.21.15.70:0/133823751 learned_addr learned my addr 172.21.15.70:0/133823751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701dd1d640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70180cea10 0x7f701815d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 -- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f701815de20 msgr2=0x7f70180cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f701815de20 0x7f70180cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 -- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70180cea10 msgr2=0x7f701815d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70180cea10 0x7f701815d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 -- 172.21.15.70:0/133823751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70180cc9b0 con 0x7f70180b6db0 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701dd1d640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70180cea10 0x7f701815d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701e51e640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 0x7f70181f67f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7014019060 tx=0x7f701401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140b1040 con 0x7f70180b6db0 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70140acc80 con 0x7f70180b6db0 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70180ccc90 con 0x7f70180b6db0 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140b7620 con 0x7f70180b6db0 2024-04-22T01:14:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70180cd250 con 0x7f70180b6db0 2024-04-22T01:14:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7018068210 con 0x7f70180b6db0 2024-04-22T01:14:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70140bf030 con 0x7f70180b6db0 2024-04-22T01:14:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f700effd640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ffc0678c0 0x7f6ffc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f701411f410 con 0x7f70180b6db0 2024-04-22T01:14:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.300+0000 7f701dd1d640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ffc0678c0 0x7f6ffc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.304+0000 7f701dd1d640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ffc0678c0 0x7f6ffc069d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f701815d150 tx=0x7f7008008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.308+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f70140c6060 con 0x7f70180b6db0 2024-04-22T01:14:44.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.468+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f701806b9b0 con 0x7f70180b6db0 2024-04-22T01:14:44.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.644+0000 7f700effd640 1 -- 172.21.15.70:0/133823751 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f7014016070 con 0x7f70180b6db0 2024-04-22T01:14:44.646 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:14:44.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ffc0678c0 msgr2=0x7f6ffc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ffc0678c0 0x7f6ffc069d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f701815d150 tx=0x7f7008008040 comp rx=0 tx=0).stop 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 msgr2=0x7f70181f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 0x7f70181f67f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7014019060 tx=0x7f701401ac90 comp rx=0 tx=0).stop 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 shutdown_connections 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ffc0678c0 0x7f6ffc069d80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f701815de20 0x7f70180cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70180cea10 0x7f701815d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 --2- 172.21.15.70:0/133823751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70180b6db0 0x7f70181f67f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 >> 172.21.15.70:0/133823751 conn(0x7f7018072d20 msgr2=0x7f70180b4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 shutdown_connections 2024-04-22T01:14:44.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.648+0000 7f7020782640 1 -- 172.21.15.70:0/133823751 wait complete. 2024-04-22T01:14:44.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-22T01:14:44.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafc547d640 1 Processor -- start 2024-04-22T01:14:44.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafc547d640 1 -- start start 2024-04-22T01:14:44.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafc547d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 0x7fafc006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafc547d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc006b390 con 0x7fafc00cd1f0 2024-04-22T01:14:44.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafc547d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc0067440 con 0x7fafc015e0e0 2024-04-22T01:14:44.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafc547d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fafc00cdbe0 con 0x7fafc00af8e0 2024-04-22T01:14:44.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbe7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fafc00cd1f0 0x7fafc00c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43352/0 (socket says 172.21.15.70:43352) 2024-04-22T01:14:44.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbe7fc640 1 -- 172.21.15.70:0/861802268 learned_addr learned my addr 172.21.15.70:0/861802268 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbf7fe640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 0x7fafc006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141029493 0 0) 0x7fafc006b390 con 0x7fafc00cd1f0 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa8003530 con 0x7fafc00cd1f0 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 57589035 0 0) 0x7fafa8003530 con 0x7fafc00cd1f0 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 msgr2=0x7fafc006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 0x7fafc006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 >> v1:172.21.15.98:6789/0 conn(0x7fafc015e0e0 legacy=0x7fafc00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc00676e0 con 0x7fafc00cd1f0 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fafb40016a0 con 0x7fafc00cd1f0 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 >> v1:172.21.15.70:6789/0 conn(0x7fafc00cd1f0 legacy=0x7fafc00c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa80039f0 0x7fafa8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8004810 0x7fafa8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 0x7fafa80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00cdbe0 con 0x7fafa80039f0 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafa800ba80 con 0x7fafa8004810 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fafa800bc80 con 0x7fafc00af8e0 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbeffd640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa80039f0 0x7fafa8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbe7fc640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8004810 0x7fafa8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbf7fe640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 0x7fafa80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbeffd640 1 -- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 msgr2=0x7fafa80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbeffd640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafc00af8e0 0x7fafa80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbeffd640 1 -- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8004810 msgr2=0x7fafa8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbeffd640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8004810 0x7fafa8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.804+0000 7fafbeffd640 1 -- 172.21.15.70:0/861802268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa8003530 con 0x7fafa80039f0 2024-04-22T01:14:44.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafbeffd640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa80039f0 0x7fafa8003e10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafc0067610 tx=0x7fafb001c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3969d515c24bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:44.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb00013a0 con 0x7fafa80039f0 2024-04-22T01:14:44.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb00052d0 con 0x7fafa80039f0 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafbdffb640 1 -- 172.21.15.70:0/861802268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0003820 con 0x7fafa80039f0 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 -- 172.21.15.70:0/861802268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa80039f0 msgr2=0x7fafa8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 --2- 172.21.15.70:0/861802268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa80039f0 0x7fafa8003e10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafc0067610 tx=0x7fafb001c350 comp rx=0 tx=0).stop 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafbf7fe640 1 -- 172.21.15.70:0/861802268 reap_dead start 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 -- 172.21.15.70:0/861802268 shutdown_connections 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 -- 172.21.15.70:0/861802268 >> 172.21.15.70:0/861802268 conn(0x7fafc0099a60 msgr2=0x7fafc00b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 -- 172.21.15.70:0/861802268 shutdown_connections 2024-04-22T01:14:44.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 -- 172.21.15.70:0/861802268 wait complete. 2024-04-22T01:14:44.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.808+0000 7fafc547d640 1 Processor -- start 2024-04-22T01:14:44.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- start start 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafa80039f0 0x7fafc015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 0x7fafc00cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc00adf40 0x7fafc00ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0169d60 con 0x7fafc0146b80 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafc006c6d0 con 0x7fafc00adf40 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fafc0169ee0 con 0x7fafa80039f0 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 0x7fafc00cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 0x7fafc00cfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44854/0 (socket says 172.21.15.70:44854) 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 -- 172.21.15.70:0/2274529988 learned_addr learned my addr 172.21.15.70:0/2274529988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbeffd640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafa80039f0 0x7fafc015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbf7fe640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc00adf40 0x7fafc00ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 -- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafa80039f0 msgr2=0x7fafc015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafa80039f0 0x7fafc015aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 -- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc00adf40 msgr2=0x7fafc00ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc00adf40 0x7fafc00ae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 -- 172.21.15.70:0/2274529988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc0143f40 con 0x7fafc0146b80 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafbe7fc640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 0x7fafc00cfee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fafb400e780 tx=0x7fafb400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb400ee30 con 0x7fafc0146b80 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb4018070 con 0x7fafc0146b80 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc0144220 con 0x7fafc0146b80 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb400cb10 con 0x7fafc0146b80 2024-04-22T01:14:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc01447e0 con 0x7fafc0146b80 2024-04-22T01:14:44.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc01449e0 con 0x7fafc0146b80 2024-04-22T01:14:44.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fafb400ccb0 con 0x7fafc0146b80 2024-04-22T01:14:44.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.812+0000 7fafa77fe640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafac0678c0 0x7fafac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:44.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.816+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7fafb40858d0 con 0x7fafc0146b80 2024-04-22T01:14:44.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.816+0000 7fafbeffd640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafac0678c0 0x7fafac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:44.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.816+0000 7fafbeffd640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafac0678c0 0x7fafac069d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fafa80057f0 tx=0x7fafb00011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:44.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.820+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fafc01449e0 con 0x7fafc0146b80 2024-04-22T01:14:44.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:44.980+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fafc0068210 con 0x7fafc0146b80 2024-04-22T01:14:45.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.656+0000 7fafa77fe640 1 -- 172.21.15.70:0/2274529988 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7fafb4056680 con 0x7fafc0146b80 2024-04-22T01:14:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafac0678c0 msgr2=0x7fafac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafac0678c0 0x7fafac069d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fafa80057f0 tx=0x7fafb00011a0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 msgr2=0x7fafc00cfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 0x7fafc00cfee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fafb400e780 tx=0x7fafb400ec40 comp rx=0 tx=0).stop 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 shutdown_connections 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafac0678c0 0x7fafac069d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc00adf40 0x7fafc00ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafc0146b80 0x7fafc00cfee0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 --2- 172.21.15.70:0/2274529988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafa80039f0 0x7fafc015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 >> 172.21.15.70:0/2274529988 conn(0x7fafc0099a60 msgr2=0x7fafc006edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 shutdown_connections 2024-04-22T01:14:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.660+0000 7fafc547d640 1 -- 172.21.15.70:0/2274529988 wait complete. 2024-04-22T01:14:45.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:14:45.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.812+0000 7fdf8b847640 1 Processor -- start 2024-04-22T01:14:45.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.812+0000 7fdf8b847640 1 -- start start 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf840af650 0x7fdf840b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdf8406c5e0 0x7fdf8415b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf840671d0 con 0x7fdf840afa30 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdf840b6370 con 0x7fdf8406c5e0 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf840cc520 con 0x7fdf840af650 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf89de4640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdf840afa30 0x7fdf840cffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43354/0 (socket says 172.21.15.70:43354) 2024-04-22T01:14:45.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf89de4640 1 -- 172.21.15.70:0/1250592568 learned_addr learned my addr 172.21.15.70:0/1250592568 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:45.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf88de2640 1 --2- 172.21.15.70:0/1250592568 >> v2:172.21.15.70:3300/0 conn(0x7fdf8406c5e0 0x7fdf8415b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:45.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf895e3640 1 --2- 172.21.15.70:0/1250592568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf840af650 0x7fdf840b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:45.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf88de2640 1 -- 172.21.15.70:0/1250592568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf840af650 msgr2=0x7fdf840b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:45.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf88de2640 1 --2- 172.21.15.70:0/1250592568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf840af650 0x7fdf840b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf88de2640 1 -- 172.21.15.70:0/1250592568 >> v1:172.21.15.70:6789/0 conn(0x7fdf840afa30 legacy=0x7fdf840cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:45.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf88de2640 1 -- 172.21.15.70:0/1250592568 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf84067440 con 0x7fdf8406c5e0 2024-04-22T01:14:45.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf7bfff640 1 -- 172.21.15.70:0/1250592568 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2348600079 0 0) 0x7fdf840671d0 con 0x7fdf840afa30 2024-04-22T01:14:45.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf88de2640 1 --2- 172.21.15.70:0/1250592568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf8406c5e0 0x7fdf8415b9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf740097c0 tx=0x7fdf7401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32628edcffd6a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:45.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf7bfff640 1 -- 172.21.15.70:0/1250592568 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf740ab600 con 0x7fdf8406c5e0 2024-04-22T01:14:45.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf7bfff640 1 -- 172.21.15.70:0/1250592568 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf740b0070 con 0x7fdf8406c5e0 2024-04-22T01:14:45.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf7bfff640 1 -- 172.21.15.70:0/1250592568 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf740ab9c0 con 0x7fdf8406c5e0 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 -- 172.21.15.70:0/1250592568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf8406c5e0 msgr2=0x7fdf8415b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 --2- 172.21.15.70:0/1250592568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf8406c5e0 0x7fdf8415b9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf740097c0 tx=0x7fdf7401c3e0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 -- 172.21.15.70:0/1250592568 shutdown_connections 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 --2- 172.21.15.70:0/1250592568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf8406c5e0 0x7fdf8415b9f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.816+0000 7fdf8b847640 1 --2- 172.21.15.70:0/1250592568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf840af650 0x7fdf840b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- 172.21.15.70:0/1250592568 >> 172.21.15.70:0/1250592568 conn(0x7fdf84099a60 msgr2=0x7fdf840b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- 172.21.15.70:0/1250592568 shutdown_connections 2024-04-22T01:14:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- 172.21.15.70:0/1250592568 wait complete. 2024-04-22T01:14:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 Processor -- start 2024-04-22T01:14:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- start start 2024-04-22T01:14:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf8406c5e0 0x7fdf84142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf840af650 0x7fdf84158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 0x7fdf840b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840ccc50 con 0x7fdf840afa30 2024-04-22T01:14:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840ccad0 con 0x7fdf840af650 2024-04-22T01:14:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf84168af0 con 0x7fdf8406c5e0 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 0x7fdf840b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 0x7fdf840b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44874/0 (socket says 172.21.15.70:44874) 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 -- 172.21.15.70:0/4271595689 learned_addr learned my addr 172.21.15.70:0/4271595689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf88de2640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf840af650 0x7fdf84158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf895e3640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf8406c5e0 0x7fdf84142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 -- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf8406c5e0 msgr2=0x7fdf84142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf8406c5e0 0x7fdf84142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 -- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf840af650 msgr2=0x7fdf84158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf840af650 0x7fdf84158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 -- 172.21.15.70:0/4271595689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf841570c0 con 0x7fdf840afa30 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf88de2640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf840af650 0x7fdf84158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf895e3640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf8406c5e0 0x7fdf84142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf89de4640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 0x7fdf840b1660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00c7d0 tx=0x7fdf7c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c018070 con 0x7fdf840afa30 2024-04-22T01:14:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7c00a5e0 con 0x7fdf840afa30 2024-04-22T01:14:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf841573a0 con 0x7fdf840afa30 2024-04-22T01:14:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c00aa10 con 0x7fdf840afa30 2024-04-22T01:14:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.820+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf84157960 con 0x7fdf840afa30 2024-04-22T01:14:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.824+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8415d3d0 con 0x7fdf840afa30 2024-04-22T01:14:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.824+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf7c01b070 con 0x7fdf840afa30 2024-04-22T01:14:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.824+0000 7fdf79ffb640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf680678c0 0x7fdf68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.824+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fdf7c085180 con 0x7fdf840afa30 2024-04-22T01:14:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.824+0000 7fdf895e3640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf680678c0 0x7fdf68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.824+0000 7fdf895e3640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf680678c0 0x7fdf68069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdf8409e530 tx=0x7fdf80008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:45.828+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdf7c055db0 con 0x7fdf840afa30 2024-04-22T01:14:46.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.024+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdf84068920 con 0x7fdf840afa30 2024-04-22T01:14:46.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.655+0000 7fdf79ffb640 1 -- 172.21.15.70:0/4271595689 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fdf7c0556b0 con 0x7fdf840afa30 2024-04-22T01:14:46.659 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 2 and data pool 3 2024-04-22T01:14:46.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf680678c0 msgr2=0x7fdf68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf680678c0 0x7fdf68069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdf8409e530 tx=0x7fdf80008040 comp rx=0 tx=0).stop 2024-04-22T01:14:46.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 msgr2=0x7fdf840b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 0x7fdf840b1660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00c7d0 tx=0x7fdf7c00cc90 comp rx=0 tx=0).stop 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 shutdown_connections 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf680678c0 0x7fdf68069d80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf840afa30 0x7fdf840b1660 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf840af650 0x7fdf84158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 --2- 172.21.15.70:0/4271595689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf8406c5e0 0x7fdf84142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 >> 172.21.15.70:0/4271595689 conn(0x7fdf84099a60 msgr2=0x7fdf8406e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 shutdown_connections 2024-04-22T01:14:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.663+0000 7fdf8b847640 1 -- 172.21.15.70:0/4271595689 wait complete. 2024-04-22T01:14:46.703 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-22T01:14:46.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-22T01:14:46.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f0904ddb640 1 Processor -- start 2024-04-22T01:14:46.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f0904ddb640 1 -- start start 2024-04-22T01:14:46.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f0904ddb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 0x7f09000cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f0904ddb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f090015e2d0 con 0x7f09000d08c0 2024-04-22T01:14:46.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f0904ddb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f090015e570 con 0x7f09000cfce0 2024-04-22T01:14:46.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f0904ddb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f090015e810 con 0x7f090015def0 2024-04-22T01:14:46.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fed9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f09000d08c0 0x7f090006d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46346/0 (socket says 172.21.15.70:46346) 2024-04-22T01:14:46.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fed9d640 1 -- 172.21.15.70:0/694993118 learned_addr learned my addr 172.21.15.70:0/694993118 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593335137 0 0) 0x7f090015e2d0 con 0x7f09000d08c0 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e8003530 con 0x7f09000d08c0 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fe59c640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 0x7f09000cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3046589859 0 0) 0x7f08e8003530 con 0x7f09000d08c0 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 msgr2=0x7f09000cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 0x7f09000cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 >> v1:172.21.15.98:6789/0 conn(0x7f09000cfce0 legacy=0x7f09000d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090015eaf0 con 0x7f09000d08c0 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f08f8001a00 con 0x7f09000d08c0 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 >> v1:172.21.15.70:6789/0 conn(0x7f09000d08c0 legacy=0x7f090006d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 0x7f08e80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8004020 0x7f08e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8004d50 0x7f08e80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f090015e810 con 0x7f08e8004020 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08e800bb90 con 0x7f08e8004d50 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f08e800bd90 con 0x7f090015def0 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fdd9b640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8004020 0x7f08e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.827+0000 7f08fe59c640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 0x7f08e80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fed9d640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8004d50 0x7f08e80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fdd9b640 1 -- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 msgr2=0x7f08e80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fdd9b640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f090015def0 0x7f08e80039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fdd9b640 1 -- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8004d50 msgr2=0x7f08e80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fdd9b640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8004d50 0x7f08e80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fdd9b640 1 -- 172.21.15.70:0/694993118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e8003530 con 0x7f08e8004020 2024-04-22T01:14:46.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fdd9b640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8004020 0x7f08e8004810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08f4001750 tx=0x7f08f401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc7ef688b4e5ff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f40013a0 con 0x7f08e8004020 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f40052f0 con 0x7f08e8004020 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fd59a640 1 -- 172.21.15.70:0/694993118 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4003840 con 0x7f08e8004020 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f0904ddb640 1 -- 172.21.15.70:0/694993118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8004020 msgr2=0x7f08e8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f0904ddb640 1 --2- 172.21.15.70:0/694993118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8004020 0x7f08e8004810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08f4001750 tx=0x7f08f401c3e0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f08fed9d640 1 -- 172.21.15.70:0/694993118 reap_dead start 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f0904ddb640 1 -- 172.21.15.70:0/694993118 shutdown_connections 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f0904ddb640 1 -- 172.21.15.70:0/694993118 >> 172.21.15.70:0/694993118 conn(0x7f09000b36c0 msgr2=0x7f09000b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f0904ddb640 1 -- 172.21.15.70:0/694993118 shutdown_connections 2024-04-22T01:14:46.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.831+0000 7f0904ddb640 1 -- 172.21.15.70:0/694993118 wait complete. 2024-04-22T01:14:46.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 Processor -- start 2024-04-22T01:14:46.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- start start 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08e8004020 0x7f09000712f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09000b56e0 0x7f09000b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 0x7f09000aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f090009ae50 con 0x7f09000ae780 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f090015f0b0 con 0x7f09000b56e0 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f090009afd0 con 0x7f08e8004020 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 0x7f09000aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 0x7f09000aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38844/0 (socket says 172.21.15.70:38844) 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 -- 172.21.15.70:0/57663287 learned_addr learned my addr 172.21.15.70:0/57663287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fdd9b640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09000b56e0 0x7f09000b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fe59c640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08e8004020 0x7f09000712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 -- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08e8004020 msgr2=0x7f09000712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08e8004020 0x7f09000712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 -- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09000b56e0 msgr2=0x7f09000b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09000b56e0 0x7f09000b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 -- 172.21.15.70:0/57663287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09000c93b0 con 0x7f09000ae780 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08fed9d640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 0x7f09000aec30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09000cf660 tx=0x7f08f8011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8010040 con 0x7f09000ae780 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f801a070 con 0x7f09000ae780 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09000b61d0 con 0x7f09000ae780 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f800fba0 con 0x7f09000ae780 2024-04-22T01:14:46.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09000b6790 con 0x7f09000ae780 2024-04-22T01:14:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.835+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0900068920 con 0x7f09000ae780 2024-04-22T01:14:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.839+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f08f800fd40 con 0x7f09000ae780 2024-04-22T01:14:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.839+0000 7f08eeffd640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f08e00678c0 0x7f08e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.839+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f08f8088b40 con 0x7f09000ae780 2024-04-22T01:14:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.839+0000 7f08fe59c640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f08e00678c0 0x7f08e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.839+0000 7f08fe59c640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f08e00678c0 0x7f08e0069d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08f000b9b0 tx=0x7f08f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:46.843+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f08f8059710 con 0x7f09000ae780 2024-04-22T01:14:47.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.011+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f090015d900 con 0x7f09000ae780 2024-04-22T01:14:47.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.659+0000 7f08eeffd640 1 -- 172.21.15.70:0/57663287 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f08f8059010 con 0x7f09000ae780 2024-04-22T01:14:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f08e00678c0 msgr2=0x7f08e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f08e00678c0 0x7f08e0069d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08f000b9b0 tx=0x7f08f0008040 comp rx=0 tx=0).stop 2024-04-22T01:14:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 msgr2=0x7f09000aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 0x7f09000aec30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09000cf660 tx=0x7f08f8011c20 comp rx=0 tx=0).stop 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 shutdown_connections 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f08e00678c0 0x7f08e0069d80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09000ae780 0x7f09000aec30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09000b56e0 0x7f09000b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 --2- 172.21.15.70:0/57663287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08e8004020 0x7f09000712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 >> 172.21.15.70:0/57663287 conn(0x7f09000b36c0 msgr2=0x7f090015c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 shutdown_connections 2024-04-22T01:14:47.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.663+0000 7f0904ddb640 1 -- 172.21.15.70:0/57663287 wait complete. 2024-04-22T01:14:47.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:14:47.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 Processor -- start 2024-04-22T01:14:47.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 -- start start 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c145470 0x7f651c071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f651c0cd600 0x7f651c0cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f651c0d0340 0x7f651c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f651c0d0770 con 0x7f651c0cd600 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f651c0cf2b0 con 0x7f651c0d0340 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.815+0000 7f65231e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f651c0721b0 con 0x7f651c145470 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f651c0cd600 0x7f651c0cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f651c0cd600 0x7f651c0cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38852/0 (socket says 172.21.15.70:38852) 2024-04-22T01:14:47.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 -- 172.21.15.70:0/705396045 learned_addr learned my addr 172.21.15.70:0/705396045 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f6520f85640 1 --2- 172.21.15.70:0/705396045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c145470 0x7f651c071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f6521786640 1 --2- 172.21.15.70:0/705396045 >> v2:172.21.15.98:3300/0 conn(0x7f651c0d0340 0x7f651c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 -- 172.21.15.70:0/705396045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c145470 msgr2=0x7f651c071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 --2- 172.21.15.70:0/705396045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c145470 0x7f651c071bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 -- 172.21.15.70:0/705396045 >> v2:172.21.15.98:3300/0 conn(0x7f651c0d0340 msgr2=0x7f651c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 --2- 172.21.15.70:0/705396045 >> v2:172.21.15.98:3300/0 conn(0x7f651c0d0340 0x7f651c15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 -- 172.21.15.70:0/705396045 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c072490 con 0x7f651c0cd600 2024-04-22T01:14:47.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651bfff640 1 --2- 172.21.15.70:0/705396045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c0cfe00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f651c071b60 tx=0x7f651001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc7b3b2c2b2602b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651b7fe640 1 -- 172.21.15.70:0/705396045 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65100b0070 con 0x7f651c0cd600 2024-04-22T01:14:47.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651b7fe640 1 -- 172.21.15.70:0/705396045 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65100ab600 con 0x7f651c0cd600 2024-04-22T01:14:47.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f651b7fe640 1 -- 172.21.15.70:0/705396045 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65100ab9d0 con 0x7f651c0cd600 2024-04-22T01:14:47.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 -- 172.21.15.70:0/705396045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 msgr2=0x7f651c0cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 --2- 172.21.15.70:0/705396045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c0cfe00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f651c071b60 tx=0x7f651001c260 comp rx=0 tx=0).stop 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 -- 172.21.15.70:0/705396045 shutdown_connections 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 --2- 172.21.15.70:0/705396045 >> v2:172.21.15.98:3300/0 conn(0x7f651c0d0340 0x7f651c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 --2- 172.21.15.70:0/705396045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c0cfe00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 --2- 172.21.15.70:0/705396045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c145470 0x7f651c071bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 -- 172.21.15.70:0/705396045 >> 172.21.15.70:0/705396045 conn(0x7f651c06b5f0 msgr2=0x7f651c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 -- 172.21.15.70:0/705396045 shutdown_connections 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.819+0000 7f65231e9640 1 -- 172.21.15.70:0/705396045 wait complete. 2024-04-22T01:14:47.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 Processor -- start 2024-04-22T01:14:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- start start 2024-04-22T01:14:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c166250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c0d0340 0x7f651c15a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f651c145470 0x7f651c15ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0b3cb0 con 0x7f651c0cd600 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0b3b30 con 0x7f651c145470 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f651c0b3e30 con 0x7f651c0d0340 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c166250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c166250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38858/0 (socket says 172.21.15.70:38858) 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 -- 172.21.15.70:0/1171814429 learned_addr learned my addr 172.21.15.70:0/1171814429 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f651bfff640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c0d0340 0x7f651c15a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6521786640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f651c145470 0x7f651c15ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 -- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c0d0340 msgr2=0x7f651c15a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c0d0340 0x7f651c15a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 -- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f651c145470 msgr2=0x7f651c15ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f651c145470 0x7f651c15ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 -- 172.21.15.70:0/1171814429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c1417d0 con 0x7f651c0cd600 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6521786640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f651c145470 0x7f651c15ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f6520f85640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c166250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f650c00d5f0 tx=0x7f650c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c0125f0 con 0x7f651c0cd600 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f650c019070 con 0x7f651c0cd600 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c141ab0 con 0x7f651c0cd600 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c0129b0 con 0x7f651c0cd600 2024-04-22T01:14:47.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c142070 con 0x7f651c0cd600 2024-04-22T01:14:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.823+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651c0af5a0 con 0x7f651c0cd600 2024-04-22T01:14:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.827+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f650c012b50 con 0x7f651c0cd600 2024-04-22T01:14:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.827+0000 7f65197fa640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f65080678c0 0x7f6508069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.827+0000 7f651bfff640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f65080678c0 0x7f6508069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.827+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f650c0861a0 con 0x7f651c0cd600 2024-04-22T01:14:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.827+0000 7f651bfff640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f65080678c0 0x7f6508069d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65100ad000 tx=0x7f6510008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.831+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f650c056bb0 con 0x7f651c0cd600 2024-04-22T01:14:47.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:47.991+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f651c0736a0 con 0x7f651c0cd600 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.723+0000 7f65197fa640 1 -- 172.21.15.70:0/1171814429 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f650c0564b0 con 0x7f651c0cd600 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data_ec' created 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f65080678c0 msgr2=0x7f6508069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f65080678c0 0x7f6508069d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65100ad000 tx=0x7f6510008040 comp rx=0 tx=0).stop 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 msgr2=0x7f651c166250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c166250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f650c00d5f0 tx=0x7f650c00dab0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 shutdown_connections 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f65080678c0 0x7f6508069d80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f651c145470 0x7f651c15ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f651c0d0340 0x7f651c15a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 --2- 172.21.15.70:0/1171814429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f651c0cd600 0x7f651c166250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 >> 172.21.15.70:0/1171814429 conn(0x7f651c06b5f0 msgr2=0x7f651c15c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 shutdown_connections 2024-04-22T01:14:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.727+0000 7f65231e9640 1 -- 172.21.15.70:0/1171814429 wait complete. 2024-04-22T01:14:49.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-22T01:14:49.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 Processor -- start 2024-04-22T01:14:49.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 -- start start 2024-04-22T01:14:49.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc15e0e0 0x7fa3cc06c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3cc0cd1f0 0x7fa3cc0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3cc06b390 con 0x7fa3cc0af8e0 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa3cc0cdbe0 con 0x7fa3cc0cd1f0 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d3993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc067440 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3cc0cd1f0 0x7fa3cc0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d172f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc15e0e0 0x7fa3cc06c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3cc0cd1f0 0x7fa3cc0c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33154/0 (socket says 172.21.15.70:33154) 2024-04-22T01:14:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 -- 172.21.15.70:0/1306319271 learned_addr learned my addr 172.21.15.70:0/1306319271 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 -- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc15e0e0 msgr2=0x7fa3cc06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 --2- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc15e0e0 0x7fa3cc06c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 -- 172.21.15.70:0/1306319271 >> v1:172.21.15.98:6789/0 conn(0x7fa3cc0af8e0 legacy=0x7fa3cc0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.883+0000 7fa3d1f30640 1 -- 172.21.15.70:0/1306319271 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc0676e0 con 0x7fa3cc0cd1f0 2024-04-22T01:14:49.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d1f30640 1 --2- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc0c5ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009870 tx=0x7fa3c801c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccef6c7cc1bc7430 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:49.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3c3fff640 1 -- 172.21.15.70:0/1306319271 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80ab430 con 0x7fa3cc0cd1f0 2024-04-22T01:14:49.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3c3fff640 1 -- 172.21.15.70:0/1306319271 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c80b0070 con 0x7fa3cc0cd1f0 2024-04-22T01:14:49.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3c3fff640 1 -- 172.21.15.70:0/1306319271 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80ab7f0 con 0x7fa3cc0cd1f0 2024-04-22T01:14:49.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 -- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 msgr2=0x7fa3cc0c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:49.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 --2- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc0c5ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009870 tx=0x7fa3c801c560 comp rx=0 tx=0).stop 2024-04-22T01:14:49.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 -- 172.21.15.70:0/1306319271 shutdown_connections 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 --2- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc0c5ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 --2- 172.21.15.70:0/1306319271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc15e0e0 0x7fa3cc06c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 -- 172.21.15.70:0/1306319271 >> 172.21.15.70:0/1306319271 conn(0x7fa3cc099a60 msgr2=0x7fa3cc0b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 -- 172.21.15.70:0/1306319271 shutdown_connections 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.887+0000 7fa3d3993640 1 -- 172.21.15.70:0/1306319271 wait complete. 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 Processor -- start 2024-04-22T01:14:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- start start 2024-04-22T01:14:49.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc0af8e0 0x7fa3cc155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc15cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 0x7fa3cc0ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc16aee0 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc16ad60 con 0x7fa3cc0cd1f0 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc16b060 con 0x7fa3cc0af8e0 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 0x7fa3cc0ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 0x7fa3cc0ae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38874/0 (socket says 172.21.15.70:38874) 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 -- 172.21.15.70:0/257491872 learned_addr learned my addr 172.21.15.70:0/257491872 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d172f640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc0af8e0 0x7fa3cc155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d0f2e640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc15cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 -- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc0af8e0 msgr2=0x7fa3cc155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc0af8e0 0x7fa3cc155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 -- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 msgr2=0x7fa3cc15cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:49.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc15cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 -- 172.21.15.70:0/257491872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc0afea0 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d1f30640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 0x7fa3cc0ae2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80ad040 tx=0x7fa3c8019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80ba070 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c80abe00 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc0b0110 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80b4620 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3cc0b06d0 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.891+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3cc068210 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.895+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa3c80bd070 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.895+0000 7fa3c1ffb640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3b00678f0 0x7fa3b0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:49.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.895+0000 7fa3d172f640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3b00678f0 0x7fa3b0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:49.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.895+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7fa3c811fe00 con 0x7fa3cc15e0e0 2024-04-22T01:14:49.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.895+0000 7fa3d172f640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3b00678f0 0x7fa3b0069db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc156600 tx=0x7fa3c400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:49.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:49.899+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa3c8016150 con 0x7fa3cc15e0e0 2024-04-22T01:14:50.035 INFO:tasks.ceph.mds.c.smithi070.stderr:2024-04-22T01:14:50.031+0000 7f84197ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:14:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.059+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fa3cc0b2af0 con 0x7fa3cc15e0e0 2024-04-22T01:14:50.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.731+0000 7fa3c1ffb640 1 -- 172.21.15.70:0/257491872 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7fa3c80f06b0 con 0x7fa3cc15e0e0 2024-04-22T01:14:50.735 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 4 allow_ec_overwrites to true 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3b00678f0 msgr2=0x7fa3b0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3b00678f0 0x7fa3b0069db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc156600 tx=0x7fa3c400b040 comp rx=0 tx=0).stop 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 msgr2=0x7fa3cc0ae2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 0x7fa3cc0ae2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80ad040 tx=0x7fa3c8019300 comp rx=0 tx=0).stop 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 shutdown_connections 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3b00678f0 0x7fa3b0069db0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3cc15e0e0 0x7fa3cc0ae2f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc0cd1f0 0x7fa3cc15cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 --2- 172.21.15.70:0/257491872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3cc0af8e0 0x7fa3cc155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 >> 172.21.15.70:0/257491872 conn(0x7fa3cc099a60 msgr2=0x7fa3cc15a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 shutdown_connections 2024-04-22T01:14:50.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.735+0000 7fa3d3993640 1 -- 172.21.15.70:0/257491872 wait complete. 2024-04-22T01:14:50.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-22T01:14:50.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8be19d640 1 Processor -- start 2024-04-22T01:14:50.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8be19d640 1 -- start start 2024-04-22T01:14:50.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8be19d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8b8155370 0x7fe8b806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8be19d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b80b3800 con 0x7fe8b80ce920 2024-04-22T01:14:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8be19d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b80b3ae0 con 0x7fe8b815e6e0 2024-04-22T01:14:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8be19d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe8b80cde40 con 0x7fe8b8155370 2024-04-22T01:14:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b6ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe8b80ce920 0x7fe8b8154c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46360/0 (socket says 172.21.15.70:46360) 2024-04-22T01:14:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b6ffd640 1 -- 172.21.15.70:0/280015402 learned_addr learned my addr 172.21.15.70:0/280015402 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/280015402 >> v2:172.21.15.98:3300/0 conn(0x7fe8b8155370 0x7fe8b806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2237431314 0 0) 0x7fe8b80b3800 con 0x7fe8b80ce920 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8a0003530 con 0x7fe8b80ce920 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3407805467 0 0) 0x7fe8a0003530 con 0x7fe8b80ce920 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 >> v2:172.21.15.98:3300/0 conn(0x7fe8b8155370 msgr2=0x7fe8b806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 --2- 172.21.15.70:0/280015402 >> v2:172.21.15.98:3300/0 conn(0x7fe8b8155370 0x7fe8b806d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 >> v1:172.21.15.98:6789/0 conn(0x7fe8b815e6e0 legacy=0x7fe8b815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b80ce120 con 0x7fe8b80ce920 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fe8ac001680 con 0x7fe8b80ce920 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 >> v1:172.21.15.70:6789/0 conn(0x7fe8b80ce920 legacy=0x7fe8b8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00039c0 0x7fe8a0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a00051d0 0x7fe8a000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b80cde40 con 0x7fe8a00051d0 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a000c070 con 0x7fe8a0004810 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a000e480 con 0x7fe8a00039c0 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.899+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a00051d0 0x7fe8a000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b77fe640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00039c0 0x7fe8a0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b6ffd640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 -- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00039c0 msgr2=0x7fe8a0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00039c0 0x7fe8a0003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 -- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a0004810 msgr2=0x7fe8a0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 -- 172.21.15.70:0/280015402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a0003530 con 0x7fe8a00051d0 2024-04-22T01:14:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b6ffd640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 -- 172.21.15.70:0/280015402 reap_dead start 2024-04-22T01:14:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a00051d0 0x7fe8a000b9d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009fc0 tx=0x7fe8b001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7326414fdce2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0009c10 con 0x7fe8a00051d0 2024-04-22T01:14:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b00a5ad0 con 0x7fe8a00051d0 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8b67fc640 1 -- 172.21.15.70:0/280015402 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b00039a0 con 0x7fe8a00051d0 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 -- 172.21.15.70:0/280015402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a00051d0 msgr2=0x7fe8a000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a00051d0 0x7fe8a000b9d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009fc0 tx=0x7fe8b001ccc0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 -- 172.21.15.70:0/280015402 shutdown_connections 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 --2- 172.21.15.70:0/280015402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a00051d0 0x7fe8a000b9d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 -- 172.21.15.70:0/280015402 >> 172.21.15.70:0/280015402 conn(0x7fe8b80b1d40 msgr2=0x7fe8b80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 -- 172.21.15.70:0/280015402 shutdown_connections 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.903+0000 7fe8be19d640 1 -- 172.21.15.70:0/280015402 wait complete. 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 Processor -- start 2024-04-22T01:14:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- start start 2024-04-22T01:14:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00051d0 0x7fe8b8148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8b8159ea0 0x7fe8b815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 0x7fe8b8156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b806f260 con 0x7fe8b81568d0 2024-04-22T01:14:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b806f0e0 con 0x7fe8b8159ea0 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe8b80cedd0 con 0x7fe8a00051d0 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 0x7fe8b8156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 0x7fe8b8156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38894/0 (socket says 172.21.15.70:38894) 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 -- 172.21.15.70:0/1677111262 learned_addr learned my addr 172.21.15.70:0/1677111262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b77fe640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00051d0 0x7fe8b8148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b6ffd640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8b8159ea0 0x7fe8b815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 -- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00051d0 msgr2=0x7fe8b8148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00051d0 0x7fe8b8148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 -- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8b8159ea0 msgr2=0x7fe8b815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8b8159ea0 0x7fe8b815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 -- 172.21.15.70:0/1677111262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b8157620 con 0x7fe8b81568d0 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b7fff640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 0x7fe8b8156d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe8b001d1a0 tx=0x7fe8b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0009c10 con 0x7fe8b81568d0 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b00bd070 con 0x7fe8b81568d0 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b8157900 con 0x7fe8b81568d0 2024-04-22T01:14:50.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0003be0 con 0x7fe8b81568d0 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b815c1d0 con 0x7fe8b81568d0 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b8068210 con 0x7fe8b81568d0 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe8b0004030 con 0x7fe8b81568d0 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe89ffff640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a40678c0 0x7fe8a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fe8b0121a00 con 0x7fe8b81568d0 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.907+0000 7fe8b77fe640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a40678c0 0x7fe8a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:50.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.911+0000 7fe8b77fe640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a40678c0 0x7fe8a4069d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8001720 tx=0x7fe8a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:50.915+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe8b00c7020 con 0x7fe8b81568d0 2024-04-22T01:14:51.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.115+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe8b80d08e0 con 0x7fe8b81568d0 2024-04-22T01:14:51.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.739+0000 7fe89ffff640 1 -- 172.21.15.70:0/1677111262 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7fe8b00169d0 con 0x7fe8b81568d0 2024-04-22T01:14:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:added data pool 4 to fsmap 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a40678c0 msgr2=0x7fe8a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a40678c0 0x7fe8a4069d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8001720 tx=0x7fe8a8009040 comp rx=0 tx=0).stop 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 msgr2=0x7fe8b8156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 0x7fe8b8156d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe8b001d1a0 tx=0x7fe8b0009750 comp rx=0 tx=0).stop 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 shutdown_connections 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a40678c0 0x7fe8a4069d80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8b81568d0 0x7fe8b8156d60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8b8159ea0 0x7fe8b815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 --2- 172.21.15.70:0/1677111262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a00051d0 0x7fe8b8148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 >> 172.21.15.70:0/1677111262 conn(0x7fe8b80b1d40 msgr2=0x7fe8b815eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 shutdown_connections 2024-04-22T01:14:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:51.743+0000 7fe8be19d640 1 -- 172.21.15.70:0/1677111262 wait complete. 2024-04-22T01:14:51.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:14:52.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.015+0000 7f45f97b5640 1 Processor -- start 2024-04-22T01:14:52.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.015+0000 7f45f97b5640 1 -- start start 2024-04-22T01:14:52.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f97b5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45f415be40 0x7f45f40673a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f97b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f40ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f97b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f40cffd0 con 0x7f45f40ad610 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f97b5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f45f40678e0 con 0x7f45f415be40 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f97b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45f40cfd80 con 0x7f45f40b5d20 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45f415be40 0x7f45f40673a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45ea7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f40ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45f415be40 0x7f45f40673a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33206/0 (socket says 172.21.15.70:33206) 2024-04-22T01:14:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 -- 172.21.15.70:0/4136834460 learned_addr learned my addr 172.21.15.70:0/4136834460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:52.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 -- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 msgr2=0x7f45f40ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:52.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 --2- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f40ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 -- 172.21.15.70:0/4136834460 >> v1:172.21.15.98:6789/0 conn(0x7f45f40ad610 legacy=0x7f45f415f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.019+0000 7f45f2ffd640 1 -- 172.21.15.70:0/4136834460 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f40d02b0 con 0x7f45f415be40 2024-04-22T01:14:52.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f2ffd640 1 --2- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f415be40 0x7f45f40673a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45e40098a0 tx=0x7f45e401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ce5b0a5c3cb15d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f27fc640 1 -- 172.21.15.70:0/4136834460 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e40ab650 con 0x7f45f415be40 2024-04-22T01:14:52.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f27fc640 1 -- 172.21.15.70:0/4136834460 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45e40b0070 con 0x7f45f415be40 2024-04-22T01:14:52.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f27fc640 1 -- 172.21.15.70:0/4136834460 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e40aba10 con 0x7f45f415be40 2024-04-22T01:14:52.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 -- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f415be40 msgr2=0x7f45f40673a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f415be40 0x7f45f40673a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45e40098a0 tx=0x7f45e401c540 comp rx=0 tx=0).stop 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 -- 172.21.15.70:0/4136834460 shutdown_connections 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f40ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4136834460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f415be40 0x7f45f40673a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 -- 172.21.15.70:0/4136834460 >> 172.21.15.70:0/4136834460 conn(0x7f45f4099a60 msgr2=0x7f45f40b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 -- 172.21.15.70:0/4136834460 shutdown_connections 2024-04-22T01:14:52.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.023+0000 7f45f97b5640 1 -- 172.21.15.70:0/4136834460 wait complete. 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 Processor -- start 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- start start 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40ad610 0x7f45f41375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f4135ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 0x7f45f4135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45f4168c30 con 0x7f45f415be40 2024-04-22T01:14:52.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45f40ada20 con 0x7f45f40ad610 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45f4168db0 con 0x7f45f40b5d20 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 0x7f45f4135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 0x7f45f4135430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38898/0 (socket says 172.21.15.70:38898) 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 -- 172.21.15.70:0/4232633842 learned_addr learned my addr 172.21.15.70:0/4232633842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f2ffd640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40ad610 0x7f45f41375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45ea7fc640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f4135ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 -- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 msgr2=0x7f45f4135ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f4135ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 -- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40ad610 msgr2=0x7f45f41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40ad610 0x7f45f41375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 -- 172.21.15.70:0/4232633842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f4137b10 con 0x7f45f415be40 2024-04-22T01:14:52.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f37fe640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 0x7f45f4135430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c770 tx=0x7f45ec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec003b80 con 0x7f45f415be40 2024-04-22T01:14:52.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45ec003d20 con 0x7f45f415be40 2024-04-22T01:14:52.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f4136530 con 0x7f45f415be40 2024-04-22T01:14:52.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec00aa90 con 0x7f45f415be40 2024-04-22T01:14:52.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f4136af0 con 0x7f45f415be40 2024-04-22T01:14:52.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f40ae1f0 con 0x7f45f415be40 2024-04-22T01:14:52.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.027+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f45ec00ac30 con 0x7f45f415be40 2024-04-22T01:14:52.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.031+0000 7f45ebfff640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45d40678c0 0x7f45d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.031+0000 7f45f2ffd640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45d40678c0 0x7f45d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.031+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f45ec085e50 con 0x7f45f415be40 2024-04-22T01:14:52.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.031+0000 7f45f2ffd640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45d40678c0 0x7f45d4069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f45e40af000 tx=0x7f45e40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.035+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f45ec0566e0 con 0x7f45f415be40 2024-04-22T01:14:52.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.227+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45f40b5970 con 0x7f45f415be40 2024-04-22T01:14:52.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.227+0000 7f45ebfff640 1 -- 172.21.15.70:0/4232633842 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5177 (secure 0 0 0) 0x7f45ec055fe0 con 0x7f45f415be40 2024-04-22T01:14:52.227 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:52.227 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:14:51.117946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:14:52.227 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 7 2024-04-22T01:14:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45d40678c0 msgr2=0x7f45d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45d40678c0 0x7f45d4069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f45e40af000 tx=0x7f45e40a7000 comp rx=0 tx=0).stop 2024-04-22T01:14:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 msgr2=0x7f45f4135430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 0x7f45f4135430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c770 tx=0x7f45ec00cc30 comp rx=0 tx=0).stop 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 shutdown_connections 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45d40678c0 0x7f45d4069d80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f415be40 0x7f45f4135430 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45f40b5d20 0x7f45f4135ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 --2- 172.21.15.70:0/4232633842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45f40ad610 0x7f45f41375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 >> 172.21.15.70:0/4232633842 conn(0x7f45f4099a60 msgr2=0x7f45f40af160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 shutdown_connections 2024-04-22T01:14:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.231+0000 7f45f97b5640 1 -- 172.21.15.70:0/4232633842 wait complete. 2024-04-22T01:14:52.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.367+0000 7f478a460640 1 Processor -- start 2024-04-22T01:14:52.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.367+0000 7f478a460640 1 -- start start 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f478a460640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47841455b0 0x7f47840cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f478a460640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47840cc150 0x7f4784071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f478a460640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4784072060 0x7f478415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f478a460640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f47840724b0 con 0x7f47840cc150 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f478a460640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f47840d0250 con 0x7f47841455b0 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f478a460640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f47840cd7b0 con 0x7f4784072060 2024-04-22T01:14:52.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47840cc150 0x7f4784071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47840cc150 0x7f4784071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38902/0 (socket says 172.21.15.70:38902) 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 -- 172.21.15.70:0/1523719635 learned_addr learned my addr 172.21.15.70:0/1523719635 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47889fd640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4784072060 0x7f478415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f4783fff640 1 --2- 172.21.15.70:0/1523719635 >> v2:172.21.15.98:3300/0 conn(0x7f47841455b0 0x7f47840cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 -- 172.21.15.70:0/1523719635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4784072060 msgr2=0x7f478415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4784072060 0x7f478415f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 -- 172.21.15.70:0/1523719635 >> v2:172.21.15.98:3300/0 conn(0x7f47841455b0 msgr2=0x7f47840cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 --2- 172.21.15.70:0/1523719635 >> v2:172.21.15.98:3300/0 conn(0x7f47841455b0 0x7f47840cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 -- 172.21.15.70:0/1523719635 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47840cc710 con 0x7f47840cc150 2024-04-22T01:14:52.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47889fd640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4784072060 0x7f478415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:52.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f47837fe640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47840cc150 0x7f4784071b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f47840d0700 tx=0x7f477801c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60da1615988f6bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f4782ffd640 1 -- 172.21.15.70:0/1523719635 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47780b1070 con 0x7f47840cc150 2024-04-22T01:14:52.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f4782ffd640 1 -- 172.21.15.70:0/1523719635 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47780ac440 con 0x7f47840cc150 2024-04-22T01:14:52.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.371+0000 7f4782ffd640 1 -- 172.21.15.70:0/1523719635 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47780ac810 con 0x7f47840cc150 2024-04-22T01:14:52.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/1523719635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47840cc150 msgr2=0x7f4784071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47840cc150 0x7f4784071b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f47840d0700 tx=0x7f477801c140 comp rx=0 tx=0).stop 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/1523719635 shutdown_connections 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4784072060 0x7f478415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- 172.21.15.70:0/1523719635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47840cc150 0x7f4784071b20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- 172.21.15.70:0/1523719635 >> v2:172.21.15.98:3300/0 conn(0x7f47841455b0 0x7f47840cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/1523719635 >> 172.21.15.70:0/1523719635 conn(0x7f478406b1f0 msgr2=0x7f478406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/1523719635 shutdown_connections 2024-04-22T01:14:52.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/1523719635 wait complete. 2024-04-22T01:14:52.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 Processor -- start 2024-04-22T01:14:52.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- start start 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 0x7f4784161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47840cc150 0x7f4784162470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47841455b0 0x7f478413f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f478406e380 con 0x7f4784072060 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f478406e200 con 0x7f47840cc150 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f478406e500 con 0x7f47841455b0 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 0x7f4784161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 0x7f4784161270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38918/0 (socket says 172.21.15.70:38918) 2024-04-22T01:14:52.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 -- 172.21.15.70:0/441182255 learned_addr learned my addr 172.21.15.70:0/441182255 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f47837fe640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47840cc150 0x7f4784162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f47889fd640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47841455b0 0x7f478413f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 -- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47841455b0 msgr2=0x7f478413f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47841455b0 0x7f478413f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 -- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47840cc150 msgr2=0x7f4784162470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47840cc150 0x7f4784162470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 -- 172.21.15.70:0/441182255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478415a8a0 con 0x7f4784072060 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4783fff640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 0x7f4784161270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f477400d620 tx=0x7f477400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477400b620 con 0x7f4784072060 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4774019070 con 0x7f4784072060 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478415ab80 con 0x7f4784072060 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477400b9e0 con 0x7f4784072060 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.375+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47841617b0 con 0x7f4784072060 2024-04-22T01:14:52.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.379+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4784068210 con 0x7f4784072060 2024-04-22T01:14:52.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.379+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f477400bb80 con 0x7f4784072060 2024-04-22T01:14:52.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.379+0000 7f4780ff9640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47700678c0 0x7f4770069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.379+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4774086390 con 0x7f4784072060 2024-04-22T01:14:52.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.379+0000 7f47837fe640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47700678c0 0x7f4770069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.379+0000 7f47837fe640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47700678c0 0x7f4770069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f47780ae000 tx=0x7f47780a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.383+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4774056c20 con 0x7f4784072060 2024-04-22T01:14:52.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.543+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f478415f510 con 0x7f4784072060 2024-04-22T01:14:52.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f4780ff9640 1 -- 172.21.15.70:0/441182255 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f4774056520 con 0x7f4784072060 2024-04-22T01:14:52.545 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:52.545 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47700678c0 msgr2=0x7f4770069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47700678c0 0x7f4770069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f47780ae000 tx=0x7f47780a6070 comp rx=0 tx=0).stop 2024-04-22T01:14:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 msgr2=0x7f4784161270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 0x7f4784161270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f477400d620 tx=0x7f477400dae0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 shutdown_connections 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47700678c0 0x7f4770069d80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47841455b0 0x7f478413f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47840cc150 0x7f4784162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 --2- 172.21.15.70:0/441182255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4784072060 0x7f4784161270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 >> 172.21.15.70:0/441182255 conn(0x7f478406b1f0 msgr2=0x7f47840b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 shutdown_connections 2024-04-22T01:14:52.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.547+0000 7f478a460640 1 -- 172.21.15.70:0/441182255 wait complete. 2024-04-22T01:14:52.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:52.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 Processor -- start 2024-04-22T01:14:52.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 -- start start 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b00cc150 0x7fc9b0071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc9b0072060 0x7fc9b006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b00cd7b0 con 0x7fc9b01455b0 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b00cc710 con 0x7fc9b0072060 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9b55e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b00cc9f0 con 0x7fc9b00cc150 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9af7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc9b0072060 0x7fc9b006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9af7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc9b0072060 0x7fc9b006e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38934/0 (socket says 172.21.15.70:38934) 2024-04-22T01:14:52.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.687+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3626384748 learned_addr learned my addr 172.21.15.70:0/3626384748 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:52.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9a67fc640 1 --2- 172.21.15.70:0/3626384748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b00cc150 0x7fc9b0071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3626384748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b00cc150 msgr2=0x7fc9b0071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:52.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9af7fe640 1 --2- 172.21.15.70:0/3626384748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b00cc150 0x7fc9b0071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3626384748 >> v1:172.21.15.98:6789/0 conn(0x7fc9b01455b0 legacy=0x7fc9b00cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9af7fe640 1 -- 172.21.15.70:0/3626384748 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b00cccd0 con 0x7fc9b0072060 2024-04-22T01:14:52.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9af7fe640 1 --2- 172.21.15.70:0/3626384748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b0072060 0x7fc9b006e200 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009e50 tx=0x7fc9a801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b48b63a186e852 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9ae7fc640 1 -- 172.21.15.70:0/3626384748 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a80a55d0 con 0x7fc9b0072060 2024-04-22T01:14:52.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9ae7fc640 1 -- 172.21.15.70:0/3626384748 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a80a5770 con 0x7fc9b0072060 2024-04-22T01:14:52.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9ae7fc640 1 -- 172.21.15.70:0/3626384748 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a80a5b40 con 0x7fc9b0072060 2024-04-22T01:14:52.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 -- 172.21.15.70:0/3626384748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b0072060 msgr2=0x7fc9b006e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/3626384748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b0072060 0x7fc9b006e200 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009e50 tx=0x7fc9a801c910 comp rx=0 tx=0).stop 2024-04-22T01:14:52.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 -- 172.21.15.70:0/3626384748 shutdown_connections 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/3626384748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b0072060 0x7fc9b006e200 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/3626384748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b00cc150 0x7fc9b0071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 -- 172.21.15.70:0/3626384748 >> 172.21.15.70:0/3626384748 conn(0x7fc9b006b1f0 msgr2=0x7fc9b006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 -- 172.21.15.70:0/3626384748 shutdown_connections 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.691+0000 7fc9b55e0640 1 -- 172.21.15.70:0/3626384748 wait complete. 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 Processor -- start 2024-04-22T01:14:52.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- start start 2024-04-22T01:14:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072060 0x7fc9b013b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 0x7fc9b0142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b01455b0 0x7fc9b01433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b006f2e0 con 0x7fc9b00cc150 2024-04-22T01:14:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b006f160 con 0x7fc9b0072060 2024-04-22T01:14:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b0168b40 con 0x7fc9b01455b0 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 0x7fc9b0142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 0x7fc9b0142e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38948/0 (socket says 172.21.15.70:38948) 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 -- 172.21.15.70:0/2573190512 learned_addr learned my addr 172.21.15.70:0/2573190512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9aeffd640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072060 0x7fc9b013b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9af7fe640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b01455b0 0x7fc9b01433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 -- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b01455b0 msgr2=0x7fc9b01433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b01455b0 0x7fc9b01433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 -- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072060 msgr2=0x7fc9b013b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072060 0x7fc9b013b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 -- 172.21.15.70:0/2573190512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b0159600 con 0x7fc9b00cc150 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9aeffd640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072060 0x7fc9b013b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a67fc640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 0x7fc9b0142e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc99800d5f0 tx=0x7fc99800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9980125f0 con 0x7fc9b00cc150 2024-04-22T01:14:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc998019070 con 0x7fc9b00cc150 2024-04-22T01:14:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b0157fe0 con 0x7fc9b00cc150 2024-04-22T01:14:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9980129b0 con 0x7fc9b00cc150 2024-04-22T01:14:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b01585a0 con 0x7fc9b00cc150 2024-04-22T01:14:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b0068210 con 0x7fc9b00cc150 2024-04-22T01:14:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.695+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc998012b50 con 0x7fc9b00cc150 2024-04-22T01:14:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.699+0000 7fc9a7fff640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9940678c0 0x7fc994069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.699+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc998086e80 con 0x7fc9b00cc150 2024-04-22T01:14:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.699+0000 7fc9aeffd640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9940678c0 0x7fc994069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.699+0000 7fc9aeffd640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9940678c0 0x7fc994069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00016f0 tx=0x7fc9a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:52.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.699+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc998013390 con 0x7fc9b00cc150 2024-04-22T01:14:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.863+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9b015f510 con 0x7fc9b00cc150 2024-04-22T01:14:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9a7fff640 1 -- 172.21.15.70:0/2573190512 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7fc9980576f0 con 0x7fc9b00cc150 2024-04-22T01:14:52.866 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:52.866 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9940678c0 msgr2=0x7fc994069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9940678c0 0x7fc994069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00016f0 tx=0x7fc9a0009040 comp rx=0 tx=0).stop 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 msgr2=0x7fc9b0142e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 0x7fc9b0142e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc99800d5f0 tx=0x7fc99800dab0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 shutdown_connections 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9940678c0 0x7fc994069d80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b01455b0 0x7fc9b01433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9b00cc150 0x7fc9b0142e90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 --2- 172.21.15.70:0/2573190512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072060 0x7fc9b013b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.867+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 >> 172.21.15.70:0/2573190512 conn(0x7fc9b006b1f0 msgr2=0x7fc9b00b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.871+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 shutdown_connections 2024-04-22T01:14:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:52.871+0000 7fc9b55e0640 1 -- 172.21.15.70:0/2573190512 wait complete. 2024-04-22T01:14:52.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:14:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c936fd640 1 Processor -- start 2024-04-22T01:14:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c936fd640 1 -- start start 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c936fd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6c8c15dfb0 0x7f6c8c0b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c936fd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c0b6550 con 0x7f6c8c0d3120 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c936fd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c15e390 con 0x7f6c8c0cea80 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c936fd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6c8c161e00 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c91c9a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6c8c0d3120 0x7f6c8c0b5e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46376/0 (socket says 172.21.15.70:46376) 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c91c9a640 1 -- 172.21.15.70:0/3014437737 learned_addr learned my addr 172.21.15.70:0/3014437737 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c91499640 1 --2- 172.21.15.70:0/3014437737 >> v2:172.21.15.70:3300/0 conn(0x7f6c8c15dfb0 0x7f6c8c0b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c83fff640 1 -- 172.21.15.70:0/3014437737 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1105871005 0 0) 0x7f6c8c0b6550 con 0x7f6c8c0d3120 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c83fff640 1 -- 172.21.15.70:0/3014437737 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c78003530 con 0x7f6c8c0d3120 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c91499640 1 -- 172.21.15.70:0/3014437737 >> v1:172.21.15.98:6789/0 conn(0x7f6c8c0cea80 legacy=0x7f6c8c0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:53.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c91499640 1 -- 172.21.15.70:0/3014437737 >> v1:172.21.15.70:6789/0 conn(0x7f6c8c0d3120 legacy=0x7f6c8c0b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.023+0000 7f6c91499640 1 -- 172.21.15.70:0/3014437737 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c1620e0 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c91499640 1 --2- 172.21.15.70:0/3014437737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c0b95d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009d90 tx=0x7f6c8401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e888e35543f2eff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c83fff640 1 -- 172.21.15.70:0/3014437737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840b2070 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c83fff640 1 -- 172.21.15.70:0/3014437737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c840a5940 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c83fff640 1 -- 172.21.15.70:0/3014437737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840a5d10 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- 172.21.15.70:0/3014437737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 msgr2=0x7f6c8c0b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3014437737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c0b95d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009d90 tx=0x7f6c8401c970 comp rx=0 tx=0).stop 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- 172.21.15.70:0/3014437737 shutdown_connections 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3014437737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c0b95d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- 172.21.15.70:0/3014437737 >> 172.21.15.70:0/3014437737 conn(0x7f6c8c06c9b0 msgr2=0x7f6c8c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- 172.21.15.70:0/3014437737 shutdown_connections 2024-04-22T01:14:53.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- 172.21.15.70:0/3014437737 wait complete. 2024-04-22T01:14:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 Processor -- start 2024-04-22T01:14:53.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- start start 2024-04-22T01:14:53.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c0cea80 0x7f6c8c15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c0d3120 0x7f6c8c0c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0cf270 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c161a90 con 0x7f6c8c0cea80 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c936fd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c0cf3f0 con 0x7f6c8c0d3120 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c91c9a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c91c9a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38960/0 (socket says 172.21.15.70:38960) 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c91c9a640 1 -- 172.21.15.70:0/3178985645 learned_addr learned my addr 172.21.15.70:0/3178985645 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c90c98640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c0d3120 0x7f6c8c0c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.027+0000 7f6c91499640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c0cea80 0x7f6c8c15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91c9a640 1 -- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c0d3120 msgr2=0x7f6c8c0c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91c9a640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c0d3120 0x7f6c8c0c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91c9a640 1 -- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c0cea80 msgr2=0x7f6c8c15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91c9a640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c0cea80 0x7f6c8c15c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91c9a640 1 -- 172.21.15.70:0/3178985645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c0c8900 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91499640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c0cea80 0x7f6c8c15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91c9a640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c1550c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c390 tx=0x7f6c7c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c017070 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c7c005cf0 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c0c8be0 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c0072d0 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c0c91a0 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8c068210 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c7c01a070 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c81ffb640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c700678c0 0x7f6c70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6c7c0853b0 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91499640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c700678c0 0x7f6c70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.031+0000 7f6c91499640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c700678c0 0x7f6c70069d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c840af000 tx=0x7f6c840a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:53.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.035+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6c7c055c70 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.223+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c8c0d1950 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.223+0000 7f6c81ffb640 1 -- 172.21.15.70:0/3178985645 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5177 (secure 0 0 0) 0x7f6c7c055570 con 0x7f6c8c15dfb0 2024-04-22T01:14:53.223 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:53.224 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:14:51.117946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:14:53.224 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 7 2024-04-22T01:14:53.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c700678c0 msgr2=0x7f6c70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c700678c0 0x7f6c70069d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c840af000 tx=0x7f6c840a7070 comp rx=0 tx=0).stop 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 msgr2=0x7f6c8c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c1550c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c390 tx=0x7f6c7c00c850 comp rx=0 tx=0).stop 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 shutdown_connections 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c700678c0 0x7f6c70069d80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c15dfb0 0x7f6c8c1550c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c0d3120 0x7f6c8c0c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 --2- 172.21.15.70:0/3178985645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c0cea80 0x7f6c8c15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 >> 172.21.15.70:0/3178985645 conn(0x7f6c8c06c9b0 msgr2=0x7f6c8c0d3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 shutdown_connections 2024-04-22T01:14:53.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.227+0000 7f6c936fd640 1 -- 172.21.15.70:0/3178985645 wait complete. 2024-04-22T01:14:53.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.347+0000 7f9b5d391640 1 Processor -- start 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.347+0000 7f9b5d391640 1 -- start start 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9b580cc150 0x7f9b58071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b580cd7b0 con 0x7f9b581455b0 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b580cc710 con 0x7f9b58072060 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9b58072490 con 0x7f9b580cc150 2024-04-22T01:14:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b56ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9b581455b0 0x7f9b580cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46380/0 (socket says 172.21.15.70:46380) 2024-04-22T01:14:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b56ffd640 1 -- 172.21.15.70:0/101453441 learned_addr learned my addr 172.21.15.70:0/101453441 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b567fc640 1 --2- 172.21.15.70:0/101453441 >> v2:172.21.15.70:3300/0 conn(0x7f9b580cc150 0x7f9b58071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b567fc640 1 -- 172.21.15.70:0/101453441 >> v1:172.21.15.98:6789/0 conn(0x7f9b58072060 legacy=0x7f9b5806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b567fc640 1 -- 172.21.15.70:0/101453441 >> v1:172.21.15.70:6789/0 conn(0x7f9b581455b0 legacy=0x7f9b580cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b567fc640 1 -- 172.21.15.70:0/101453441 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b580cc9f0 con 0x7f9b580cc150 2024-04-22T01:14:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b567fc640 1 --2- 172.21.15.70:0/101453441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580cc150 0x7f9b58071b20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0097c0 tx=0x7f9b4c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad268ccbc733941f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b55ffb640 1 -- 172.21.15.70:0/101453441 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c0b0070 con 0x7f9b580cc150 2024-04-22T01:14:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b55ffb640 1 -- 172.21.15.70:0/101453441 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4c0ab600 con 0x7f9b580cc150 2024-04-22T01:14:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b55ffb640 1 -- 172.21.15.70:0/101453441 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c0ab9d0 con 0x7f9b580cc150 2024-04-22T01:14:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- 172.21.15.70:0/101453441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580cc150 msgr2=0x7f9b58071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 --2- 172.21.15.70:0/101453441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580cc150 0x7f9b58071b20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0097c0 tx=0x7f9b4c01c340 comp rx=0 tx=0).stop 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- 172.21.15.70:0/101453441 shutdown_connections 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 --2- 172.21.15.70:0/101453441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580cc150 0x7f9b58071b20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- 172.21.15.70:0/101453441 >> 172.21.15.70:0/101453441 conn(0x7f9b5806b1f0 msgr2=0x7f9b5806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- 172.21.15.70:0/101453441 shutdown_connections 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.351+0000 7f9b5d391640 1 -- 172.21.15.70:0/101453441 wait complete. 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 Processor -- start 2024-04-22T01:14:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- start start 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b58072060 0x7f9b5813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b581455b0 0x7f9b5815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 0x7f9b5815e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58169ee0 con 0x7f9b5815ab20 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58169d60 con 0x7f9b58072060 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9b5816a060 con 0x7f9b581455b0 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 0x7f9b5815e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 0x7f9b5815e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38972/0 (socket says 172.21.15.70:38972) 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 -- 172.21.15.70:0/422137711 learned_addr learned my addr 172.21.15.70:0/422137711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b567fc640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b581455b0 0x7f9b5815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b56ffd640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b58072060 0x7f9b5813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 -- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b581455b0 msgr2=0x7f9b5815a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b581455b0 0x7f9b5815a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 -- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b58072060 msgr2=0x7f9b5813ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b58072060 0x7f9b5813ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 -- 172.21.15.70:0/422137711 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5813ef40 con 0x7f9b5815ab20 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b577fe640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 0x7f9b5815e6d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800b990 tx=0x7f9b4800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48019070 con 0x7f9b5815ab20 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4800cba0 con 0x7f9b5815ab20 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48014440 con 0x7f9b5815ab20 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b580b01c0 con 0x7f9b5815ab20 2024-04-22T01:14:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b58158320 con 0x7f9b5815ab20 2024-04-22T01:14:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.355+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b58068210 con 0x7f9b5815ab20 2024-04-22T01:14:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.359+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b4801c070 con 0x7f9b5815ab20 2024-04-22T01:14:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.359+0000 7f9b3f7fe640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b440678c0 0x7f9b44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.359+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9b48085d00 con 0x7f9b5815ab20 2024-04-22T01:14:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.359+0000 7f9b56ffd640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b440678c0 0x7f9b44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.359+0000 7f9b56ffd640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b440678c0 0x7f9b44069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b50000f70 tx=0x7f9b50009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.363+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9b48056500 con 0x7f9b5815ab20 2024-04-22T01:14:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.511+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b5815f510 con 0x7f9b5815ab20 2024-04-22T01:14:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.511+0000 7f9b3f7fe640 1 -- 172.21.15.70:0/422137711 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f9b48055e00 con 0x7f9b5815ab20 2024-04-22T01:14:53.511 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:53.511 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b440678c0 msgr2=0x7f9b44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b440678c0 0x7f9b44069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b50000f70 tx=0x7f9b50009040 comp rx=0 tx=0).stop 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 msgr2=0x7f9b5815e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 0x7f9b5815e6d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800b990 tx=0x7f9b4800be50 comp rx=0 tx=0).stop 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 shutdown_connections 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b440678c0 0x7f9b44069d80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5815ab20 0x7f9b5815e6d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b581455b0 0x7f9b5815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 --2- 172.21.15.70:0/422137711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b58072060 0x7f9b5813ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 >> 172.21.15.70:0/422137711 conn(0x7f9b5806b1f0 msgr2=0x7f9b580af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 shutdown_connections 2024-04-22T01:14:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:53.515+0000 7f9b5d391640 1 -- 172.21.15.70:0/422137711 wait complete. 2024-04-22T01:14:53.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:14:46.030124+0000', 'modified': '2024-04-22T01:14:51.117946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4251': {'gid': 4251, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/3139539459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6835', 'nonce': 3139539459}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3139539459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4251, 'qdb_cluster': [4251]} 2024-04-22T01:14:53.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:14:53.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T01:14:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.063+0000 7fc2cbe8a640 1 Processor -- start 2024-04-22T01:14:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.063+0000 7fc2cbe8a640 1 -- start start 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2c415bd10 0x7fc2c40b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2c406b1f0 0x7fc2c406b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c40b2960 con 0x7fc2c40b5fb0 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc2c40b2c40 con 0x7fc2c415bd10 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc2c40b5270 con 0x7fc2c406b1f0 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2c415bd10 0x7fc2c40b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2c415bd10 0x7fc2c40b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38980/0 (socket says 172.21.15.70:38980) 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3611515280 learned_addr learned my addr 172.21.15.70:0/3611515280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2ca427640 1 --2- 172.21.15.70:0/3611515280 >> v2:172.21.15.98:3300/0 conn(0x7fc2c406b1f0 0x7fc2c406b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3611515280 >> v2:172.21.15.98:3300/0 conn(0x7fc2c406b1f0 msgr2=0x7fc2c406b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/3611515280 >> v2:172.21.15.98:3300/0 conn(0x7fc2c406b1f0 0x7fc2c406b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3611515280 >> v1:172.21.15.98:6789/0 conn(0x7fc2c40b5fb0 legacy=0x7fc2c40cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3611515280 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c40b5550 con 0x7fc2c415bd10 2024-04-22T01:14:54.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2ca427640 1 --2- 172.21.15.70:0/3611515280 >> v2:172.21.15.98:3300/0 conn(0x7fc2c406b1f0 0x7fc2c406b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:54.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/3611515280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c415bd10 0x7fc2c40b5a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4071ef0 tx=0x7fc2c001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16305aa4a85cfb1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c8c24640 1 -- 172.21.15.70:0/3611515280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00ab5f0 con 0x7fc2c415bd10 2024-04-22T01:14:54.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c8c24640 1 -- 172.21.15.70:0/3611515280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00b0070 con 0x7fc2c415bd10 2024-04-22T01:14:54.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2c8c24640 1 -- 172.21.15.70:0/3611515280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00ab9b0 con 0x7fc2c415bd10 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3611515280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c415bd10 msgr2=0x7fc2c40b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3611515280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c415bd10 0x7fc2c40b5a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4071ef0 tx=0x7fc2c001c4e0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3611515280 shutdown_connections 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3611515280 >> v2:172.21.15.98:3300/0 conn(0x7fc2c406b1f0 0x7fc2c406b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3611515280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c415bd10 0x7fc2c40b5a70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3611515280 >> 172.21.15.70:0/3611515280 conn(0x7fc2c4099a60 msgr2=0x7fc2c40b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3611515280 shutdown_connections 2024-04-22T01:14:54.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.067+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3611515280 wait complete. 2024-04-22T01:14:54.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 Processor -- start 2024-04-22T01:14:54.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 -- start start 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 0x7fc2c4152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c40b5fb0 0x7fc2c41553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2c415bd10 0x7fc2c415f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c41689f0 con 0x7fc2c406b1f0 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c40b2610 con 0x7fc2c40b5fb0 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c4168b70 con 0x7fc2c415bd10 2024-04-22T01:14:54.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 0x7fc2c4152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 0x7fc2c4152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38992/0 (socket says 172.21.15.70:38992) 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3289735202 learned_addr learned my addr 172.21.15.70:0/3289735202 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9425640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c40b5fb0 0x7fc2c41553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2c415bd10 msgr2=0x7fc2c415f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2ca427640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2c415bd10 0x7fc2c415f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2c415bd10 0x7fc2c415f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c40b5fb0 msgr2=0x7fc2c41553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c40b5fb0 0x7fc2c41553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 -- 172.21.15.70:0/3289735202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c40adbf0 con 0x7fc2c406b1f0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9425640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c40b5fb0 0x7fc2c41553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 0x7fc2c4152e80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc2c001a110 tx=0x7fc2c001ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2ba7fc640 1 -- 172.21.15.70:0/3289735202 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00be5f0 con 0x7fc2c406b1f0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2ba7fc640 1 -- 172.21.15.70:0/3289735202 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00be790 con 0x7fc2c406b1f0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c40aded0 con 0x7fc2c406b1f0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2ba7fc640 1 -- 172.21.15.70:0/3289735202 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00beb80 con 0x7fc2c406b1f0 2024-04-22T01:14:54.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.071+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c41548e0 con 0x7fc2c406b1f0 2024-04-22T01:14:54.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2ba7fc640 1 -- 172.21.15.70:0/3289735202 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2c00b43f0 con 0x7fc2c406b1f0 2024-04-22T01:14:54.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2ba7fc640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2a80678c0 0x7fc2a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2ba7fc640 1 -- 172.21.15.70:0/3289735202 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2c0120a60 con 0x7fc2c406b1f0 2024-04-22T01:14:54.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9425640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2a80678c0 0x7fc2a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9425640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2a80678c0 0x7fc2a8069d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4154120 tx=0x7fc2b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 Processor -- start 2024-04-22T01:14:54.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- start start 2024-04-22T01:14:54.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 0x55b95afba2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95af70680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95af70bc0 0x55b95af70fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b95adbe5e0 con 0x55b95afb9ef0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b95ad45c50 con 0x55b95afba810 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55b95adba5a0 con 0x55b95af70bc0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 0x55b95afba2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 0x55b95afba2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39002/0 (socket says 172.21.15.70:39002) 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4149717946 learned_addr learned my addr 172.21.15.70:0/4149717946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4149717946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95af70bc0 msgr2=0x55b95af70fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95af70bc0 0x55b95af70fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4149717946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 msgr2=0x55b95af70680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95af70680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4149717946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95ad43d10 con 0x55b95afb9ef0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 0x55b95afba2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00f9320 tx=0x7fc2c0019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c383b998eca714 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2a77fe640 1 -- 172.21.15.70:0/4149717946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c011d070 con 0x55b95afb9ef0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2a77fe640 1 -- 172.21.15.70:0/4149717946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00b0020 con 0x55b95afb9ef0 2024-04-22T01:14:54.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2a77fe640 1 -- 172.21.15.70:0/4149717946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00f9850 con 0x55b95afb9ef0 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- 172.21.15.70:0/4149717946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 msgr2=0x55b95afba2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 0x55b95afba2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00f9320 tx=0x7fc2c0019280 comp rx=0 tx=0).stop 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- 172.21.15.70:0/4149717946 shutdown_connections 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95af70bc0 0x55b95af70fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95af70680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4149717946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95afb9ef0 0x55b95afba2d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- 172.21.15.70:0/4149717946 >> 172.21.15.70:0/4149717946 conn(0x55b95af6cb70 msgr2=0x55b95af6ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- 172.21.15.70:0/4149717946 shutdown_connections 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.075+0000 7fc2cf003000 1 -- 172.21.15.70:0/4149717946 wait complete. 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- start start 2024-04-22T01:14:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 Processor -- start 2024-04-22T01:14:54.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95af9ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95af9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95af9f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b95adbe5e0 con 0x55b95af70bc0 2024-04-22T01:14:54.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b95ad78520 con 0x55b95afb9ef0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55b95ae70e80 con 0x55b95afba810 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 10 client.0 _ll_register_callbacks cb 0x55b95af5b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95af9ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95af9ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39008/0 (socket says 172.21.15.70:39008) 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 -- 172.21.15.70:0/728051780 learned_addr learned my addr 172.21.15.70:0/728051780 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 -- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 msgr2=0x55b95af9f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95af9f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9425640 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95af9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 -- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 msgr2=0x55b95af9f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95af9f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 -- 172.21.15.70:0/728051780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95ad567b0 con 0x55b95af70bc0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95af9ef20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2c001ab90 tx=0x7fc2c0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3855d93a36fa777f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/728051780 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00fd070 con 0x55b95af70bc0 2024-04-22T01:14:54.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/728051780 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2c00f02c0 con 0x55b95af70bc0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c00b0020 con 0x55b95af70bc0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/728051780 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc2c011d070 con 0x55b95af70bc0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 10 client.4401 fetch_fsmap learned FSMap version 7 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55b95ad45ea0 con 0x55b95af70bc0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/728051780 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00b0020 con 0x55b95af70bc0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/728051780 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fc2c00f7220 con 0x55b95af70bc0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 10 client.4401 fetch_fsmap finished waiting for FSMap version 7 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 10 client.4401 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 20 client.4401 populate_metadata read hostname 'smithi070' 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afc9f90 0x55b95afcc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 4 client.4401 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 --> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b95ad59ab0 con 0x55b95afc9f90 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9425640 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afc9f90 0x55b95afcc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2c9425640 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afc9f90 0x55b95afcc380 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7e173b36563892 server_cookie=cdea918c8e4c5bd2 in_seq=0 out_seq=0 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 10 client.4401 ms_handle_connect on v2:172.21.15.70:6835/3139539459 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/728051780 <== mds.0 v2:172.21.15.70:6835/3139539459 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b95ad59ab0 con 0x55b95afc9f90 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 10 client.4401 handle_command_reply: tid=0 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 client.4401 shutdown 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 msgr2=0x55b95af9ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95af9ef20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2c001ab90 tx=0x7fc2c0000bd0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 shutdown_connections 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afc9f90 0x55b95afcc380 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95af9f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95af9f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2a4ff9640 0 client.4401 ms_handle_reset on v2:172.21.15.70:6835/3139539459 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 --2- 172.21.15.70:0/728051780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95af9ef20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.079+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 >> 172.21.15.70:0/728051780 conn(0x55b95af6cb70 msgr2=0x55b95af6f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.083+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 shutdown_connections 2024-04-22T01:14:54.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.083+0000 7fc2cf003000 1 -- 172.21.15.70:0/728051780 wait complete. 2024-04-22T01:14:54.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.083+0000 7fc2cf003000 20 client.4401 trim_cache size 0 max 16384 2024-04-22T01:14:54.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 Processor -- start 2024-04-22T01:14:54.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 -- start start 2024-04-22T01:14:54.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b025b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95b026070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95b0265b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c4203ae0 con 0x55b95af70bc0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c4202e40 con 0x55b95afb9ef0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c4203490 con 0x55b95afba810 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b025b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b025b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39016/0 (socket says 172.21.15.70:39016) 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 -- 172.21.15.70:0/2771020146 learned_addr learned my addr 172.21.15.70:0/2771020146 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9425640 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95b026070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 -- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 msgr2=0x55b95b0265b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2ca427640 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95b0265b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95b0265b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 -- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 msgr2=0x55b95b026070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95b026070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 -- 172.21.15.70:0/2771020146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95ad46e80 con 0x55b95af70bc0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b025b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00f0e30 tx=0x7fc2c011d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f489c78b30f87b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2a6ffd640 1 -- 172.21.15.70:0/2771020146 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00fd070 con 0x55b95af70bc0 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2a6ffd640 1 -- 172.21.15.70:0/2771020146 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00b0050 con 0x55b95af70bc0 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.099+0000 7fc2a6ffd640 1 -- 172.21.15.70:0/2771020146 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00f02c0 con 0x55b95af70bc0 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/2771020146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 msgr2=0x55b95b025b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b025b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00f0e30 tx=0x7fc2c011d000 comp rx=0 tx=0).stop 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/2771020146 shutdown_connections 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afba810 0x55b95b0265b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afb9ef0 0x55b95b026070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/2771020146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b025b30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/2771020146 >> 172.21.15.70:0/2771020146 conn(0x55b95af6cb70 msgr2=0x55b95afb87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/2771020146 shutdown_connections 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/2771020146 wait complete. 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- start start 2024-04-22T01:14:54.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 Processor -- start 2024-04-22T01:14:54.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b029870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afb9ef0 0x55b95b029db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95afccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b95adbe5e0 con 0x55b95af70bc0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b95ad78520 con 0x55b95afba810 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55b95ad87810 con 0x55b95afb9ef0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 10 client.0 _ll_register_callbacks cb 0x7fc2c011b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b029870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b029870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39032/0 (socket says 172.21.15.70:39032) 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4113718546 learned_addr learned my addr 172.21.15.70:0/4113718546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9425640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afb9ef0 0x55b95b029db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2ca427640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95afccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afb9ef0 msgr2=0x55b95b029db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afb9ef0 0x55b95b029db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 msgr2=0x55b95afccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95afccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 -- 172.21.15.70:0/4113718546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95ad8b760 con 0x55b95af70bc0 2024-04-22T01:14:54.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9c26640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b029870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00f0970 tx=0x7fc2c0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ac4504231e491e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/4113718546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00fd070 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/4113718546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2c0114e00 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c00b0050 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/4113718546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc2c00f0370 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 10 client.4407 fetch_fsmap learned FSMap version 7 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55b95ae179e0 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/4113718546 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00b0050 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/4113718546 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fc2c0118200 con 0x55b95af70bc0 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 10 client.4407 fetch_fsmap finished waiting for FSMap version 7 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 10 client.4407 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 20 client.4407 populate_metadata read hostname 'smithi070' 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afcd4b0 0x55b95afcd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 4 client.4407 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:14:54.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 --> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] -- command(tid 0: {"prefix": "status"}) -- 0x55b95adcbb20 con 0x55b95afcd4b0 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9425640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afcd4b0 0x55b95afcd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2c9425640 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afcd4b0 0x55b95afcd890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6f988ae1515552c server_cookie=a03d0eeb104b427c in_seq=0 out_seq=0 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 10 client.4407 ms_handle_connect on v2:172.21.15.70:6835/3139539459 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 1 -- 172.21.15.70:0/4113718546 <== mds.0 v2:172.21.15.70:6835/3139539459 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55b95adcbb20 con 0x55b95afcd4b0 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 10 client.4407 handle_command_reply: tid=0 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 client.4407 shutdown 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 msgr2=0x55b95b029870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b029870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00f0970 tx=0x7fc2c0009240 comp rx=0 tx=0).stop 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 shutdown_connections 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55b95afcd4b0 0x55b95afcd890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b95afba810 0x55b95afccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b95afb9ef0 0x55b95b029db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2a4ff9640 0 client.4407 ms_handle_reset on v2:172.21.15.70:6835/3139539459 2024-04-22T01:14:54.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 --2- 172.21.15.70:0/4113718546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b95af70bc0 0x55b95b029870 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 >> 172.21.15.70:0/4113718546 conn(0x55b95af6cb70 msgr2=0x55b95afb9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 shutdown_connections 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 1 -- 172.21.15.70:0/4113718546 wait complete. 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.103+0000 7fc2cf003000 20 client.4407 trim_cache size 0 max 16384 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4251, 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:14:54.110 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:14:54.111 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 7.4432947650000001, 2024-04-22T01:14:54.111 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 7, 2024-04-22T01:14:54.111 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 24, 2024-04-22T01:14:54.111 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 23, 2024-04-22T01:14:54.111 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 14.077885479000001 2024-04-22T01:14:54.111 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2a80678c0 msgr2=0x7fc2a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2a80678c0 0x7fc2a8069d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4154120 tx=0x7fc2b4009040 comp rx=0 tx=0).stop 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 msgr2=0x7fc2c4152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 0x7fc2c4152e80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc2c001a110 tx=0x7fc2c001ad20 comp rx=0 tx=0).stop 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 shutdown_connections 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2a80678c0 0x7fc2a8069d80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2c415bd10 0x7fc2c415f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c40b5fb0 0x7fc2c41553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 --2- 172.21.15.70:0/3289735202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2c406b1f0 0x7fc2c4152e80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 >> 172.21.15.70:0/3289735202 conn(0x7fc2c4099a60 msgr2=0x7fc2c40744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 shutdown_connections 2024-04-22T01:14:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.107+0000 7fc2cbe8a640 1 -- 172.21.15.70:0/3289735202 wait complete. 2024-04-22T01:14:54.121 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:14:54.121 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:54.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:14:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 Processor -- start 2024-04-22T01:14:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 -- start start 2024-04-22T01:14:54.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fad1c15f360 0x7fad1c0cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad1c099cc0 0x7fad1c09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad1c0cd890 con 0x7fad1c0c5cd0 2024-04-22T01:14:54.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fad1c0c5840 con 0x7fad1c15f360 2024-04-22T01:14:54.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad22785640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fad1c0cdb70 con 0x7fad1c099cc0 2024-04-22T01:14:54.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fad1c15f360 0x7fad1c0cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fad1c15f360 0x7fad1c0cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39038/0 (socket says 172.21.15.70:39038) 2024-04-22T01:14:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 -- 172.21.15.70:0/1757320041 learned_addr learned my addr 172.21.15.70:0/1757320041 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad13fff640 1 --2- 172.21.15.70:0/1757320041 >> v2:172.21.15.98:3300/0 conn(0x7fad1c099cc0 0x7fad1c09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 -- 172.21.15.70:0/1757320041 >> v2:172.21.15.98:3300/0 conn(0x7fad1c099cc0 msgr2=0x7fad1c09a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 --2- 172.21.15.70:0/1757320041 >> v2:172.21.15.98:3300/0 conn(0x7fad1c099cc0 0x7fad1c09a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 -- 172.21.15.70:0/1757320041 >> v1:172.21.15.70:6789/0 conn(0x7fad1c0c5cd0 legacy=0x7fad1c0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 -- 172.21.15.70:0/1757320041 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c0cde50 con 0x7fad1c15f360 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1b7fe640 1 -- 172.21.15.70:0/1757320041 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 934096564 0 0) 0x7fad1c0cd890 con 0x7fad1c0c5cd0 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.235+0000 7fad1bfff640 1 --2- 172.21.15.70:0/1757320041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c0cd2b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0ce490 tx=0x7fad0c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edaea29311cb6dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad1b7fe640 1 -- 172.21.15.70:0/1757320041 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0c7000 con 0x7fad1c15f360 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad1b7fe640 1 -- 172.21.15.70:0/1757320041 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c0b0070 con 0x7fad1c15f360 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad1b7fe640 1 -- 172.21.15.70:0/1757320041 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0ab650 con 0x7fad1c15f360 2024-04-22T01:14:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/1757320041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 msgr2=0x7fad1c0cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 --2- 172.21.15.70:0/1757320041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c0cd2b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0ce490 tx=0x7fad0c01c520 comp rx=0 tx=0).stop 2024-04-22T01:14:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/1757320041 shutdown_connections 2024-04-22T01:14:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 --2- 172.21.15.70:0/1757320041 >> v2:172.21.15.98:3300/0 conn(0x7fad1c099cc0 0x7fad1c09a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 --2- 172.21.15.70:0/1757320041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c0cd2b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/1757320041 >> 172.21.15.70:0/1757320041 conn(0x7fad1c0b35d0 msgr2=0x7fad1c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/1757320041 shutdown_connections 2024-04-22T01:14:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/1757320041 wait complete. 2024-04-22T01:14:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 Processor -- start 2024-04-22T01:14:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- start start 2024-04-22T01:14:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad1c099cc0 0x7fad1c0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad1c0c5cd0 0x7fad1c156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c168dc0 con 0x7fad1c15f360 2024-04-22T01:14:54.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c15f950 con 0x7fad1c0c5cd0 2024-04-22T01:14:54.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fad1c168f40 con 0x7fad1c099cc0 2024-04-22T01:14:54.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39040/0 (socket says 172.21.15.70:39040) 2024-04-22T01:14:54.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 -- 172.21.15.70:0/3766119773 learned_addr learned my addr 172.21.15.70:0/3766119773 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:54.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad13fff640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad1c0c5cd0 0x7fad1c156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad1bfff640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad1c099cc0 0x7fad1c0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 -- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad1c099cc0 msgr2=0x7fad1c0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad1c099cc0 0x7fad1c0cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 -- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad1c0c5cd0 msgr2=0x7fad1c156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad1c0c5cd0 0x7fad1c156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 -- 172.21.15.70:0/3766119773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c0cbac0 con 0x7fad1c15f360 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad1bfff640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad1c099cc0 0x7fad1c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad20d22640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c157110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c890 tx=0x7fad1400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14017070 con 0x7fad1c15f360 2024-04-22T01:14:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad14003db0 con 0x7fad1c15f360 2024-04-22T01:14:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c0b56b0 con 0x7fad1c15f360 2024-04-22T01:14:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad140052b0 con 0x7fad1c15f360 2024-04-22T01:14:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.239+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c0b6b40 con 0x7fad1c15f360 2024-04-22T01:14:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.243+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad1c068210 con 0x7fad1c15f360 2024-04-22T01:14:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.243+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad1401a070 con 0x7fad1c15f360 2024-04-22T01:14:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.243+0000 7fad197fa640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facfc0678c0 0x7facfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:54.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.243+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fad140859d0 con 0x7fad1c15f360 2024-04-22T01:14:54.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.243+0000 7fad1bfff640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facfc0678c0 0x7facfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:54.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.243+0000 7fad1bfff640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facfc0678c0 0x7facfc069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0af000 tx=0x7fad0c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:54.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.247+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fad14021020 con 0x7fad1c15f360 2024-04-22T01:14:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.431+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fad1c0c7f90 con 0x7fad1c15f360 2024-04-22T01:14:54.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.431+0000 7fad197fa640 1 -- 172.21.15.70:0/3766119773 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7fad14056400 con 0x7fad1c15f360 2024-04-22T01:14:54.430 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:14:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facfc0678c0 msgr2=0x7facfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facfc0678c0 0x7facfc069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0af000 tx=0x7fad0c0a7000 comp rx=0 tx=0).stop 2024-04-22T01:14:54.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 msgr2=0x7fad1c157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c157110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c890 tx=0x7fad1400cd50 comp rx=0 tx=0).stop 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 shutdown_connections 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facfc0678c0 0x7facfc069d80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c15f360 0x7fad1c157110 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad1c0c5cd0 0x7fad1c156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 --2- 172.21.15.70:0/3766119773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad1c099cc0 0x7fad1c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 >> 172.21.15.70:0/3766119773 conn(0x7fad1c0b35d0 msgr2=0x7fad1c0c7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 shutdown_connections 2024-04-22T01:14:54.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54.435+0000 7fad22785640 1 -- 172.21.15.70:0/3766119773 wait complete. 2024-04-22T01:14:54.447 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-22T01:14:54.447 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:14:54.448 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-22T01:14:54.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:54.448 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:14:54.454 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42527sec preferred_lft 42527sec 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:14:54.455 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:14:54.456 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:14:54.456 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T01:14:54.457 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:14:54.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:14:54.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:54.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:54.578 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:14:54.628 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:14:54.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:54.630 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:14:54.703 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:14:54.703 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:14:54.703 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:14:54.703 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:14:54.703 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:14:54.703 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:14:54.704 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:14:54.704 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:14:54.704 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:14:54.704 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:14:54.704 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:14:54.704 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:14:54.704 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:14:54.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:14:54.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:14:54.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:14:54.705 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:14:54.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:14:54.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:54.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:54.816 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:14:54.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:54.865 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-22T01:14:54.912 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:14:54.912 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:14:54.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:54.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-22T01:14:54.913 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:14:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:14:55.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:55.027 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-22T01:14:55.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-22T01:14:55.028 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:14:55.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:14:55.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:55.169 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:14:55.169 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:14:55.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:14:55.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T01:14:55.169 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:14:55.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:14:55.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:55.269 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-22T01:14:55.270 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:14:55.270 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:55.320 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-22T01:14:55.320 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:14:55.321 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:55.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:14:55.423 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:14:55.471 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:14:55.471 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:14:55.472 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:14:55.473 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:14:55.474 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:14:55.474 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:14:55.474 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:14:55.474 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:55.474 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:55.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:55.475 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:14:55.521 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:14:55.521 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-22T01:14:55.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:14:55.580 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:14:55.631 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:14:55.631 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:14:55.631 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:14:55.631 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:14:55.631 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:14:55.631 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:14:55.632 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:14:55.633 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:14:55.634 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:14:55.634 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:14:55.634 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:14:55.634 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:55.634 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:55.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:55.634 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:14:55.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:2024-04-22T01:14:55.647+0000 7f8208173580 -1 init, newargv = 0x55f6ca9d5a60 newargc=15 2024-04-22T01:14:55.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[33395]: starting ceph client 2024-04-22T01:14:55.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[33395]: starting fuse 2024-04-22T01:14:56.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:14:56.767 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:14:56.814 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:14:56.815 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:14:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:56.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:56.817 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:14:56.869 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:14:56.869 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:14:56.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:14:56.869 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:14:56.869 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:14:56.870 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.admin")) 2024-04-22T01:14:56.871 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:14:56.991 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.admin.33395.asok 2024-04-22T01:14:56.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:57.003 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.33395.asok 2024-04-22T01:14:57.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:57.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.33395.asok status 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "admin", 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "33395", 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:14:57.159 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4425, 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 4425 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 274193664 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:14:57.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 274193664 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.4425 192.168.0.1:0/274193664", 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/274193664", 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 7, 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 24, 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:14:57.161 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:14:57.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:57.172 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:14:57.172 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:57.178 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:14:57.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:57.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:57.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:14:57.233 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-22T01:14:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T01:14:57.358 INFO:teuthology.orchestra.run.smithi070.stdout:# file: . 2024-04-22T01:14:57.358 INFO:teuthology.orchestra.run.smithi070.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-22T01:14:57.358 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:57.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:14:57.365 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:57.365 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:14:57.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 Processor -- start 2024-04-22T01:14:57.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 -- start start 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c4415aa30 0x7f4c440b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4c4415bb50 0x7f4c440ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c440aec70 0x7f4c4415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4c440af070 con 0x7f4c4415bb50 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4406dbd0 con 0x7f4c4415aa30 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c4956c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4406de70 con 0x7f4c440aec70 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4c4415bb50 0x7f4c440ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4c4415bb50 0x7f4c440ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60764/0 (socket says 172.21.15.70:60764) 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/17986245 learned_addr learned my addr 172.21.15.70:0/17986245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c42ffd640 1 --2- 172.21.15.70:0/17986245 >> v2:172.21.15.98:3300/0 conn(0x7f4c4415aa30 0x7f4c440b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c437fe640 1 --2- 172.21.15.70:0/17986245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c440aec70 0x7f4c4415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/17986245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c440aec70 msgr2=0x7f4c4415e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:57.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 --2- 172.21.15.70:0/17986245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c440aec70 0x7f4c4415e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/17986245 >> v2:172.21.15.98:3300/0 conn(0x7f4c4415aa30 msgr2=0x7f4c440b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:57.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 --2- 172.21.15.70:0/17986245 >> v2:172.21.15.98:3300/0 conn(0x7f4c4415aa30 0x7f4c440b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.535+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/17986245 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4406e150 con 0x7f4c4415bb50 2024-04-22T01:14:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 --2- 172.21.15.70:0/17986245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415bb50 0x7f4c440ae730 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406ecd0 tx=0x7f4c3401c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c80399f5a8fb2ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c427fc640 1 -- 172.21.15.70:0/17986245 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c340b0070 con 0x7f4c4415bb50 2024-04-22T01:14:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c427fc640 1 -- 172.21.15.70:0/17986245 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c340ab600 con 0x7f4c4415bb50 2024-04-22T01:14:57.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c427fc640 1 -- 172.21.15.70:0/17986245 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c340ab9d0 con 0x7f4c4415bb50 2024-04-22T01:14:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/17986245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415bb50 msgr2=0x7f4c440ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- 172.21.15.70:0/17986245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415bb50 0x7f4c440ae730 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406ecd0 tx=0x7f4c3401c230 comp rx=0 tx=0).stop 2024-04-22T01:14:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/17986245 shutdown_connections 2024-04-22T01:14:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- 172.21.15.70:0/17986245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c440aec70 0x7f4c4415e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- 172.21.15.70:0/17986245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415bb50 0x7f4c440ae730 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- 172.21.15.70:0/17986245 >> v2:172.21.15.98:3300/0 conn(0x7f4c4415aa30 0x7f4c440b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/17986245 >> 172.21.15.70:0/17986245 conn(0x7f4c4406b1f0 msgr2=0x7f4c4406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/17986245 shutdown_connections 2024-04-22T01:14:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/17986245 wait complete. 2024-04-22T01:14:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 Processor -- start 2024-04-22T01:14:57.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- start start 2024-04-22T01:14:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c440aec70 0x7f4c4414c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 0x7f4c44164d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4415bb50 0x7f4c44150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4415af90 con 0x7f4c4415aa30 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4415ae10 con 0x7f4c440aec70 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4406e950 con 0x7f4c4415bb50 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 0x7f4c44164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 0x7f4c44164d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60776/0 (socket says 172.21.15.70:60776) 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/4283229081 learned_addr learned my addr 172.21.15.70:0/4283229081 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c42ffd640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c440aec70 0x7f4c4414c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c437fe640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4415bb50 0x7f4c44150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4415bb50 msgr2=0x7f4c44150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4415bb50 0x7f4c44150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c440aec70 msgr2=0x7f4c4414c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c440aec70 0x7f4c4414c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 -- 172.21.15.70:0/4283229081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4414e660 con 0x7f4c4415aa30 2024-04-22T01:14:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c437fe640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4415bb50 0x7f4c44150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c42ffd640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c440aec70 0x7f4c4414c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3a7fc640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 0x7f4c44164d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4c340ae040 tx=0x7f4c34018fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c340b9070 con 0x7f4c4415aa30 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c340b4400 con 0x7f4c4415aa30 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4414e940 con 0x7f4c4415aa30 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c340b0040 con 0x7f4c4415aa30 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.539+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4414ef00 con 0x7f4c4415aa30 2024-04-22T01:14:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.543+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c44068210 con 0x7f4c4415aa30 2024-04-22T01:14:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.543+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4c340bc070 con 0x7f4c4415aa30 2024-04-22T01:14:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.543+0000 7f4c3bfff640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4c240678f0 0x7f4c24069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.543+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4c3411f010 con 0x7f4c4415aa30 2024-04-22T01:14:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.543+0000 7f4c42ffd640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4c240678f0 0x7f4c24069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.543+0000 7f4c42ffd640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4c240678f0 0x7f4c24069db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0097a0 tx=0x7f4c2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.547+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4c34016110 con 0x7f4c4415aa30 2024-04-22T01:14:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.707+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c4406f190 con 0x7f4c4415aa30 2024-04-22T01:14:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c3bfff640 1 -- 172.21.15.70:0/4283229081 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f4c340ef780 con 0x7f4c4415aa30 2024-04-22T01:14:57.709 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:57.709 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4c240678f0 msgr2=0x7f4c24069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4c240678f0 0x7f4c24069db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0097a0 tx=0x7f4c2c008040 comp rx=0 tx=0).stop 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 msgr2=0x7f4c44164d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 0x7f4c44164d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4c340ae040 tx=0x7f4c34018fd0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 shutdown_connections 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4c240678f0 0x7f4c24069db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4415bb50 0x7f4c44150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c4415aa30 0x7f4c44164d30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 --2- 172.21.15.70:0/4283229081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c440aec70 0x7f4c4414c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 >> 172.21.15.70:0/4283229081 conn(0x7f4c4406b1f0 msgr2=0x7f4c441567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 shutdown_connections 2024-04-22T01:14:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.711+0000 7f4c4956c640 1 -- 172.21.15.70:0/4283229081 wait complete. 2024-04-22T01:14:57.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:14:57.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:14:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 Processor -- start 2024-04-22T01:14:57.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 -- start start 2024-04-22T01:14:57.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdf74072280 0x7fdf740ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf740cd050 0x7fdf740b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7415c290 con 0x7fdf740b4450 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdf7415c570 con 0x7fdf74072280 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf78a6b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdf7415c850 con 0x7fdf740cd050 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf71d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdf74072280 0x7fdf740ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf7259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdf740b4450 0x7fdf74071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:47916/0 (socket says 172.21.15.70:47916) 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf7259c640 1 -- 172.21.15.70:0/2951862895 learned_addr learned my addr 172.21.15.70:0/2951862895 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2951862895 >> v2:172.21.15.98:3300/0 conn(0x7fdf740cd050 msgr2=0x7fdf740b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf72d9d640 1 --2- 172.21.15.70:0/2951862895 >> v2:172.21.15.98:3300/0 conn(0x7fdf740cd050 0x7fdf740b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf71d9b640 1 --2- 172.21.15.70:0/2951862895 >> v2:172.21.15.98:3300/0 conn(0x7fdf740cd050 0x7fdf740b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2951862895 >> v1:172.21.15.70:6789/0 conn(0x7fdf740b4450 legacy=0x7fdf74071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.855+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2951862895 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7415cb30 con 0x7fdf74072280 2024-04-22T01:14:57.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 --2- 172.21.15.70:0/2951862895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf74072280 0x7fdf740ccb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf740b27c0 tx=0x7fdf5c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5bc380d193e2d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:57.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf7159a640 1 -- 172.21.15.70:0/2951862895 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0b0070 con 0x7fdf74072280 2024-04-22T01:14:57.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf7159a640 1 -- 172.21.15.70:0/2951862895 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0ab600 con 0x7fdf74072280 2024-04-22T01:14:57.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf7159a640 1 -- 172.21.15.70:0/2951862895 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0ab9d0 con 0x7fdf74072280 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2951862895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf74072280 msgr2=0x7fdf740ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2951862895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf74072280 0x7fdf740ccb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf740b27c0 tx=0x7fdf5c01c3f0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2951862895 shutdown_connections 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2951862895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf74072280 0x7fdf740ccb10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2951862895 >> v2:172.21.15.98:3300/0 conn(0x7fdf740cd050 0x7fdf740b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2951862895 >> 172.21.15.70:0/2951862895 conn(0x7fdf740b1d40 msgr2=0x7fdf740cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2951862895 shutdown_connections 2024-04-22T01:14:57.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2951862895 wait complete. 2024-04-22T01:14:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 Processor -- start 2024-04-22T01:14:57.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- start start 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf74072280 0x7fdf7414ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 0x7fdf74157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf740cd050 0x7fdf74154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf740cd980 con 0x7fdf740b4450 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf740cd800 con 0x7fdf740cd050 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf740b3790 con 0x7fdf74072280 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 0x7fdf74157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 0x7fdf74157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60802/0 (socket says 172.21.15.70:60802) 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2831256636 learned_addr learned my addr 172.21.15.70:0/2831256636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:57.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf74072280 msgr2=0x7fdf7414ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf74072280 0x7fdf7414ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf72d9d640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf740cd050 0x7fdf74154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf740cd050 msgr2=0x7fdf74154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf740cd050 0x7fdf74154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 -- 172.21.15.70:0/2831256636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7414bdd0 con 0x7fdf740b4450 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf71d9b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 0x7fdf74157070 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c01aaa0 tx=0x7fdf5c019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0b8020 con 0x7fdf740b4450 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0b4610 con 0x7fdf740b4450 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7414c070 con 0x7fdf740b4450 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0b0040 con 0x7fdf740b4450 2024-04-22T01:14:57.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.859+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf741575b0 con 0x7fdf740b4450 2024-04-22T01:14:57.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.863+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7414c590 con 0x7fdf740b4450 2024-04-22T01:14:57.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.863+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf5c0bb020 con 0x7fdf740b4450 2024-04-22T01:14:57.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.863+0000 7fdf6affd640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf540678c0 0x7fdf54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:57.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.863+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdf5c11fc30 con 0x7fdf740b4450 2024-04-22T01:14:57.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.863+0000 7fdf7259c640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf540678c0 0x7fdf54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:57.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.863+0000 7fdf7259c640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf540678c0 0x7fdf54069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf64000f70 tx=0x7fdf64007a90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:57.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:57.867+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdf7414c590 con 0x7fdf740b4450 2024-04-22T01:14:58.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.027+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf74068210 con 0x7fdf740b4450 2024-04-22T01:14:58.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.027+0000 7fdf6affd640 1 -- 172.21.15.70:0/2831256636 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fdf5c0c2030 con 0x7fdf740b4450 2024-04-22T01:14:58.026 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:14:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf540678c0 msgr2=0x7fdf54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf540678c0 0x7fdf54069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf64000f70 tx=0x7fdf64007a90 comp rx=0 tx=0).stop 2024-04-22T01:14:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 msgr2=0x7fdf74157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 0x7fdf74157070 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c01aaa0 tx=0x7fdf5c019190 comp rx=0 tx=0).stop 2024-04-22T01:14:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 shutdown_connections 2024-04-22T01:14:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf540678c0 0x7fdf54069d80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf740cd050 0x7fdf74154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf740b4450 0x7fdf74157070 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 --2- 172.21.15.70:0/2831256636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf74072280 0x7fdf7414ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 >> 172.21.15.70:0/2831256636 conn(0x7fdf740b1d40 msgr2=0x7fdf740b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 shutdown_connections 2024-04-22T01:14:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.031+0000 7fdf78a6b640 1 -- 172.21.15.70:0/2831256636 wait complete. 2024-04-22T01:14:58.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:14:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.167+0000 7fa3c5dfd640 1 Processor -- start 2024-04-22T01:14:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.167+0000 7fa3c5dfd640 1 -- start start 2024-04-22T01:14:58.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3c5dfd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3c015f480 0x7fa3c00b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3c5dfd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c006c310 0x7fa3c00ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3c5dfd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c00678d0 con 0x7fa3c006b400 2024-04-22T01:14:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3c5dfd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa3c00673a0 con 0x7fa3c015f480 2024-04-22T01:14:58.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3c5dfd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c00675f0 con 0x7fa3c006c310 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3c015f480 0x7fa3c00b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c006c310 0x7fa3c00ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3c015f480 0x7fa3c00b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:33848/0 (socket says 172.21.15.70:33848) 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/1711544712 learned_addr learned my addr 172.21.15.70:0/1711544712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c006c310 msgr2=0x7fa3c00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 --2- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c006c310 0x7fa3c00ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/1711544712 >> v1:172.21.15.98:6789/0 conn(0x7fa3c006b400 legacy=0x7fa3c006bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.171+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/1711544712 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c00b5100 con 0x7fa3c015f480 2024-04-22T01:14:58.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 --2- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c015f480 0x7fa3c00b5be0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009870 tx=0x7fa3a801c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6335f082c7abfb3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3be7fc640 1 -- 172.21.15.70:0/1711544712 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80b2070 con 0x7fa3c015f480 2024-04-22T01:14:58.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3be7fc640 1 -- 172.21.15.70:0/1711544712 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a80a9070 con 0x7fa3c015f480 2024-04-22T01:14:58.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3be7fc640 1 -- 172.21.15.70:0/1711544712 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80ad880 con 0x7fa3c015f480 2024-04-22T01:14:58.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c015f480 msgr2=0x7fa3c00b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c015f480 0x7fa3c00b5be0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009870 tx=0x7fa3a801c560 comp rx=0 tx=0).stop 2024-04-22T01:14:58.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/1711544712 shutdown_connections 2024-04-22T01:14:58.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c006c310 0x7fa3c00ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/1711544712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c015f480 0x7fa3c00b5be0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/1711544712 >> 172.21.15.70:0/1711544712 conn(0x7fa3c0099a60 msgr2=0x7fa3c00b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/1711544712 shutdown_connections 2024-04-22T01:14:58.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/1711544712 wait complete. 2024-04-22T01:14:58.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 Processor -- start 2024-04-22T01:14:58.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- start start 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 0x7fa3c015c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c006c310 0x7fa3c015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c015f480 0x7fa3c00c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c00cb380 con 0x7fa3c006b400 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c00cb200 con 0x7fa3c006c310 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3c5dfd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c00b2c00 con 0x7fa3c015f480 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 0x7fa3c015c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 0x7fa3c015c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60812/0 (socket says 172.21.15.70:60812) 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/2733162782 learned_addr learned my addr 172.21.15.70:0/2733162782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bffff640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c015f480 0x7fa3c00c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3beffd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c006c310 0x7fa3c015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c015f480 msgr2=0x7fa3c00c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c015f480 0x7fa3c00c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c006c310 msgr2=0x7fa3c015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c006c310 0x7fa3c015d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bf7fe640 1 -- 172.21.15.70:0/2733162782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c0159bb0 con 0x7fa3c006b400 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.175+0000 7fa3bffff640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c015f480 0x7fa3c00c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3bf7fe640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 0x7fa3c015c750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80b0040 tx=0x7fa3a8019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80ba020 con 0x7fa3c006b400 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a80adda0 con 0x7fa3c006b400 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80b2040 con 0x7fa3c006b400 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c0159e30 con 0x7fa3c006b400 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c015a3f0 con 0x7fa3c006b400 2024-04-22T01:14:58.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3c0068210 con 0x7fa3c006b400 2024-04-22T01:14:58.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa3a80bd020 con 0x7fa3c006b400 2024-04-22T01:14:58.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3b7fff640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3a00678c0 0x7fa3a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3beffd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3a00678c0 0x7fa3a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa3a811fae0 con 0x7fa3c006b400 2024-04-22T01:14:58.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.179+0000 7fa3beffd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3a00678c0 0x7fa3a0069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00cea40 tx=0x7fa3b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.183+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa3a80c4030 con 0x7fa3c006b400 2024-04-22T01:14:58.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.343+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3c006b9e0 con 0x7fa3c006b400 2024-04-22T01:14:58.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3b7fff640 1 -- 172.21.15.70:0/2733162782 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fa3a80f6020 con 0x7fa3c006b400 2024-04-22T01:14:58.344 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3a00678c0 msgr2=0x7fa3a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3a00678c0 0x7fa3a0069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00cea40 tx=0x7fa3b0009040 comp rx=0 tx=0).stop 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 msgr2=0x7fa3c015c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 0x7fa3c015c750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80b0040 tx=0x7fa3a8019300 comp rx=0 tx=0).stop 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 shutdown_connections 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa3a00678c0 0x7fa3a0069d80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3c015f480 0x7fa3c00c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3c006c310 0x7fa3c015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 --2- 172.21.15.70:0/2733162782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3c006b400 0x7fa3c015c750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 >> 172.21.15.70:0/2733162782 conn(0x7fa3c0099a60 msgr2=0x7fa3c00d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 shutdown_connections 2024-04-22T01:14:58.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.347+0000 7fa3c5dfd640 1 -- 172.21.15.70:0/2733162782 wait complete. 2024-04-22T01:14:58.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:14:58.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec6c70640 1 Processor -- start 2024-04-22T01:14:58.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec6c70640 1 -- start start 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec6c70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ec00b4450 0x7f3ec0071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec6c70640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ec015c290 con 0x7f3ec00cd220 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec6c70640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ec015c570 con 0x7f3ec00720b0 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec6c70640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec015c850 con 0x7f3ec00b4450 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec520d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3ec00cd220 0x7f3ec00b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:47918/0 (socket says 172.21.15.70:47918) 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec520d640 1 -- 172.21.15.70:0/946289996 learned_addr learned my addr 172.21.15.70:0/946289996 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ec4a0c640 1 --2- 172.21.15.70:0/946289996 >> v2:172.21.15.98:3300/0 conn(0x7f3ec00b4450 0x7f3ec0071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938420442 0 0) 0x7f3ec015c290 con 0x7f3ec00cd220 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.487+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea8003530 con 0x7f3ec00cd220 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1725034313 0 0) 0x7f3ea8003530 con 0x7f3ec00cd220 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 >> v2:172.21.15.98:3300/0 conn(0x7f3ec00b4450 msgr2=0x7f3ec0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 --2- 172.21.15.70:0/946289996 >> v2:172.21.15.98:3300/0 conn(0x7f3ec00b4450 0x7f3ec0071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 >> v1:172.21.15.98:6789/0 conn(0x7f3ec00720b0 legacy=0x7f3ec00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec015cb30 con 0x7f3ec00cd220 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3eb8001a00 con 0x7f3ec00cd220 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 >> v1:172.21.15.70:6789/0 conn(0x7f3ec00cd220 legacy=0x7f3ec00b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80039c0 0x7f3ea8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ea8004810 0x7f3ea8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ea80051d0 0x7f3ea800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec015c850 con 0x7f3ea80051d0 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea800c070 con 0x7f3ea80039c0 2024-04-22T01:14:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea800c270 con 0x7f3ea8004810 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ea80051d0 0x7f3ea800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec4a0c640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80039c0 0x7f3ea8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebffff640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ea8004810 0x7f3ea8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 -- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ea8004810 msgr2=0x7f3ea8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ea8004810 0x7f3ea8004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 -- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80039c0 msgr2=0x7f3ea8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80039c0 0x7f3ea8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 -- 172.21.15.70:0/946289996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea8003530 con 0x7f3ea80051d0 2024-04-22T01:14:58.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 -- 172.21.15.70:0/946289996 reap_dead start 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec520d640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ea80051d0 0x7f3ea800b9d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800a840 tx=0x7f3eb801d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f90c7d0d654e16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb800a460 con 0x7f3ea80051d0 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb80ac640 con 0x7f3ea80051d0 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ebf7fe640 1 -- 172.21.15.70:0/946289996 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80aca30 con 0x7f3ea80051d0 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 -- 172.21.15.70:0/946289996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ea80051d0 msgr2=0x7f3ea800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ea80051d0 0x7f3ea800b9d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800a840 tx=0x7f3eb801d360 comp rx=0 tx=0).stop 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 -- 172.21.15.70:0/946289996 shutdown_connections 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/946289996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ea80051d0 0x7f3ea800b9d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 -- 172.21.15.70:0/946289996 >> 172.21.15.70:0/946289996 conn(0x7f3ec00b1d40 msgr2=0x7f3ec00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 -- 172.21.15.70:0/946289996 shutdown_connections 2024-04-22T01:14:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.491+0000 7f3ec6c70640 1 -- 172.21.15.70:0/946289996 wait complete. 2024-04-22T01:14:58.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 Processor -- start 2024-04-22T01:14:58.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- start start 2024-04-22T01:14:58.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80051d0 0x7f3ec01477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ec014cd00 0x7f3ec0146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 0x7f3ec014d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec00b49b0 con 0x7f3ec014d570 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec00b4830 con 0x7f3ea80051d0 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec00cd650 con 0x7f3ec014cd00 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 0x7f3ec014d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 0x7f3ec014d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60826/0 (socket says 172.21.15.70:60826) 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 -- 172.21.15.70:0/1110396812 learned_addr learned my addr 172.21.15.70:0/1110396812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec4a0c640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80051d0 0x7f3ec01477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ebffff640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ec014cd00 0x7f3ec0146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 -- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ec014cd00 msgr2=0x7f3ec0146b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ec014cd00 0x7f3ec0146b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 -- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80051d0 msgr2=0x7f3ec01477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80051d0 0x7f3ec01477b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 -- 172.21.15.70:0/1110396812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec014e3f0 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec520d640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 0x7f3ec014d9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3eb801d190 tx=0x7f3ea8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8001fc0 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb80019c0 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80ac9f0 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec0147070 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec014f980 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec014d0e0 con 0x7f3ec014d570 2024-04-22T01:14:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.495+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3eb80070c0 con 0x7f3ec014d570 2024-04-22T01:14:58.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.499+0000 7f3ebd7fa640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3eac0678c0 0x7f3eac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.499+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3eb8120180 con 0x7f3ec014d570 2024-04-22T01:14:58.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.499+0000 7f3ec4a0c640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3eac0678c0 0x7f3eac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.499+0000 7f3ec4a0c640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3eac0678c0 0x7f3eac069d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00099b0 tx=0x7f3eb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.503+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3eb8017070 con 0x7f3ec014d570 2024-04-22T01:14:58.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.659+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3ec0068210 con 0x7f3ec014d570 2024-04-22T01:14:58.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ebd7fa640 1 -- 172.21.15.70:0/1110396812 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f3eb80f0330 con 0x7f3ec014d570 2024-04-22T01:14:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:14:58.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3eac0678c0 msgr2=0x7f3eac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3eac0678c0 0x7f3eac069d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00099b0 tx=0x7f3eb0008040 comp rx=0 tx=0).stop 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 msgr2=0x7f3ec014d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 0x7f3ec014d9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3eb801d190 tx=0x7f3ea8005840 comp rx=0 tx=0).stop 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 shutdown_connections 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3eac0678c0 0x7f3eac069d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ec014d570 0x7f3ec014d9f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ec014cd00 0x7f3ec0146b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 --2- 172.21.15.70:0/1110396812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea80051d0 0x7f3ec01477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 >> 172.21.15.70:0/1110396812 conn(0x7f3ec00b1d40 msgr2=0x7f3ec015e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 shutdown_connections 2024-04-22T01:14:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.663+0000 7f3ec6c70640 1 -- 172.21.15.70:0/1110396812 wait complete. 2024-04-22T01:14:58.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:14:58.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 Processor -- start 2024-04-22T01:14:58.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 -- start start 2024-04-22T01:14:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3c180cfc80 0x7f3c18160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c180d0060 0x7f3c1815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c1815cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3c180cd790 con 0x7f3c180cfc80 2024-04-22T01:14:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3c18160880 con 0x7f3c180d0060 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1ff1a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3c180ccb80 con 0x7f3c1815c700 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3c180cfc80 0x7f3c18160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3c180cfc80 0x7f3c18160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60842/0 (socket says 172.21.15.70:60842) 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/2079177381 learned_addr learned my addr 172.21.15.70:0/2079177381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1e4b7640 1 --2- 172.21.15.70:0/2079177381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c1815cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/2079177381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 msgr2=0x7f3c1815cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 --2- 172.21.15.70:0/2079177381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c1815cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1d4b5640 1 --2- 172.21.15.70:0/2079177381 >> v2:172.21.15.98:3300/0 conn(0x7f3c180d0060 0x7f3c1815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/2079177381 >> v2:172.21.15.98:3300/0 conn(0x7f3c180d0060 msgr2=0x7f3c1815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 --2- 172.21.15.70:0/2079177381 >> v2:172.21.15.98:3300/0 conn(0x7f3c180d0060 0x7f3c1815c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.807+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/2079177381 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c180cce60 con 0x7f3c180cfc80 2024-04-22T01:14:58.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 --2- 172.21.15.70:0/2079177381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c18160340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3c180733b0 tx=0x7f3c1401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85bcaf92368d0dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ccb4640 1 -- 172.21.15.70:0/2079177381 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c140ab620 con 0x7f3c180cfc80 2024-04-22T01:14:58.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ccb4640 1 -- 172.21.15.70:0/2079177381 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c140b0070 con 0x7f3c180cfc80 2024-04-22T01:14:58.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ccb4640 1 -- 172.21.15.70:0/2079177381 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c140ab9e0 con 0x7f3c180cfc80 2024-04-22T01:14:58.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/2079177381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 msgr2=0x7f3c18160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/2079177381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c18160340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3c180733b0 tx=0x7f3c1401c460 comp rx=0 tx=0).stop 2024-04-22T01:14:58.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/2079177381 shutdown_connections 2024-04-22T01:14:58.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/2079177381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c1815cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/2079177381 >> v2:172.21.15.98:3300/0 conn(0x7f3c180d0060 0x7f3c1815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/2079177381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c18160340 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/2079177381 >> 172.21.15.70:0/2079177381 conn(0x7f3c180b36f0 msgr2=0x7f3c180b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/2079177381 shutdown_connections 2024-04-22T01:14:58.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/2079177381 wait complete. 2024-04-22T01:14:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 Processor -- start 2024-04-22T01:14:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- start start 2024-04-22T01:14:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c1815dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c180d0060 0x7f3c18164e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c18152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1815d230 con 0x7f3c180cfc80 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1815d0b0 con 0x7f3c180d0060 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3c180d0470 con 0x7f3c1815c700 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c1815dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c1815dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60850/0 (socket says 172.21.15.70:60850) 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/3397089569 learned_addr learned my addr 172.21.15.70:0/3397089569 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1d4b5640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c180d0060 0x7f3c18164e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 msgr2=0x7f3c18152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1e4b7640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c18152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c18152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c180d0060 msgr2=0x7f3c18164e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c180d0060 0x7f3c18164e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 -- 172.21.15.70:0/3397089569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c18165860 con 0x7f3c180cfc80 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1d4b5640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c180d0060 0x7f3c18164e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1dcb6640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c1815dec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3c1401ab10 tx=0x7f3c1401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c140abda0 con 0x7f3c180cfc80 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c140be3f0 con 0x7f3c180cfc80 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c18165b40 con 0x7f3c180cfc80 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.811+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c140be7e0 con 0x7f3c180cfc80 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c18166100 con 0x7f3c180cfc80 2024-04-22T01:14:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c18068920 con 0x7f3c180cfc80 2024-04-22T01:14:58.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3c140b43f0 con 0x7f3c180cfc80 2024-04-22T01:14:58.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c0e7fc640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bfc0678c0 0x7f3bfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:58.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3c14120420 con 0x7f3c180cfc80 2024-04-22T01:14:58.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c1d4b5640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bfc0678c0 0x7f3bfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:58.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.815+0000 7f3c1d4b5640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bfc0678c0 0x7f3bfc069d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3c08006fd0 tx=0x7f3c08008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:58.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.819+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3c140c4060 con 0x7f3c180cfc80 2024-04-22T01:14:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3c180ce3f0 con 0x7f3c180cfc80 2024-04-22T01:14:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c0e7fc640 1 -- 172.21.15.70:0/3397089569 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f3c14016110 con 0x7f3c180cfc80 2024-04-22T01:14:58.982 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bfc0678c0 msgr2=0x7f3bfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bfc0678c0 0x7f3bfc069d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3c08006fd0 tx=0x7f3c08008040 comp rx=0 tx=0).stop 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 msgr2=0x7f3c1815dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c1815dec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3c1401ab10 tx=0x7f3c1401aca0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 shutdown_connections 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bfc0678c0 0x7f3bfc069d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1815c700 0x7f3c18152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c180d0060 0x7f3c18164e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 --2- 172.21.15.70:0/3397089569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c180cfc80 0x7f3c1815dec0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.983+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 >> 172.21.15.70:0/3397089569 conn(0x7f3c180b36f0 msgr2=0x7f3c1815d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.987+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 shutdown_connections 2024-04-22T01:14:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:58.987+0000 7f3c1ff1a640 1 -- 172.21.15.70:0/3397089569 wait complete. 2024-04-22T01:14:59.000 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:14:59.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2e941640 1 Processor -- start 2024-04-22T01:14:59.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2e941640 1 -- start start 2024-04-22T01:14:59.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2e941640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f2806f480 0x7f0f28166c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2e941640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f280cd7a0 con 0x7f0f2815bf30 2024-04-22T01:14:59.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2e941640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2815fb00 con 0x7f0f2815bb50 2024-04-22T01:14:59.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2e941640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0f280cd580 con 0x7f0f2806f480 2024-04-22T01:14:59.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f277fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0f2815bf30 0x7f0f2806ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:47920/0 (socket says 172.21.15.70:47920) 2024-04-22T01:14:59.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f277fe640 1 -- 172.21.15.70:0/3040214669 learned_addr learned my addr 172.21.15.70:0/3040214669 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:59.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f2cede640 1 --2- 172.21.15.70:0/3040214669 >> v2:172.21.15.98:3300/0 conn(0x7f0f2806f480 0x7f0f28166c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3860917593 0 0) 0x7f0f280cd7a0 con 0x7f0f2815bf30 2024-04-22T01:14:59.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f10003530 con 0x7f0f2815bf30 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1761171828 0 0) 0x7f0f10003530 con 0x7f0f2815bf30 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 >> v2:172.21.15.98:3300/0 conn(0x7f0f2806f480 msgr2=0x7f0f28166c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 --2- 172.21.15.70:0/3040214669 >> v2:172.21.15.98:3300/0 conn(0x7f0f2806f480 0x7f0f28166c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 >> v1:172.21.15.98:6789/0 conn(0x7f0f2815bb50 legacy=0x7f0f280cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2815fde0 con 0x7f0f2815bf30 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f0f1c001680 con 0x7f0f2815bf30 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 >> v1:172.21.15.70:6789/0 conn(0x7f0f2815bf30 legacy=0x7f0f2806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f10003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f10004810 0x7f0f10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f100051d0 0x7f0f1000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f280cd580 con 0x7f0f10003a60 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1000c170 con 0x7f0f10004810 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.119+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1000c370 con 0x7f0f100051d0 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f10003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f277fe640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f10004810 0x7f0f10004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2cede640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f100051d0 0x7f0f1000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 -- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f100051d0 msgr2=0x7f0f1000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f100051d0 0x7f0f1000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 -- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f10004810 msgr2=0x7f0f10004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f10004810 0x7f0f10004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 -- 172.21.15.70:0/3040214669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f10003530 con 0x7f0f10003a60 2024-04-22T01:14:59.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2cede640 1 -- 172.21.15.70:0/3040214669 reap_dead start 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f27fff640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f10003e80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0f18001780 tx=0x7f0f1801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfdc2833be3e3e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f180013a0 con 0x7f0f10003a60 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f180052f0 con 0x7f0f10003a60 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f26ffd640 1 -- 172.21.15.70:0/3040214669 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18003840 con 0x7f0f10003a60 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 -- 172.21.15.70:0/3040214669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 msgr2=0x7f0f10003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f10003e80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0f18001780 tx=0x7f0f1801c400 comp rx=0 tx=0).stop 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 -- 172.21.15.70:0/3040214669 shutdown_connections 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 --2- 172.21.15.70:0/3040214669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f10003e80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 -- 172.21.15.70:0/3040214669 >> 172.21.15.70:0/3040214669 conn(0x7f0f280cf9e0 msgr2=0x7f0f280cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 -- 172.21.15.70:0/3040214669 shutdown_connections 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.123+0000 7f0f2e941640 1 -- 172.21.15.70:0/3040214669 wait complete. 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 Processor -- start 2024-04-22T01:14:59.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- start start 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f280c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f28142060 0x7f0f28159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f281424e0 0x7f0f28159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2815f8b0 con 0x7f0f10003a60 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f280cf670 con 0x7f0f281424e0 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0f280cce70 con 0x7f0f28142060 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f280c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f280c9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60876/0 (socket says 172.21.15.70:60876) 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 -- 172.21.15.70:0/2004325486 learned_addr learned my addr 172.21.15.70:0/2004325486 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f277fe640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f28142060 0x7f0f28159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2cede640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f281424e0 0x7f0f28159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 -- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f28142060 msgr2=0x7f0f28159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f28142060 0x7f0f28159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 -- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f281424e0 msgr2=0x7f0f28159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f281424e0 0x7f0f28159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 -- 172.21.15.70:0/2004325486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f280c8df0 con 0x7f0f10003a60 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2cede640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f281424e0 0x7f0f28159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f27fff640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f280c9e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0f1801c230 tx=0x7f0f10005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f180013a0 con 0x7f0f10003a60 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f180ba070 con 0x7f0f10003a60 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18007a40 con 0x7f0f10003a60 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f280c9090 con 0x7f0f10003a60 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f280c9650 con 0x7f0f10003a60 2024-04-22T01:14:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.127+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2804f510 con 0x7f0f10003a60 2024-04-22T01:14:59.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.131+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0f180052f0 con 0x7f0f10003a60 2024-04-22T01:14:59.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.131+0000 7f0f24ff9640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f140678c0 0x7f0f14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.131+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0f18120510 con 0x7f0f10003a60 2024-04-22T01:14:59.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.131+0000 7f0f277fe640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f140678c0 0x7f0f14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.131+0000 7f0f277fe640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f140678c0 0x7f0f14069d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c001060 tx=0x7f0f1c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.135+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0f18016110 con 0x7f0f10003a60 2024-04-22T01:14:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.307+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0f281439d0 con 0x7f0f10003a60 2024-04-22T01:14:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.307+0000 7f0f24ff9640 1 -- 172.21.15.70:0/2004325486 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f0f180b3e70 con 0x7f0f10003a60 2024-04-22T01:14:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f140678c0 msgr2=0x7f0f14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f140678c0 0x7f0f14069d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c001060 tx=0x7f0f1c009040 comp rx=0 tx=0).stop 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 msgr2=0x7f0f280c9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f280c9e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0f1801c230 tx=0x7f0f10005840 comp rx=0 tx=0).stop 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 shutdown_connections 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f140678c0 0x7f0f14069d80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f281424e0 0x7f0f28159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f28142060 0x7f0f28159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 --2- 172.21.15.70:0/2004325486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10003a60 0x7f0f280c9e50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 >> 172.21.15.70:0/2004325486 conn(0x7f0f280cf9e0 msgr2=0x7f0f280cbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 shutdown_connections 2024-04-22T01:14:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.311+0000 7f0f2e941640 1 -- 172.21.15.70:0/2004325486 wait complete. 2024-04-22T01:14:59.325 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:14:59.326 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:14:59.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:14:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.487+0000 7f3008019640 1 Processor -- start 2024-04-22T01:14:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.487+0000 7f3008019640 1 -- start start 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f30000b5d20 0x7f30000ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30000d0970 con 0x7f300015a520 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300006e5f0 con 0x7f300006f470 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f30000d1130 con 0x7f30000b5d20 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3005db5640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f300015a520 0x7f300006ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:47924/0 (socket says 172.21.15.70:47924) 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3005db5640 1 -- 172.21.15.70:0/3969080768 learned_addr learned my addr 172.21.15.70:0/3969080768 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:14:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f30065b6640 1 --2- 172.21.15.70:0/3969080768 >> v2:172.21.15.70:3300/0 conn(0x7f30000b5d20 0x7f30000ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f30065b6640 1 -- 172.21.15.70:0/3969080768 >> v1:172.21.15.98:6789/0 conn(0x7f300006f470 legacy=0x7f30000ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f30065b6640 1 -- 172.21.15.70:0/3969080768 >> v1:172.21.15.70:6789/0 conn(0x7f300015a520 legacy=0x7f300006ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f30065b6640 1 -- 172.21.15.70:0/3969080768 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300006e8d0 con 0x7f30000b5d20 2024-04-22T01:14:59.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f30065b6640 1 --2- 172.21.15.70:0/3969080768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000ade50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009d90 tx=0x7f2ff801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce245ebe333cfc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3004db3640 1 -- 172.21.15.70:0/3969080768 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80b2070 con 0x7f30000b5d20 2024-04-22T01:14:59.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3004db3640 1 -- 172.21.15.70:0/3969080768 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ff80a5940 con 0x7f30000b5d20 2024-04-22T01:14:59.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3004db3640 1 -- 172.21.15.70:0/3969080768 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80a5d10 con 0x7f30000b5d20 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- 172.21.15.70:0/3969080768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 msgr2=0x7f30000ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 --2- 172.21.15.70:0/3969080768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000ade50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009d90 tx=0x7f2ff801c970 comp rx=0 tx=0).stop 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- 172.21.15.70:0/3969080768 shutdown_connections 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 --2- 172.21.15.70:0/3969080768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000ade50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- 172.21.15.70:0/3969080768 >> 172.21.15.70:0/3969080768 conn(0x7f3000099a60 msgr2=0x7f30000b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- 172.21.15.70:0/3969080768 shutdown_connections 2024-04-22T01:14:59.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.491+0000 7f3008019640 1 -- 172.21.15.70:0/3969080768 wait complete. 2024-04-22T01:14:59.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 Processor -- start 2024-04-22T01:14:59.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 -- start start 2024-04-22T01:14:59.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f300006f470 0x7f30000b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300015a520 0x7f300014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30000ae920 con 0x7f30000b5d20 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30000ae7a0 con 0x7f300015a520 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3000168c00 con 0x7f300006f470 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60896/0 (socket says 172.21.15.70:60896) 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 -- 172.21.15.70:0/501464651 learned_addr learned my addr 172.21.15.70:0/501464651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3005db5640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f300006f470 0x7f30000b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30065b6640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300015a520 0x7f300014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 -- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f300006f470 msgr2=0x7f30000b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f300006f470 0x7f30000b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 -- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300015a520 msgr2=0x7f300014d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300015a520 0x7f300014d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 -- 172.21.15.70:0/501464651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3000156240 con 0x7f30000b5d20 2024-04-22T01:14:59.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30065b6640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300015a520 0x7f300014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:59.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f30055b4640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000b01a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c3a0 tx=0x7f2ffc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc0052d0 con 0x7f30000b5d20 2024-04-22T01:14:59.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ffc005470 con 0x7f30000b5d20 2024-04-22T01:14:59.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3000156520 con 0x7f30000b5d20 2024-04-22T01:14:59.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc007260 con 0x7f30000b5d20 2024-04-22T01:14:59.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.495+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3000156ae0 con 0x7f30000b5d20 2024-04-22T01:14:59.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.499+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3000068210 con 0x7f30000b5d20 2024-04-22T01:14:59.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.499+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2ffc007400 con 0x7f30000b5d20 2024-04-22T01:14:59.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.499+0000 7f2ff67fc640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fe40678c0 0x7f2fe4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.499+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2ffc085470 con 0x7f30000b5d20 2024-04-22T01:14:59.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.499+0000 7f3005db5640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fe40678c0 0x7f2fe4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.499+0000 7f3005db5640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fe40678c0 0x7f2fe4069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0005890 tx=0x7f2ff0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.503+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2ffc055c70 con 0x7f30000b5d20 2024-04-22T01:14:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.663+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f300006ef90 con 0x7f30000b5d20 2024-04-22T01:14:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.663+0000 7f2ff67fc640 1 -- 172.21.15.70:0/501464651 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f2ffc055570 con 0x7f30000b5d20 2024-04-22T01:14:59.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:14:59.663 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fe40678c0 msgr2=0x7f2fe4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fe40678c0 0x7f2fe4069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0005890 tx=0x7f2ff0009040 comp rx=0 tx=0).stop 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 msgr2=0x7f30000b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000b01a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c3a0 tx=0x7f2ffc00c860 comp rx=0 tx=0).stop 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 shutdown_connections 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fe40678c0 0x7f2fe4069d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f300015a520 0x7f300014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30000b5d20 0x7f30000b01a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 --2- 172.21.15.70:0/501464651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f300006f470 0x7f30000b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 >> 172.21.15.70:0/501464651 conn(0x7f3000099a60 msgr2=0x7f3000152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 shutdown_connections 2024-04-22T01:14:59.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.667+0000 7f3008019640 1 -- 172.21.15.70:0/501464651 wait complete. 2024-04-22T01:14:59.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:14:59.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:14:59.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 Processor -- start 2024-04-22T01:14:59.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 -- start start 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbddc15f480 0x7fbddc0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbddc06b400 0x7fbddc06bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc0678d0 con 0x7fbddc06c140 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbddc06c590 con 0x7fbddc06b400 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbde12da640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbddc0673a0 con 0x7fbddc15f480 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbddc06b400 0x7fbddc06bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbddc06b400 0x7fbddc06bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60908/0 (socket says 172.21.15.70:60908) 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 -- 172.21.15.70:0/4005059160 learned_addr learned my addr 172.21.15.70:0/4005059160 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbddaffd640 1 --2- 172.21.15.70:0/4005059160 >> v2:172.21.15.98:3300/0 conn(0x7fbddc15f480 0x7fbddc0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 -- 172.21.15.70:0/4005059160 >> v2:172.21.15.98:3300/0 conn(0x7fbddc15f480 msgr2=0x7fbddc0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:59.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 --2- 172.21.15.70:0/4005059160 >> v2:172.21.15.98:3300/0 conn(0x7fbddc15f480 0x7fbddc0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 -- 172.21.15.70:0/4005059160 >> v1:172.21.15.98:6789/0 conn(0x7fbddc06c140 legacy=0x7fbddc0ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:59.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbdd3fff640 1 -- 172.21.15.70:0/4005059160 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbddc0675f0 con 0x7fbddc06b400 2024-04-22T01:14:59.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.799+0000 7fbddaffd640 1 --2- 172.21.15.70:0/4005059160 >> v2:172.21.15.98:3300/0 conn(0x7fbddc15f480 0x7fbddc0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 --2- 172.21.15.70:0/4005059160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06b400 0x7fbddc06bc00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbddc071a30 tx=0x7fbdc401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5733226e5f67df7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdda7fc640 1 -- 172.21.15.70:0/4005059160 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40b0070 con 0x7fbddc06b400 2024-04-22T01:14:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdda7fc640 1 -- 172.21.15.70:0/4005059160 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc40ab600 con 0x7fbddc06b400 2024-04-22T01:14:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdda7fc640 1 -- 172.21.15.70:0/4005059160 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40ab9d0 con 0x7fbddc06b400 2024-04-22T01:14:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/4005059160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06b400 msgr2=0x7fbddc06bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 --2- 172.21.15.70:0/4005059160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06b400 0x7fbddc06bc00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbddc071a30 tx=0x7fbdc401c4e0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/4005059160 shutdown_connections 2024-04-22T01:14:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 --2- 172.21.15.70:0/4005059160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06b400 0x7fbddc06bc00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 --2- 172.21.15.70:0/4005059160 >> v2:172.21.15.98:3300/0 conn(0x7fbddc15f480 0x7fbddc0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/4005059160 >> 172.21.15.70:0/4005059160 conn(0x7fbddc099a60 msgr2=0x7fbddc0b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:59.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/4005059160 shutdown_connections 2024-04-22T01:14:59.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/4005059160 wait complete. 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 Processor -- start 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- start start 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbddc06b400 0x7fbddc0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 0x7fbddc0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbddc15f480 0x7fbddc0c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbddc0b6120 con 0x7fbddc06c140 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbddc0b2c00 con 0x7fbddc06b400 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbddc15a6a0 con 0x7fbddc15f480 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 0x7fbddc0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 0x7fbddc0b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60924/0 (socket says 172.21.15.70:60924) 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 -- 172.21.15.70:0/1904306281 learned_addr learned my addr 172.21.15.70:0/1904306281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbddaffd640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbddc06b400 0x7fbddc0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbddb7fe640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbddc15f480 0x7fbddc0c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 -- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbddc15f480 msgr2=0x7fbddc0c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbddc15f480 0x7fbddc0c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 -- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbddc06b400 msgr2=0x7fbddc0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbddc06b400 0x7fbddc0b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 -- 172.21.15.70:0/1904306281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbddc15a980 con 0x7fbddc06c140 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbddb7fe640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbddc15f480 0x7fbddc0c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd3fff640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 0x7fbddc0b1ac0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbdc401a110 tx=0x7fbdc4019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40ba070 con 0x7fbddc06c140 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc40b4620 con 0x7fbddc06c140 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbddc15ac60 con 0x7fbddc06c140 2024-04-22T01:14:59.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40b0040 con 0x7fbddc06c140 2024-04-22T01:14:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.803+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbddc15b220 con 0x7fbddc06c140 2024-04-22T01:14:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.807+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbddc068210 con 0x7fbddc06c140 2024-04-22T01:14:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.807+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbdc40bd070 con 0x7fbddc06c140 2024-04-22T01:14:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.807+0000 7fbdd37fe640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbdbc0678f0 0x7fbdbc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.807+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbdc411f240 con 0x7fbddc06c140 2024-04-22T01:14:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.807+0000 7fbddaffd640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbdbc0678f0 0x7fbdbc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.807+0000 7fbddaffd640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbdbc0678f0 0x7fbdbc069db0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00a9a0 tx=0x7fbdcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.811+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbdc40160d0 con 0x7fbddc06c140 2024-04-22T01:14:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.959+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbddc072760 con 0x7fbddc06c140 2024-04-22T01:14:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.959+0000 7fbdd37fe640 1 -- 172.21.15.70:0/1904306281 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fbdc40efab0 con 0x7fbddc06c140 2024-04-22T01:14:59.958 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:14:59.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbdbc0678f0 msgr2=0x7fbdbc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbdbc0678f0 0x7fbdbc069db0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00a9a0 tx=0x7fbdcc008040 comp rx=0 tx=0).stop 2024-04-22T01:14:59.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 msgr2=0x7fbddc0b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:59.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 0x7fbddc0b1ac0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbdc401a110 tx=0x7fbdc4019280 comp rx=0 tx=0).stop 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 shutdown_connections 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbdbc0678f0 0x7fbdbc069db0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbddc15f480 0x7fbddc0c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbddc06c140 0x7fbddc0b1ac0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 --2- 172.21.15.70:0/1904306281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbddc06b400 0x7fbddc0b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 >> 172.21.15.70:0/1904306281 conn(0x7fbddc099a60 msgr2=0x7fbddc0c7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 shutdown_connections 2024-04-22T01:14:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:14:59.963+0000 7fbde12da640 1 -- 172.21.15.70:0/1904306281 wait complete. 2024-04-22T01:14:59.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:00.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 Processor -- start 2024-04-22T01:15:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 -- start start 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1f40b4450 0x7fe1f4071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f40720b0 0x7fe1f40ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f415c290 con 0x7fe1f40cd050 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe1f4072510 con 0x7fe1f40b4450 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1fb167640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f40b3a50 con 0x7fe1f40720b0 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f9704640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe1f40cd050 0x7fe1f415f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:47930/0 (socket says 172.21.15.70:47930) 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f9704640 1 -- 172.21.15.70:0/4216542186 learned_addr learned my addr 172.21.15.70:0/4216542186 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:00.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/4216542186 >> v2:172.21.15.98:3300/0 conn(0x7fe1f40b4450 0x7fe1f4071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1ebfff640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f40720b0 0x7fe1f40ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800616486 0 0) 0x7fe1f415c290 con 0x7fe1f40cd050 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1dc003530 con 0x7fe1f40cd050 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 107244969 0 0) 0x7fe1dc003530 con 0x7fe1f40cd050 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f40720b0 msgr2=0x7fe1f40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f40720b0 0x7fe1f40ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 >> v2:172.21.15.98:3300/0 conn(0x7fe1f40b4450 msgr2=0x7fe1f4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 --2- 172.21.15.70:0/4216542186 >> v2:172.21.15.98:3300/0 conn(0x7fe1f40b4450 0x7fe1f4071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f415c570 con 0x7fe1f40cd050 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fe1ec001a00 con 0x7fe1f40cd050 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 >> v1:172.21.15.70:6789/0 conn(0x7fe1f40cd050 legacy=0x7fe1f415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc004810 0x7fe1dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1dc0051d0 0x7fe1dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f40b3a50 con 0x7fe1dc0039c0 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc00bf10 con 0x7fe1dc004810 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe1dc00c090 con 0x7fe1dc0051d0 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1ebfff640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f40720b0 0x7fe1f40ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f9704640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1dc0051d0 0x7fe1dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1ebfff640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc004810 0x7fe1dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 -- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1dc0051d0 msgr2=0x7fe1dc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1dc0051d0 0x7fe1dc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 -- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc004810 msgr2=0x7fe1dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1dc004810 0x7fe1dc004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.103+0000 7fe1f8f03640 1 -- 172.21.15.70:0/4216542186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc003530 con 0x7fe1dc0039c0 2024-04-22T01:15:00.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1dc003de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009b10 tx=0x7fe1e401ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7842413caa8ad63c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40b2070 con 0x7fe1dc0039c0 2024-04-22T01:15:00.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e40a5560 con 0x7fe1dc0039c0 2024-04-22T01:15:00.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1eb7fe640 1 -- 172.21.15.70:0/4216542186 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40096b0 con 0x7fe1dc0039c0 2024-04-22T01:15:00.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 -- 172.21.15.70:0/4216542186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 msgr2=0x7fe1dc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 --2- 172.21.15.70:0/4216542186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1dc003de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009b10 tx=0x7fe1e401ca80 comp rx=0 tx=0).stop 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1f9704640 1 -- 172.21.15.70:0/4216542186 reap_dead start 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 -- 172.21.15.70:0/4216542186 shutdown_connections 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 -- 172.21.15.70:0/4216542186 >> 172.21.15.70:0/4216542186 conn(0x7fe1f40b1d40 msgr2=0x7fe1f40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 -- 172.21.15.70:0/4216542186 shutdown_connections 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 -- 172.21.15.70:0/4216542186 wait complete. 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 Processor -- start 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.107+0000 7fe1fb167640 1 -- start start 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1f406f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f41514b0 0x7fe1f4161b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f415aeb0 0x7fe1f41f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f40cd5e0 con 0x7fe1dc0039c0 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f40cd460 con 0x7fe1f415aeb0 2024-04-22T01:15:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f40b4830 con 0x7fe1f41514b0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1f406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1f406f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60944/0 (socket says 172.21.15.70:60944) 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 -- 172.21.15.70:0/1958063933 learned_addr learned my addr 172.21.15.70:0/1958063933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f9704640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f415aeb0 0x7fe1f41f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1ebfff640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f41514b0 0x7fe1f4161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 -- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f41514b0 msgr2=0x7fe1f4161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f41514b0 0x7fe1f4161b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 -- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f415aeb0 msgr2=0x7fe1f41f6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f415aeb0 0x7fe1f41f6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 -- 172.21.15.70:0/1958063933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f4162090 con 0x7fe1dc0039c0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1f8f03640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1f406f520 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009a30 tx=0x7fe1dc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40bd020 con 0x7fe1dc0039c0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e40a5d30 con 0x7fe1dc0039c0 2024-04-22T01:15:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f4160ab0 con 0x7fe1dc0039c0 2024-04-22T01:15:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40b2040 con 0x7fe1dc0039c0 2024-04-22T01:15:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f4161350 con 0x7fe1dc0039c0 2024-04-22T01:15:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f4160ea0 con 0x7fe1dc0039c0 2024-04-22T01:15:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1e4004260 con 0x7fe1dc0039c0 2024-04-22T01:15:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1e97fa640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1e00678c0 0x7fe1e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe1e411fbb0 con 0x7fe1dc0039c0 2024-04-22T01:15:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1ebfff640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1e00678c0 0x7fe1e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.111+0000 7fe1ebfff640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1e00678c0 0x7fe1e0069d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40c6810 tx=0x7fe1f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.115+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe1e4016800 con 0x7fe1dc0039c0 2024-04-22T01:15:00.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.267+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe1f4068210 con 0x7fe1dc0039c0 2024-04-22T01:15:00.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1e97fa640 1 -- 172.21.15.70:0/1958063933 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fe1e4123020 con 0x7fe1dc0039c0 2024-04-22T01:15:00.269 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1e00678c0 msgr2=0x7fe1e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1e00678c0 0x7fe1e0069d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40c6810 tx=0x7fe1f0008040 comp rx=0 tx=0).stop 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 msgr2=0x7fe1f406f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1f406f520 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009a30 tx=0x7fe1dc005840 comp rx=0 tx=0).stop 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 shutdown_connections 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1e00678c0 0x7fe1e0069d80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f415aeb0 0x7fe1f41f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1f41514b0 0x7fe1f4161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 --2- 172.21.15.70:0/1958063933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1dc0039c0 0x7fe1f406f520 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 >> 172.21.15.70:0/1958063933 conn(0x7fe1f40b1d40 msgr2=0x7fe1f40746a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:00.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 shutdown_connections 2024-04-22T01:15:00.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.271+0000 7fe1fb167640 1 -- 172.21.15.70:0/1958063933 wait complete. 2024-04-22T01:15:00.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 Processor -- start 2024-04-22T01:15:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 -- start start 2024-04-22T01:15:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa9b815ad50 0x7fa9b80cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b80d0f90 0x7fa9b80b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b80cdea0 con 0x7fa9b80cd4c0 2024-04-22T01:15:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b80ce180 con 0x7fa9b815ad50 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9be5c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b80cece0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa9b80cd4c0 0x7fa9b80d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:47940/0 (socket says 172.21.15.70:47940) 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b77fe640 1 -- 172.21.15.70:0/2791032254 learned_addr learned my addr 172.21.15.70:0/2791032254 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b7fff640 1 --2- 172.21.15.70:0/2791032254 >> v2:172.21.15.70:3300/0 conn(0x7fa9b815ad50 0x7fa9b80cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9bcb60640 1 --2- 172.21.15.70:0/2791032254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b80d0f90 0x7fa9b80b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b6ffd640 1 -- 172.21.15.70:0/2791032254 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810650960 0 0) 0x7fa9b80cdea0 con 0x7fa9b80cd4c0 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b6ffd640 1 -- 172.21.15.70:0/2791032254 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9a0003530 con 0x7fa9b80cd4c0 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b7fff640 1 -- 172.21.15.70:0/2791032254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b80d0f90 msgr2=0x7fa9b80b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b7fff640 1 --2- 172.21.15.70:0/2791032254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b80d0f90 0x7fa9b80b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b7fff640 1 -- 172.21.15.70:0/2791032254 >> v1:172.21.15.70:6789/0 conn(0x7fa9b80cd4c0 legacy=0x7fa9b80d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.407+0000 7fa9b7fff640 1 -- 172.21.15.70:0/2791032254 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b80cefc0 con 0x7fa9b815ad50 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b7fff640 1 --2- 172.21.15.70:0/2791032254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b815ad50 0x7fa9b80cf4e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009d90 tx=0x7fa9a801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4097a67648681d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b6ffd640 1 -- 172.21.15.70:0/2791032254 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80b2070 con 0x7fa9b815ad50 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b6ffd640 1 -- 172.21.15.70:0/2791032254 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a80a5830 con 0x7fa9b815ad50 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b6ffd640 1 -- 172.21.15.70:0/2791032254 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80a5c00 con 0x7fa9b815ad50 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2791032254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b815ad50 msgr2=0x7fa9b80cf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2791032254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b815ad50 0x7fa9b80cf4e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009d90 tx=0x7fa9a801c970 comp rx=0 tx=0).stop 2024-04-22T01:15:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2791032254 shutdown_connections 2024-04-22T01:15:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2791032254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b80d0f90 0x7fa9b80b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2791032254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b815ad50 0x7fa9b80cf4e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2791032254 >> 172.21.15.70:0/2791032254 conn(0x7fa9b8099a60 msgr2=0x7fa9b8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2791032254 shutdown_connections 2024-04-22T01:15:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2791032254 wait complete. 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 Processor -- start 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- start start 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b80cd4c0 0x7fa9b815c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 0x7fa9b81447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b815ad50 0x7fa9b81471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b815b6d0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b815b550 con 0x7fa9b80cd4c0 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b80b00b0 con 0x7fa9b815ad50 2024-04-22T01:15:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 0x7fa9b81447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 0x7fa9b81447a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60964/0 (socket says 172.21.15.70:60964) 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 -- 172.21.15.70:0/2638701801 learned_addr learned my addr 172.21.15.70:0/2638701801 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9bcb60640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b815ad50 0x7fa9b81471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 -- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b815ad50 msgr2=0x7fa9b81471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b815ad50 0x7fa9b81471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b7fff640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b80cd4c0 0x7fa9b815c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 -- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b80cd4c0 msgr2=0x7fa9b815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b80cd4c0 0x7fa9b815c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 -- 172.21.15.70:0/2638701801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b80b4110 con 0x7fa9b80d0f90 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b77fe640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 0x7fa9b81447a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00c450 tx=0x7fa9ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac017070 con 0x7fa9b80d0f90 2024-04-22T01:15:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac005cf0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b80b43d0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac007290 con 0x7fa9b80d0f90 2024-04-22T01:15:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.411+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b80b4990 con 0x7fa9b80d0f90 2024-04-22T01:15:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.415+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b8068210 con 0x7fa9b80d0f90 2024-04-22T01:15:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.415+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9ac01a070 con 0x7fa9b80d0f90 2024-04-22T01:15:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.415+0000 7fa9b4ff9640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9a40678c0 0x7fa9a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.415+0000 7fa9b7fff640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9a40678c0 0x7fa9a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.415+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa9ac0853b0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.415+0000 7fa9b7fff640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9a40678c0 0x7fa9a4069d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80af000 tx=0x7fa9a80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.419+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa9ac055bb0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.567+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa9b80b6a60 con 0x7fa9b80d0f90 2024-04-22T01:15:00.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.567+0000 7fa9b4ff9640 1 -- 172.21.15.70:0/2638701801 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fa9ac0554b0 con 0x7fa9b80d0f90 2024-04-22T01:15:00.567 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9a40678c0 msgr2=0x7fa9a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9a40678c0 0x7fa9a4069d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80af000 tx=0x7fa9a80a7070 comp rx=0 tx=0).stop 2024-04-22T01:15:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 msgr2=0x7fa9b81447a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 0x7fa9b81447a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00c450 tx=0x7fa9ac00c910 comp rx=0 tx=0).stop 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 shutdown_connections 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9a40678c0 0x7fa9a4069d80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9b815ad50 0x7fa9b81471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b80d0f90 0x7fa9b81447a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 --2- 172.21.15.70:0/2638701801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b80cd4c0 0x7fa9b815c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 >> 172.21.15.70:0/2638701801 conn(0x7fa9b8099a60 msgr2=0x7fa9b8159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 shutdown_connections 2024-04-22T01:15:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.571+0000 7fa9be5c3640 1 -- 172.21.15.70:0/2638701801 wait complete. 2024-04-22T01:15:00.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:00.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 Processor -- start 2024-04-22T01:15:00.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 -- start start 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3bc0b4450 0x7fd3bc071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3bc0720b0 0x7fd3bc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd3bc15c290 con 0x7fd3bc0b4450 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd3bc15c570 con 0x7fd3bc0720b0 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3c0824640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3bc15c850 con 0x7fd3bc0cd050 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3bc0b4450 0x7fd3bc071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3bc0b4450 0x7fd3bc071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60978/0 (socket says 172.21.15.70:60978) 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 -- 172.21.15.70:0/958091809 learned_addr learned my addr 172.21.15.70:0/958091809 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3bad9d640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 -- 172.21.15.70:0/958091809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 msgr2=0x7fd3bc15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3b9d9b640 1 --2- 172.21.15.70:0/958091809 >> v2:172.21.15.98:3300/0 conn(0x7fd3bc0720b0 0x7fd3bc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 -- 172.21.15.70:0/958091809 >> v2:172.21.15.98:3300/0 conn(0x7fd3bc0720b0 msgr2=0x7fd3bc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 --2- 172.21.15.70:0/958091809 >> v2:172.21.15.98:3300/0 conn(0x7fd3bc0720b0 0x7fd3bc0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3ba59c640 1 -- 172.21.15.70:0/958091809 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc15cb30 con 0x7fd3bc0b4450 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3bad9d640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.711+0000 7fd3b9d9b640 1 --2- 172.21.15.70:0/958091809 >> v2:172.21.15.98:3300/0 conn(0x7fd3bc0720b0 0x7fd3bc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:00.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc071b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc0cf9a0 tx=0x7fd3ac01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccb3e5e53aac201 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3b959a640 1 -- 172.21.15.70:0/958091809 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0b0070 con 0x7fd3bc0b4450 2024-04-22T01:15:00.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3b959a640 1 -- 172.21.15.70:0/958091809 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac0ab600 con 0x7fd3bc0b4450 2024-04-22T01:15:00.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3b959a640 1 -- 172.21.15.70:0/958091809 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0ab9d0 con 0x7fd3bc0b4450 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- 172.21.15.70:0/958091809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0b4450 msgr2=0x7fd3bc071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc071b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc0cf9a0 tx=0x7fd3ac01c260 comp rx=0 tx=0).stop 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- 172.21.15.70:0/958091809 shutdown_connections 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- 172.21.15.70:0/958091809 >> v2:172.21.15.98:3300/0 conn(0x7fd3bc0720b0 0x7fd3bc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- 172.21.15.70:0/958091809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc071b70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- 172.21.15.70:0/958091809 >> 172.21.15.70:0/958091809 conn(0x7fd3bc0b1d40 msgr2=0x7fd3bc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:00.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- 172.21.15.70:0/958091809 shutdown_connections 2024-04-22T01:15:00.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- 172.21.15.70:0/958091809 wait complete. 2024-04-22T01:15:00.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 Processor -- start 2024-04-22T01:15:00.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- start start 2024-04-22T01:15:00.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 0x7fd3bc14b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc14e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc0cd980 con 0x7fd3bc0720b0 2024-04-22T01:15:00.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc0cd800 con 0x7fd3bc0b4450 2024-04-22T01:15:00.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3c0824640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3bc0b2a40 con 0x7fd3bc0cd050 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 0x7fd3bc14b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 0x7fd3bc14b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60984/0 (socket says 172.21.15.70:60984) 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 -- 172.21.15.70:0/1498487043 learned_addr learned my addr 172.21.15.70:0/1498487043 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3bad9d640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3b9d9b640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc14e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 -- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 msgr2=0x7fd3bc14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 -- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3bc0b4450 msgr2=0x7fd3bc14e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc14e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 -- 172.21.15.70:0/1498487043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc14c2c0 con 0x7fd3bc0720b0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.715+0000 7fd3ba59c640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 0x7fd3bc14b050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0ae040 tx=0x7fd3ac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0b0040 con 0x7fd3bc0720b0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac0b9070 con 0x7fd3bc0720b0 2024-04-22T01:15:00.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3bc14d160 con 0x7fd3bc0720b0 2024-04-22T01:15:00.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0b48c0 con 0x7fd3bc0720b0 2024-04-22T01:15:00.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3bc14cba0 con 0x7fd3bc0720b0 2024-04-22T01:15:00.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3bc14c490 con 0x7fd3bc0720b0 2024-04-22T01:15:00.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3ac0ab4c0 con 0x7fd3bc0720b0 2024-04-22T01:15:00.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3aaffd640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd39c0678c0 0x7fd39c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:00.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd3ac11fd10 con 0x7fd3bc0720b0 2024-04-22T01:15:00.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3b9d9b640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd39c0678c0 0x7fd39c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:00.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.719+0000 7fd3b9d9b640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd39c0678c0 0x7fd39c069d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc165e00 tx=0x7fd3b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:00.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.723+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd3ac016110 con 0x7fd3bc0720b0 2024-04-22T01:15:00.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.883+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd3bc068210 con 0x7fd3bc0720b0 2024-04-22T01:15:00.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.883+0000 7fd3aaffd640 1 -- 172.21.15.70:0/1498487043 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fd3ac0f05a0 con 0x7fd3bc0720b0 2024-04-22T01:15:00.884 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:00.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd39c0678c0 msgr2=0x7fd39c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd39c0678c0 0x7fd39c069d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc165e00 tx=0x7fd3b0008040 comp rx=0 tx=0).stop 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 msgr2=0x7fd3bc14b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 0x7fd3bc14b050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0ae040 tx=0x7fd3ac019000 comp rx=0 tx=0).stop 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 shutdown_connections 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd39c0678c0 0x7fd39c069d80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3bc0cd050 0x7fd3bc14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3bc0b4450 0x7fd3bc14e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 --2- 172.21.15.70:0/1498487043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3bc0720b0 0x7fd3bc14b050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 >> 172.21.15.70:0/1498487043 conn(0x7fd3bc0b1d40 msgr2=0x7fd3bc0b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:00.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 shutdown_connections 2024-04-22T01:15:00.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:00.887+0000 7fd3c0824640 1 -- 172.21.15.70:0/1498487043 wait complete. 2024-04-22T01:15:00.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 Processor -- start 2024-04-22T01:15:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 -- start start 2024-04-22T01:15:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47b015bb50 0x7f47b00cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b006f480 0x7f47b0071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b00cd7a0 con 0x7f47b015bf30 2024-04-22T01:15:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f47b015f8b0 con 0x7f47b015bb50 2024-04-22T01:15:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47b614b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f47b015fb00 con 0x7f47b006f480 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47b015bb50 0x7f47b00cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47b015bb50 0x7f47b00cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60998/0 (socket says 172.21.15.70:60998) 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 -- 172.21.15.70:0/3678759906 learned_addr learned my addr 172.21.15.70:0/3678759906 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47affff640 1 --2- 172.21.15.70:0/3678759906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b006f480 0x7f47b0071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 -- 172.21.15.70:0/3678759906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b006f480 msgr2=0x7f47b0071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 --2- 172.21.15.70:0/3678759906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b006f480 0x7f47b0071af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 -- 172.21.15.70:0/3678759906 >> v1:172.21.15.98:6789/0 conn(0x7f47b015bf30 legacy=0x7f47b006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.023+0000 7f47af7fe640 1 -- 172.21.15.70:0/3678759906 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b015fde0 con 0x7f47b015bb50 2024-04-22T01:15:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47af7fe640 1 --2- 172.21.15.70:0/3678759906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bb50 0x7f47b00cce70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009870 tx=0x7f47a801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2fd51fb5e40d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47ae7fc640 1 -- 172.21.15.70:0/3678759906 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a80ab5f0 con 0x7f47b015bb50 2024-04-22T01:15:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47ae7fc640 1 -- 172.21.15.70:0/3678759906 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a80b0070 con 0x7f47b015bb50 2024-04-22T01:15:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47ae7fc640 1 -- 172.21.15.70:0/3678759906 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a80ab9b0 con 0x7f47b015bb50 2024-04-22T01:15:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- 172.21.15.70:0/3678759906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bb50 msgr2=0x7f47b00cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 --2- 172.21.15.70:0/3678759906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bb50 0x7f47b00cce70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009870 tx=0x7f47a801c520 comp rx=0 tx=0).stop 2024-04-22T01:15:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- 172.21.15.70:0/3678759906 shutdown_connections 2024-04-22T01:15:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 --2- 172.21.15.70:0/3678759906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b006f480 0x7f47b0071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 --2- 172.21.15.70:0/3678759906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bb50 0x7f47b00cce70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- 172.21.15.70:0/3678759906 >> 172.21.15.70:0/3678759906 conn(0x7f47b00cf9e0 msgr2=0x7f47b00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- 172.21.15.70:0/3678759906 shutdown_connections 2024-04-22T01:15:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- 172.21.15.70:0/3678759906 wait complete. 2024-04-22T01:15:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 Processor -- start 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- start start 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b006f480 0x7f47b015d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b015bb50 0x7f47b01510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 0x7f47b006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47b0160760 con 0x7f47b015bf30 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47b01605e0 con 0x7f47b006f480 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f47b00cf460 con 0x7f47b015bb50 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 0x7f47b006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 0x7f47b006e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32776/0 (socket says 172.21.15.70:32776) 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 -- 172.21.15.70:0/769816328 learned_addr learned my addr 172.21.15.70:0/769816328 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47aeffd640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b015bb50 0x7f47b01510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47af7fe640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b006f480 0x7f47b015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 -- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b015bb50 msgr2=0x7f47b01510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b015bb50 0x7f47b01510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 -- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b006f480 msgr2=0x7f47b015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b006f480 0x7f47b015d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 -- 172.21.15.70:0/769816328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b015d5e0 con 0x7f47b015bf30 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47af7fe640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b006f480 0x7f47b015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47affff640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 0x7f47b006e000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f47a000cd70 tx=0x7f47a000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0003ba0 con 0x7f47b015bf30 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a0003d40 con 0x7f47b015bf30 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0004240 con 0x7f47b015bf30 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.027+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b015d8c0 con 0x7f47b015bf30 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b015de80 con 0x7f47b015bf30 2024-04-22T01:15:01.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b015e360 con 0x7f47b015bf30 2024-04-22T01:15:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f47a00043e0 con 0x7f47b015bf30 2024-04-22T01:15:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47a7fff640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47900678c0 0x7f4790069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f47a0086ab0 con 0x7f47b015bf30 2024-04-22T01:15:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47af7fe640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47900678c0 0x7f4790069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.031+0000 7f47af7fe640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47900678c0 0x7f4790069d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47a80af000 tx=0x7f47a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:01.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.035+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f47b015e360 con 0x7f47b015bf30 2024-04-22T01:15:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.195+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f47b004f510 con 0x7f47b015bf30 2024-04-22T01:15:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.195+0000 7f47a7fff640 1 -- 172.21.15.70:0/769816328 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f47a0057340 con 0x7f47b015bf30 2024-04-22T01:15:01.194 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47900678c0 msgr2=0x7f4790069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47900678c0 0x7f4790069d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47a80af000 tx=0x7f47a8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 msgr2=0x7f47b006e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 0x7f47b006e000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f47a000cd70 tx=0x7f47a000a660 comp rx=0 tx=0).stop 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 shutdown_connections 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47900678c0 0x7f4790069d80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47b015bf30 0x7f47b006e000 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47b015bb50 0x7f47b01510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 --2- 172.21.15.70:0/769816328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b006f480 0x7f47b015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:01.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 >> 172.21.15.70:0/769816328 conn(0x7f47b00cf9e0 msgr2=0x7f47b00b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 shutdown_connections 2024-04-22T01:15:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:01.199+0000 7f47b614b640 1 -- 172.21.15.70:0/769816328 wait complete. 2024-04-22T01:15:01.213 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:15:01.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:01.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:15:01.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[33395]: fuse finished with error 0 and tester_r 0 2024-04-22T01:15:01.244 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:15:07.346 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:15:07.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:07.346 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-22T01:15:07.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:07.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 Processor -- start 2024-04-22T01:15:07.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 -- start start 2024-04-22T01:15:07.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6b5c15dfb0 0x7f6b5c0b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b5c15e390 con 0x7f6b5c0cea80 2024-04-22T01:15:07.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6b5c0cf420 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b616e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5c161e00 con 0x7f6b5c0d3120 2024-04-22T01:15:07.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6b5c15dfb0 0x7f6b5c0b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6b5c15dfb0 0x7f6b5c0b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54412/0 (socket says 172.21.15.70:54412) 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 -- 172.21.15.70:0/3023904685 learned_addr learned my addr 172.21.15.70:0/3023904685 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5b7fe640 1 --2- 172.21.15.70:0/3023904685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 -- 172.21.15.70:0/3023904685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 msgr2=0x7f6b5c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 --2- 172.21.15.70:0/3023904685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 -- 172.21.15.70:0/3023904685 >> v1:172.21.15.98:6789/0 conn(0x7f6b5c0cea80 legacy=0x7f6b5c0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:07.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 -- 172.21.15.70:0/3023904685 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b5c1620e0 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.603+0000 7f6b5affd640 1 --2- 172.21.15.70:0/3023904685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c0b95d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0d4d80 tx=0x7f6b5401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd94ae84625f936 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b59ffb640 1 -- 172.21.15.70:0/3023904685 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b540a5660 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b59ffb640 1 -- 172.21.15.70:0/3023904685 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b540a5800 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b59ffb640 1 -- 172.21.15.70:0/3023904685 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b540a5bd0 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/3023904685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 msgr2=0x7f6b5c0b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 --2- 172.21.15.70:0/3023904685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c0b95d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0d4d80 tx=0x7f6b5401c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/3023904685 shutdown_connections 2024-04-22T01:15:07.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 --2- 172.21.15.70:0/3023904685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 --2- 172.21.15.70:0/3023904685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c0b95d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/3023904685 >> 172.21.15.70:0/3023904685 conn(0x7f6b5c06c9b0 msgr2=0x7f6b5c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:07.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/3023904685 shutdown_connections 2024-04-22T01:15:07.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/3023904685 wait complete. 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 Processor -- start 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- start start 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c0cea80 0x7f6b5c153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5c16add0 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5c161a90 con 0x7f6b5c0cea80 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5c16af50 con 0x7f6b5c0d3120 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c14dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54420/0 (socket says 172.21.15.70:54420) 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 -- 172.21.15.70:0/501775071 learned_addr learned my addr 172.21.15.70:0/501775071 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5a7fc640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5affd640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c0cea80 0x7f6b5c153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 -- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 msgr2=0x7f6b5c0b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 -- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c0cea80 msgr2=0x7f6b5c153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c0cea80 0x7f6b5c153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 -- 172.21.15.70:0/501775071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b5c15d190 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5a7fc640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5affd640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c0cea80 0x7f6b5c153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b5b7fe640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c14dac0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00d550 tx=0x7f6b4c00da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c0125f0 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b4c019070 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b5c15d470 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c0129d0 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.607+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b5c0b8260 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.611+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b5c068210 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.611+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6b4c012b70 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.611+0000 7f6b537fe640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6b3c0678c0 0x7f6b3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.611+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6b4c086e10 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.611+0000 7f6b5affd640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6b3c0678c0 0x7f6b3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.611+0000 7f6b5affd640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6b3c0678c0 0x7f6b3c069d80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b540ae000 tx=0x7f6b540a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:07.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.615+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6b4c057680 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.775+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b5c0b6810 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b537fe640 1 -- 172.21.15.70:0/501775071 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f6b4c056f80 con 0x7f6b5c15dfb0 2024-04-22T01:15:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:07.778 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6b3c0678c0 msgr2=0x7f6b3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6b3c0678c0 0x7f6b3c069d80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b540ae000 tx=0x7f6b540a7070 comp rx=0 tx=0).stop 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 msgr2=0x7f6b5c14dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c14dac0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00d550 tx=0x7f6b4c00da10 comp rx=0 tx=0).stop 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 shutdown_connections 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6b3c0678c0 0x7f6b3c069d80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b5c15dfb0 0x7f6b5c14dac0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0d3120 0x7f6b5c0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 --2- 172.21.15.70:0/501775071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c0cea80 0x7f6b5c153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 >> 172.21.15.70:0/501775071 conn(0x7f6b5c06c9b0 msgr2=0x7f6b5c0d3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 shutdown_connections 2024-04-22T01:15:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.779+0000 7f6b616e2640 1 -- 172.21.15.70:0/501775071 wait complete. 2024-04-22T01:15:07.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:07.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.919+0000 7fefc23ea640 1 Processor -- start 2024-04-22T01:15:07.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.919+0000 7fefc23ea640 1 -- start start 2024-04-22T01:15:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc23ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefbc071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc23ea640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefbc072060 0x7fefbc06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc23ea640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc0cd7b0 con 0x7fefbc1455b0 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc23ea640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fefbc0cc710 con 0x7fefbc072060 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc23ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fefbc0cc9f0 con 0x7fefbc0cc150 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fefbc1455b0 0x7fefbc0cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40680/0 (socket says 172.21.15.70:40680) 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbbfff640 1 -- 172.21.15.70:0/2185331452 learned_addr learned my addr 172.21.15.70:0/2185331452 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc0987640 1 --2- 172.21.15.70:0/2185331452 >> v2:172.21.15.98:3300/0 conn(0x7fefbc072060 0x7fefbc06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982022756 0 0) 0x7fefbc0cd7b0 con 0x7fefbc1455b0 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefbc071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefa4003530 con 0x7fefbc1455b0 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4208669083 0 0) 0x7fefa4003530 con 0x7fefbc1455b0 2024-04-22T01:15:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 msgr2=0x7fefbc071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefbc071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 >> v2:172.21.15.98:3300/0 conn(0x7fefbc072060 msgr2=0x7fefbc06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 --2- 172.21.15.70:0/2185331452 >> v2:172.21.15.98:3300/0 conn(0x7fefbc072060 0x7fefbc06e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc0cccd0 con 0x7fefbc1455b0 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fefac001680 con 0x7fefbc1455b0 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefbc071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 >> v1:172.21.15.70:6789/0 conn(0x7fefbc1455b0 legacy=0x7fefbc0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40039c0 0x7fefa4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefa4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefa40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc0cc710 con 0x7fefa4004810 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc0cc9f0 con 0x7fefa40039c0 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fefa400bb00 con 0x7fefbc0cc150 2024-04-22T01:15:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefa4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbbfff640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40039c0 0x7fefa4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefc0987640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefa40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 -- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 msgr2=0x7fefa40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc0cc150 0x7fefa40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 -- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40039c0 msgr2=0x7fefa4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40039c0 0x7fefa4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.923+0000 7fefbb7fe640 1 -- 172.21.15.70:0/2185331452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa4003530 con 0x7fefa4004810 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefa4004c90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fefb00098a0 tx=0x7fefb001c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc180a06ac19d9b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb00b1070 con 0x7fefa4004810 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefb00ac600 con 0x7fefa4004810 2024-04-22T01:15:07.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbaffd640 1 -- 172.21.15.70:0/2185331452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb000f040 con 0x7fefa4004810 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- 172.21.15.70:0/2185331452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 msgr2=0x7fefa4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 --2- 172.21.15.70:0/2185331452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefa4004c90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fefb00098a0 tx=0x7fefb001c570 comp rx=0 tx=0).stop 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc0987640 1 -- 172.21.15.70:0/2185331452 reap_dead start 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- 172.21.15.70:0/2185331452 shutdown_connections 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- 172.21.15.70:0/2185331452 >> 172.21.15.70:0/2185331452 conn(0x7fefbc06b1f0 msgr2=0x7fefbc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- 172.21.15.70:0/2185331452 shutdown_connections 2024-04-22T01:15:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- 172.21.15.70:0/2185331452 wait complete. 2024-04-22T01:15:07.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 Processor -- start 2024-04-22T01:15:07.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- start start 2024-04-22T01:15:07.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefbc0ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc13e3e0 0x7fefbc0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc15b080 0x7fefbc15b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc06f2e0 con 0x7fefa4004810 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc06f160 con 0x7fefbc15b080 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fefbc072470 con 0x7fefbc13e3e0 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefbc0ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefbc0ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54436/0 (socket says 172.21.15.70:54436) 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 -- 172.21.15.70:0/3344319154 learned_addr learned my addr 172.21.15.70:0/3344319154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc13e3e0 0x7fefbc0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 -- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc13e3e0 msgr2=0x7fefbc0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc13e3e0 0x7fefbc0b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 -- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc15b080 msgr2=0x7fefbc15b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc15b080 0x7fefbc15b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 -- 172.21.15.70:0/3344319154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc15ec50 con 0x7fefa4004810 2024-04-22T01:15:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefbbfff640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefbc0ae090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fefac004f00 tx=0x7fefac007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac00f730 con 0x7fefa4004810 2024-04-22T01:15:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac018070 con 0x7fefa4004810 2024-04-22T01:15:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.927+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefbc15ef30 con 0x7fefa4004810 2024-04-22T01:15:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac00fb10 con 0x7fefa4004810 2024-04-22T01:15:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefbc0b0c50 con 0x7fefa4004810 2024-04-22T01:15:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefbc15f100 con 0x7fefa4004810 2024-04-22T01:15:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fefac00fcb0 con 0x7fefa4004810 2024-04-22T01:15:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefb8ff9640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefa80678c0 0x7fefa8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fefac085ed0 con 0x7fefa4004810 2024-04-22T01:15:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefa80678c0 0x7fefa8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.931+0000 7fefbb7fe640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefa80678c0 0x7fefa8069d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fefb00ae000 tx=0x7fefb00046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:07.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:07.935+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fefbc15f100 con 0x7fefa4004810 2024-04-22T01:15:08.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.099+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefbc0b29f0 con 0x7fefa4004810 2024-04-22T01:15:08.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.099+0000 7fefb8ff9640 1 -- 172.21.15.70:0/3344319154 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7fefac056760 con 0x7fefa4004810 2024-04-22T01:15:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:08.100 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:08.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefa80678c0 msgr2=0x7fefa8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefa80678c0 0x7fefa8069d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fefb00ae000 tx=0x7fefb00046d0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 msgr2=0x7fefbc0ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefbc0ae090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fefac004f00 tx=0x7fefac007430 comp rx=0 tx=0).stop 2024-04-22T01:15:08.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 shutdown_connections 2024-04-22T01:15:08.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefa80678c0 0x7fefa8069d80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc15b080 0x7fefbc15b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefbc13e3e0 0x7fefbc0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 --2- 172.21.15.70:0/3344319154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefa4004810 0x7fefbc0ae090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 >> 172.21.15.70:0/3344319154 conn(0x7fefbc06b1f0 msgr2=0x7fefbc142bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 shutdown_connections 2024-04-22T01:15:08.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.103+0000 7fefc23ea640 1 -- 172.21.15.70:0/3344319154 wait complete. 2024-04-22T01:15:08.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:15:08.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 Processor -- start 2024-04-22T01:15:08.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 -- start start 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f903c15def0 0x7f903c0cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f903c0cfce0 0x7f903c0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f903c15e2d0 con 0x7f903c0d06f0 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f903c0d0b20 con 0x7f903c0cfce0 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9041c6d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f903c15e570 con 0x7f903c15def0 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f903c0cfce0 0x7f903c0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f903c0cfce0 0x7f903c0d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54442/0 (socket says 172.21.15.70:54442) 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 -- 172.21.15.70:0/128252095 learned_addr learned my addr 172.21.15.70:0/128252095 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f903b7fe640 1 --2- 172.21.15.70:0/128252095 >> v2:172.21.15.98:3300/0 conn(0x7f903c15def0 0x7f903c0cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 -- 172.21.15.70:0/128252095 >> v2:172.21.15.98:3300/0 conn(0x7f903c15def0 msgr2=0x7f903c0cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 --2- 172.21.15.70:0/128252095 >> v2:172.21.15.98:3300/0 conn(0x7f903c15def0 0x7f903c0cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 -- 172.21.15.70:0/128252095 >> v1:172.21.15.70:6789/0 conn(0x7f903c0d06f0 legacy=0x7f903c06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 -- 172.21.15.70:0/128252095 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c15e810 con 0x7f903c0cfce0 2024-04-22T01:15:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.239+0000 7f9033fff640 1 --2- 172.21.15.70:0/128252095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c0cfce0 0x7f903c0d01b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9024009870 tx=0x7f902401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937e82075f1dcf81 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903affd640 1 -- 172.21.15.70:0/128252095 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240aa650 con 0x7f903c0cfce0 2024-04-22T01:15:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903affd640 1 -- 172.21.15.70:0/128252095 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90240af070 con 0x7f903c0cfce0 2024-04-22T01:15:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903affd640 1 -- 172.21.15.70:0/128252095 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240aaa10 con 0x7f903c0cfce0 2024-04-22T01:15:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/128252095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c0cfce0 msgr2=0x7f903c0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 --2- 172.21.15.70:0/128252095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c0cfce0 0x7f903c0d01b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9024009870 tx=0x7f902401c4f0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/128252095 shutdown_connections 2024-04-22T01:15:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 --2- 172.21.15.70:0/128252095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c0cfce0 0x7f903c0d01b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 --2- 172.21.15.70:0/128252095 >> v2:172.21.15.98:3300/0 conn(0x7f903c15def0 0x7f903c0cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/128252095 >> 172.21.15.70:0/128252095 conn(0x7f903c0b36c0 msgr2=0x7f903c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/128252095 shutdown_connections 2024-04-22T01:15:08.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/128252095 wait complete. 2024-04-22T01:15:08.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 Processor -- start 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- start start 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903c0cfce0 0x7f903c0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f903c0d06f0 0x7f903c168ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 0x7f903c169220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f903c09afd0 con 0x7f903c15def0 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f903c09ae50 con 0x7f903c0cfce0 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f903c06dfa0 con 0x7f903c0d06f0 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 0x7f903c169220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 0x7f903c169220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54454/0 (socket says 172.21.15.70:54454) 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 -- 172.21.15.70:0/2905020827 learned_addr learned my addr 172.21.15.70:0/2905020827 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 -- 172.21.15.70:0/2905020827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f903c0d06f0 msgr2=0x7f903c168ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f903c0d06f0 0x7f903c168ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 -- 172.21.15.70:0/2905020827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903c0cfce0 msgr2=0x7f903c0cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903c0cfce0 0x7f903c0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 -- 172.21.15.70:0/2905020827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c162940 con 0x7f903c15def0 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f903bfff640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 0x7f903c169220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f903400c730 tx=0x7f903400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034016020 con 0x7f903c15def0 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9034006d60 con 0x7f903c15def0 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903c0c7fc0 con 0x7f903c15def0 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034003bc0 con 0x7f903c15def0 2024-04-22T01:15:08.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.243+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f903c140f40 con 0x7f903c15def0 2024-04-22T01:15:08.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.247+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f903c0c83b0 con 0x7f903c15def0 2024-04-22T01:15:08.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.247+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f90340050d0 con 0x7f903c15def0 2024-04-22T01:15:08.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.247+0000 7f9038ff9640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f901c0678c0 0x7f901c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.247+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9034085b40 con 0x7f903c15def0 2024-04-22T01:15:08.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.247+0000 7f903b7fe640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f901c0678c0 0x7f901c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.247+0000 7f903b7fe640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f901c0678c0 0x7f901c069d80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f902c00a8b0 tx=0x7f902c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.251+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9034020020 con 0x7f903c15def0 2024-04-22T01:15:08.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.439+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f903c0b45d0 con 0x7f903c15def0 2024-04-22T01:15:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.815+0000 7f9038ff9640 1 -- 172.21.15.70:0/2905020827 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f90340079d0 con 0x7f903c15def0 2024-04-22T01:15:08.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f901c0678c0 msgr2=0x7f901c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f901c0678c0 0x7f901c069d80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f902c00a8b0 tx=0x7f902c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 msgr2=0x7f903c169220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 0x7f903c169220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f903400c730 tx=0x7f903400cb30 comp rx=0 tx=0).stop 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 shutdown_connections 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f901c0678c0 0x7f901c069d80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f903c15def0 0x7f903c169220 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f903c0d06f0 0x7f903c168ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 --2- 172.21.15.70:0/2905020827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903c0cfce0 0x7f903c0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 >> 172.21.15.70:0/2905020827 conn(0x7f903c0b36c0 msgr2=0x7f903c0744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 shutdown_connections 2024-04-22T01:15:08.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.819+0000 7f9041c6d640 1 -- 172.21.15.70:0/2905020827 wait complete. 2024-04-22T01:15:08.839 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-22T01:15:08.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:15:08.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.963+0000 7f16a154f640 1 Processor -- start 2024-04-22T01:15:08.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.963+0000 7f16a154f640 1 -- start start 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f16a154f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 0x7f169c071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f16a154f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c06da70 con 0x7f169c0cd410 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f16a154f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c06dd50 con 0x7f169c15a320 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f16a154f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f169c06f860 con 0x7f169c071b70 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f169c0cd410 0x7f169c0cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40694/0 (socket says 172.21.15.70:40694) 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169b7fe640 1 -- 172.21.15.70:0/2588187590 learned_addr learned my addr 172.21.15.70:0/2588187590 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 0x7f169c071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3188572214 0 0) 0x7f169c06da70 con 0x7f169c0cd410 2024-04-22T01:15:08.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1680004640 con 0x7f169c0cd410 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1801453187 0 0) 0x7f1680004640 con 0x7f169c0cd410 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 msgr2=0x7f169c071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 0x7f169c071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 >> v1:172.21.15.98:6789/0 conn(0x7f169c15a320 legacy=0x7f169c06f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c06e030 con 0x7f169c0cd410 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1694001a00 con 0x7f169c0cd410 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 >> v1:172.21.15.70:6789/0 conn(0x7f169c0cd410 legacy=0x7f169c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680004b00 0x7f1680004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1680005920 0x7f1680005da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 0x7f16800062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f169c06f860 con 0x7f1680005920 2024-04-22T01:15:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1680003520 con 0x7f1680004b00 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1680003720 con 0x7f169c071b70 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1680005920 0x7f1680005da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 -- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 msgr2=0x7f16800062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169b7fe640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 0x7f16800062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169affd640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680004b00 0x7f1680004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c071b70 0x7f16800062e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 -- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680004b00 msgr2=0x7f1680004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680004b00 0x7f1680004f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.967+0000 7f169a7fc640 1 -- 172.21.15.70:0/2588187590 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1680004640 con 0x7f1680005920 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169a7fc640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1680005920 0x7f1680005da0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1688009ae0 tx=0x7f168801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca69f63abba47c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16880b2070 con 0x7f1680005920 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16880a5660 con 0x7f1680005920 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f1699ffb640 1 -- 172.21.15.70:0/2588187590 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16880096b0 con 0x7f1680005920 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/2588187590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1680005920 msgr2=0x7f1680005da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 --2- 172.21.15.70:0/2588187590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1680005920 0x7f1680005da0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1688009ae0 tx=0x7f168801ca70 comp rx=0 tx=0).stop 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 -- 172.21.15.70:0/2588187590 reap_dead start 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/2588187590 shutdown_connections 2024-04-22T01:15:08.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/2588187590 >> 172.21.15.70:0/2588187590 conn(0x7f169c0b1d40 msgr2=0x7f169c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/2588187590 shutdown_connections 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/2588187590 wait complete. 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 Processor -- start 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- start start 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680005920 0x7f169c1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c148ac0 0x7f169c15a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 0x7f169c157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f169c15a700 con 0x7f169c157790 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f169c06e5f0 con 0x7f1680005920 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f169c0cf420 con 0x7f169c148ac0 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 0x7f169c157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 0x7f169c157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54480/0 (socket says 172.21.15.70:54480) 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 -- 172.21.15.70:0/3240674652 learned_addr learned my addr 172.21.15.70:0/3240674652 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:08.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169affd640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680005920 0x7f169c1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169a7fc640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c148ac0 0x7f169c15a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 -- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c148ac0 msgr2=0x7f169c15a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c148ac0 0x7f169c15a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 -- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680005920 msgr2=0x7f169c1f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680005920 0x7f169c1f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 -- 172.21.15.70:0/3240674652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c148f50 con 0x7f169c157790 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f169b7fe640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 0x7f169c157c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f169c072ff0 tx=0x7f1694005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169401a070 con 0x7f169c157790 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1694011040 con 0x7f169c157790 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c149230 con 0x7f169c157790 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169400ed80 con 0x7f169c157790 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.971+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c1497f0 con 0x7f169c157790 2024-04-22T01:15:08.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.975+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c068210 con 0x7f169c157790 2024-04-22T01:15:08.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.975+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f169401d070 con 0x7f169c157790 2024-04-22T01:15:08.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.975+0000 7f16937fe640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16840678c0 0x7f1684069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.975+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1694088930 con 0x7f169c157790 2024-04-22T01:15:08.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.975+0000 7f169affd640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16840678c0 0x7f1684069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.975+0000 7f169affd640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16840678c0 0x7f1684069d80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f168c007fd0 tx=0x7f168c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:08.979+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1694058a40 con 0x7f169c157790 2024-04-22T01:15:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.175+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f169c1f2de0 con 0x7f169c157790 2024-04-22T01:15:09.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.814+0000 7f16937fe640 1 -- 172.21.15.70:0/3240674652 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f169405cd40 con 0x7f169c157790 2024-04-22T01:15:09.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16840678c0 msgr2=0x7f1684069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:09.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16840678c0 0x7f1684069d80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f168c007fd0 tx=0x7f168c009040 comp rx=0 tx=0).stop 2024-04-22T01:15:09.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 msgr2=0x7f169c157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:09.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 0x7f169c157c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f169c072ff0 tx=0x7f1694005ac0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 shutdown_connections 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16840678c0 0x7f1684069d80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f169c157790 0x7f169c157c10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c148ac0 0x7f169c15a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 --2- 172.21.15.70:0/3240674652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1680005920 0x7f169c1f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 >> 172.21.15.70:0/3240674652 conn(0x7f169c0b1d40 msgr2=0x7f169c147040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 shutdown_connections 2024-04-22T01:15:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.818+0000 7f16a154f640 1 -- 172.21.15.70:0/3240674652 wait complete. 2024-04-22T01:15:09.839 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:15:09.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:09.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3ca2e8640 1 Processor -- start 2024-04-22T01:15:09.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3ca2e8640 1 -- start start 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3ca2e8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe3c40cc150 0x7fe3c4071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3ca2e8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c40cd7b0 con 0x7fe3c41455b0 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3ca2e8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c40cc710 con 0x7fe3c4072060 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3ca2e8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c4072490 con 0x7fe3c40cc150 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe3c40cc150 0x7fe3c4071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe3c41455b0 0x7fe3c40cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40700/0 (socket says 172.21.15.70:40700) 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c3fff640 1 -- 172.21.15.70:0/858065454 learned_addr learned my addr 172.21.15.70:0/858065454 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c37fe640 1 -- 172.21.15.70:0/858065454 >> v1:172.21.15.98:6789/0 conn(0x7fe3c4072060 legacy=0x7fe3c406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c37fe640 1 -- 172.21.15.70:0/858065454 >> v1:172.21.15.70:6789/0 conn(0x7fe3c41455b0 legacy=0x7fe3c40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c2ffd640 1 -- 172.21.15.70:0/858065454 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550284218 0 0) 0x7fe3c40cd7b0 con 0x7fe3c41455b0 2024-04-22T01:15:09.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.962+0000 7fe3c37fe640 1 -- 172.21.15.70:0/858065454 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c40cc9f0 con 0x7fe3c40cc150 2024-04-22T01:15:09.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c37fe640 1 --2- 172.21.15.70:0/858065454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c40cc150 0x7fe3c4071b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0097c0 tx=0x7fe3ac01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2899bc5667f89db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:09.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c2ffd640 1 -- 172.21.15.70:0/858065454 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0b0070 con 0x7fe3c40cc150 2024-04-22T01:15:09.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c2ffd640 1 -- 172.21.15.70:0/858065454 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3ac0ab600 con 0x7fe3c40cc150 2024-04-22T01:15:09.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c2ffd640 1 -- 172.21.15.70:0/858065454 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0ab9d0 con 0x7fe3c40cc150 2024-04-22T01:15:09.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/858065454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c40cc150 msgr2=0x7fe3c4071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:09.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/858065454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c40cc150 0x7fe3c4071b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0097c0 tx=0x7fe3ac01c340 comp rx=0 tx=0).stop 2024-04-22T01:15:09.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/858065454 shutdown_connections 2024-04-22T01:15:09.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/858065454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c40cc150 0x7fe3c4071b20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/858065454 >> 172.21.15.70:0/858065454 conn(0x7fe3c406b1f0 msgr2=0x7fe3c406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:09.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/858065454 shutdown_connections 2024-04-22T01:15:09.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/858065454 wait complete. 2024-04-22T01:15:09.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 Processor -- start 2024-04-22T01:15:09.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- start start 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 0x7fe3c4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c40cc150 0x7fe3c415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c41455b0 0x7fe3c415e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4169ee0 con 0x7fe3c4072060 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4169d60 con 0x7fe3c41455b0 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3ca2e8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c416a060 con 0x7fe3c40cc150 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 0x7fe3c4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 0x7fe3c4159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54492/0 (socket says 172.21.15.70:54492) 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 -- 172.21.15.70:0/2171596192 learned_addr learned my addr 172.21.15.70:0/2171596192 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:09.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c8885640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c41455b0 0x7fe3c415e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c37fe640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c40cc150 0x7fe3c415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 -- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c40cc150 msgr2=0x7fe3c415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c40cc150 0x7fe3c415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 -- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c41455b0 msgr2=0x7fe3c415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c41455b0 0x7fe3c415e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c3fff640 1 -- 172.21.15.70:0/2171596192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c413e1c0 con 0x7fe3c4072060 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c8885640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c41455b0 0x7fe3c415e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.966+0000 7fe3c37fe640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c40cc150 0x7fe3c415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c3fff640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 0x7fe3c4159530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00c480 tx=0x7fe3bc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc018070 con 0x7fe3c4072060 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3bc007500 con 0x7fe3c4072060 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c413e4a0 con 0x7fe3c4072060 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc004a30 con 0x7fe3c4072060 2024-04-22T01:15:09.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c413ea60 con 0x7fe3c4072060 2024-04-22T01:15:09.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c4068210 con 0x7fe3c4072060 2024-04-22T01:15:09.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe3bc01b070 con 0x7fe3c4072060 2024-04-22T01:15:09.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c0ff9640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3a40678c0 0x7fe3a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:09.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe3bc0852b0 con 0x7fe3c4072060 2024-04-22T01:15:09.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c37fe640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3a40678c0 0x7fe3a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:09.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.970+0000 7fe3c37fe640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3a40678c0 0x7fe3a4069d80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0ad000 tx=0x7fe3ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:09.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:09.974+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe3bc055b40 con 0x7fe3c4072060 2024-04-22T01:15:10.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.170+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3c415f510 con 0x7fe3c4072060 2024-04-22T01:15:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3c0ff9640 1 -- 172.21.15.70:0/2171596192 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fe3bc055440 con 0x7fe3c4072060 2024-04-22T01:15:10.172 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:10.172 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3a40678c0 msgr2=0x7fe3a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3a40678c0 0x7fe3a4069d80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0ad000 tx=0x7fe3ac008040 comp rx=0 tx=0).stop 2024-04-22T01:15:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 msgr2=0x7fe3c4159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 0x7fe3c4159530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00c480 tx=0x7fe3bc00c940 comp rx=0 tx=0).stop 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 shutdown_connections 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3a40678c0 0x7fe3a4069d80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c41455b0 0x7fe3c415e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c40cc150 0x7fe3c415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 --2- 172.21.15.70:0/2171596192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4072060 0x7fe3c4159530 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 >> 172.21.15.70:0/2171596192 conn(0x7fe3c406b1f0 msgr2=0x7fe3c4142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 shutdown_connections 2024-04-22T01:15:10.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.174+0000 7fe3ca2e8640 1 -- 172.21.15.70:0/2171596192 wait complete. 2024-04-22T01:15:10.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:10.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 Processor -- start 2024-04-22T01:15:10.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 -- start start 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f862415aa30 0x7f86240b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86240aee40 0x7f86240b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862406dbd0 con 0x7f862415bb50 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f862406de70 con 0x7f862415aa30 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862baf1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f862406e150 con 0x7f86240aee40 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f862415aa30 0x7f86240b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f862415aa30 0x7f86240b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54494/0 (socket says 172.21.15.70:54494) 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 -- 172.21.15.70:0/3908691175 learned_addr learned my addr 172.21.15.70:0/3908691175 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:10.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862a08e640 1 --2- 172.21.15.70:0/3908691175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86240aee40 0x7f86240b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 -- 172.21.15.70:0/3908691175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86240aee40 msgr2=0x7f86240b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:10.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 --2- 172.21.15.70:0/3908691175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86240aee40 0x7f86240b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 -- 172.21.15.70:0/3908691175 >> v1:172.21.15.98:6789/0 conn(0x7f862415bb50 legacy=0x7f86240ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.310+0000 7f862988d640 1 -- 172.21.15.70:0/3908691175 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862406e430 con 0x7f862415aa30 2024-04-22T01:15:10.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 --2- 172.21.15.70:0/3908691175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f862415aa30 0x7f86240b4720 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f862000bb10 tx=0x7f862001ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d3532048a01052 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:10.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862888b640 1 -- 172.21.15.70:0/3908691175 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200b4070 con 0x7f862415aa30 2024-04-22T01:15:10.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862888b640 1 -- 172.21.15.70:0/3908691175 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86200a7cf0 con 0x7f862415aa30 2024-04-22T01:15:10.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862888b640 1 -- 172.21.15.70:0/3908691175 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200af660 con 0x7f862415aa30 2024-04-22T01:15:10.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/3908691175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f862415aa30 msgr2=0x7f86240b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 --2- 172.21.15.70:0/3908691175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f862415aa30 0x7f86240b4720 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f862000bb10 tx=0x7f862001ec00 comp rx=0 tx=0).stop 2024-04-22T01:15:10.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/3908691175 shutdown_connections 2024-04-22T01:15:10.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 --2- 172.21.15.70:0/3908691175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86240aee40 0x7f86240b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 --2- 172.21.15.70:0/3908691175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f862415aa30 0x7f86240b4720 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/3908691175 >> 172.21.15.70:0/3908691175 conn(0x7f862406b1f0 msgr2=0x7f862406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/3908691175 shutdown_connections 2024-04-22T01:15:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/3908691175 wait complete. 2024-04-22T01:15:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 Processor -- start 2024-04-22T01:15:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- start start 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 0x7f8624146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f862415aa30 0x7f86240af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862415bb50 0x7f86240b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f862406ead0 con 0x7f86240aee40 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f862406e950 con 0x7f862415bb50 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8624071b20 con 0x7f862415aa30 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 0x7f8624146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 0x7f8624146960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54510/0 (socket says 172.21.15.70:54510) 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 -- 172.21.15.70:0/719620615 learned_addr learned my addr 172.21.15.70:0/719620615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862a08e640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862415bb50 0x7f86240b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862908c640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f862415aa30 0x7f86240af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 -- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f862415aa30 msgr2=0x7f86240af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f862415aa30 0x7f86240af560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 -- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862415bb50 msgr2=0x7f86240b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862415bb50 0x7f86240b1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 -- 172.21.15.70:0/719620615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862414af60 con 0x7f86240aee40 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862988d640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 0x7f8624146960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f862001d2b0 tx=0x7f862001b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200b4040 con 0x7f86240aee40 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86200bf070 con 0x7f86240aee40 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862414b240 con 0x7f86240aee40 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200ba680 con 0x7f86240aee40 2024-04-22T01:15:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.314+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86241507e0 con 0x7f86240aee40 2024-04-22T01:15:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.318+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8624068210 con 0x7f86240aee40 2024-04-22T01:15:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.318+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f86200ba820 con 0x7f86240aee40 2024-04-22T01:15:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.318+0000 7f861a7fc640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86080678c0 0x7f8608069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.318+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8620123530 con 0x7f86240aee40 2024-04-22T01:15:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.318+0000 7f862908c640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86080678c0 0x7f8608069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.318+0000 7f862908c640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86080678c0 0x7f8608069d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8614007fd0 tx=0x7f8614009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:10.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.322+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8620018820 con 0x7f86240aee40 2024-04-22T01:15:10.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.486+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f862406f190 con 0x7f86240aee40 2024-04-22T01:15:10.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.486+0000 7f861a7fc640 1 -- 172.21.15.70:0/719620615 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f86200f3ee0 con 0x7f86240aee40 2024-04-22T01:15:10.488 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:10.488 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:10.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86080678c0 msgr2=0x7f8608069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86080678c0 0x7f8608069d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8614007fd0 tx=0x7f8614009040 comp rx=0 tx=0).stop 2024-04-22T01:15:10.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 msgr2=0x7f8624146960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 0x7f8624146960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f862001d2b0 tx=0x7f862001b9a0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 shutdown_connections 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86080678c0 0x7f8608069d80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862415bb50 0x7f86240b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f862415aa30 0x7f86240af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 --2- 172.21.15.70:0/719620615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86240aee40 0x7f8624146960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 >> 172.21.15.70:0/719620615 conn(0x7f862406b1f0 msgr2=0x7f862415c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 shutdown_connections 2024-04-22T01:15:10.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.490+0000 7f862baf1640 1 -- 172.21.15.70:0/719620615 wait complete. 2024-04-22T01:15:10.507 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:15:10.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:15:10.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.626+0000 7f0f9eb83640 1 Processor -- start 2024-04-22T01:15:10.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.626+0000 7f0f9eb83640 1 -- start start 2024-04-22T01:15:10.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9eb83640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 0x7f0f980cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9eb83640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f980d0250 con 0x7f0f98072230 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9eb83640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f980cd7b0 con 0x7f0f980cc150 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9eb83640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0f980cc710 con 0x7f0f981455b0 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0f98072230 0x7f0f9806e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40704/0 (socket says 172.21.15.70:40704) 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 -- 172.21.15.70:0/1835380323 learned_addr learned my addr 172.21.15.70:0/1835380323 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9c91f640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 0x7f0f980cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059013448 0 0) 0x7f0f980d0250 con 0x7f0f98072230 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f80003530 con 0x7f0f98072230 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2445344912 0 0) 0x7f0f80003530 con 0x7f0f98072230 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 msgr2=0x7f0f980cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 0x7f0f980cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 >> v1:172.21.15.98:6789/0 conn(0x7f0f980cc150 legacy=0x7f0f98071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f980cc9f0 con 0x7f0f98072230 2024-04-22T01:15:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f0f90001a00 con 0x7f0f98072230 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 >> v1:172.21.15.70:6789/0 conn(0x7f0f98072230 legacy=0x7f0f9806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f800039f0 0x7f0f80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 0x7f0f80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80004d50 0x7f0f800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f980cc710 con 0x7f0f80004d50 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8000bb60 con 0x7f0f800039f0 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8000bd30 con 0x7f0f981455b0 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80004d50 0x7f0f800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9c91f640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f800039f0 0x7f0f80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f8ffff640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 0x7f0f80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 -- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 msgr2=0x7f0f80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f981455b0 0x7f0f80004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 -- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f800039f0 msgr2=0x7f0f80003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f800039f0 0x7f0f80003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9d120640 1 -- 172.21.15.70:0/1835380323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f80003530 con 0x7f0f80004d50 2024-04-22T01:15:10.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.630+0000 7f0f9c91f640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f800039f0 0x7f0f80003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9d120640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80004d50 0x7f0f800051d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000a8f0 tx=0x7f0f9001d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb70dfc8d14f07b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000a4c0 con 0x7f0f80004d50 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f900ad650 con 0x7f0f80004d50 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f8f7fe640 1 -- 172.21.15.70:0/1835380323 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f900ada40 con 0x7f0f80004d50 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- 172.21.15.70:0/1835380323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80004d50 msgr2=0x7f0f800051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/1835380323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f80004d50 0x7f0f800051d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000a8f0 tx=0x7f0f9001d3c0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9d120640 1 -- 172.21.15.70:0/1835380323 reap_dead start 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- 172.21.15.70:0/1835380323 shutdown_connections 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- 172.21.15.70:0/1835380323 >> 172.21.15.70:0/1835380323 conn(0x7f0f9806b1f0 msgr2=0x7f0f9806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- 172.21.15.70:0/1835380323 shutdown_connections 2024-04-22T01:15:10.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- 172.21.15.70:0/1835380323 wait complete. 2024-04-22T01:15:10.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 Processor -- start 2024-04-22T01:15:10.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- start start 2024-04-22T01:15:10.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 0x7f0f9813ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f9813e700 0x7f0f980b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f980af460 0x7f0f980af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f980cfd10 con 0x7f0f981502d0 2024-04-22T01:15:10.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f980ccfb0 con 0x7f0f9813e700 2024-04-22T01:15:10.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9eb83640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0f980cfe90 con 0x7f0f980af460 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9c91f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 0x7f0f9813ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9c91f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 0x7f0f9813ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54532/0 (socket says 172.21.15.70:54532) 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f9c91f640 1 -- 172.21.15.70:0/2242022737 learned_addr learned my addr 172.21.15.70:0/2242022737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.634+0000 7f0f8ffff640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f9813e700 0x7f0f980b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9d120640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f980af460 0x7f0f980af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9c91f640 1 -- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f980af460 msgr2=0x7f0f980af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9c91f640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f980af460 0x7f0f980af910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9c91f640 1 -- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f9813e700 msgr2=0x7f0f980b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9c91f640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f9813e700 0x7f0f980b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9c91f640 1 -- 172.21.15.70:0/2242022737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f98140560 con 0x7f0f981502d0 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9d120640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f980af460 0x7f0f980af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9c91f640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 0x7f0f9813ff80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009fd0 tx=0x7f0f94011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400f960 con 0x7f0f981502d0 2024-04-22T01:15:10.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f9400fb00 con 0x7f0f981502d0 2024-04-22T01:15:10.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f98140840 con 0x7f0f981502d0 2024-04-22T01:15:10.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94015460 con 0x7f0f981502d0 2024-04-22T01:15:10.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f98140e00 con 0x7f0f981502d0 2024-04-22T01:15:10.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f98068210 con 0x7f0f981502d0 2024-04-22T01:15:10.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0f94015600 con 0x7f0f981502d0 2024-04-22T01:15:10.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8d7fa640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f840678f0 0x7f0f84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:10.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0f94086900 con 0x7f0f981502d0 2024-04-22T01:15:10.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8ffff640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f840678f0 0x7f0f84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:10.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.638+0000 7f0f8ffff640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f840678f0 0x7f0f84069db0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000f70 tx=0x7f0f88000ee0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:10.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.642+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0f94019030 con 0x7f0f981502d0 2024-04-22T01:15:10.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.802+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f980ac8d0 con 0x7f0f840678f0 2024-04-22T01:15:10.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.818+0000 7f0f8d7fa640 1 -- 172.21.15.70:0/2242022737 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f0f980ac8d0 con 0x7f0f840678f0 2024-04-22T01:15:10.821 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:10.824 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f840678f0 msgr2=0x7f0f84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f840678f0 0x7f0f84069db0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000f70 tx=0x7f0f88000ee0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 msgr2=0x7f0f9813ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 0x7f0f9813ff80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009fd0 tx=0x7f0f94011b30 comp rx=0 tx=0).stop 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 shutdown_connections 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0f840678f0 0x7f0f84069db0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f980af460 0x7f0f980af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f9813e700 0x7f0f980b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 --2- 172.21.15.70:0/2242022737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f981502d0 0x7f0f9813ff80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 >> 172.21.15.70:0/2242022737 conn(0x7f0f9806b1f0 msgr2=0x7f0f9806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:10.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 shutdown_connections 2024-04-22T01:15:10.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:10.822+0000 7f0f9eb83640 1 -- 172.21.15.70:0/2242022737 wait complete. 2024-04-22T01:15:10.841 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-22T01:15:10.593713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236020,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738700,"statfs":{"total":773094113280,"available":772852428800,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004689"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805910+0000","last_change":"2024-04-22T01:14:45.690356+0000","last_active":"2024-04-22T01:14:52.805910+0000","last_peered":"2024-04-22T01:14:52.805910+0000","last_clean":"2024-04-22T01:14:52.805910+0000","last_became_active":"2024-04-22T01:14:45.690080+0000","last_became_peered":"2024-04-22T01:14:45.690080+0000","last_unstale":"2024-04-22T01:14:52.805910+0000","last_undegraded":"2024-04-22T01:14:52.805910+0000","last_fullsized":"2024-04-22T01:14:52.805910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917875+0000","last_change":"2024-04-22T01:14:46.720257+0000","last_active":"2024-04-22T01:14:53.917875+0000","last_peered":"2024-04-22T01:14:53.917875+0000","last_clean":"2024-04-22T01:14:53.917875+0000","last_became_active":"2024-04-22T01:14:46.720047+0000","last_became_peered":"2024-04-22T01:14:46.720047+0000","last_unstale":"2024-04-22T01:14:53.917875+0000","last_undegraded":"2024-04-22T01:14:53.917875+0000","last_fullsized":"2024-04-22T01:14:53.917875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916284+0000","last_change":"2024-04-22T01:14:50.933593+0000","last_active":"2024-04-22T01:14:53.916284+0000","last_peered":"2024-04-22T01:14:53.916284+0000","last_clean":"2024-04-22T01:14:53.916284+0000","last_became_active":"2024-04-22T01:14:50.933273+0000","last_became_peered":"2024-04-22T01:14:50.933273+0000","last_unstale":"2024-04-22T01:14:53.916284+0000","last_undegraded":"2024-04-22T01:14:53.916284+0000","last_fullsized":"2024-04-22T01:14:53.916284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:46.804215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915513+0000","last_change":"2024-04-22T01:14:45.689486+0000","last_active":"2024-04-22T01:14:53.915513+0000","last_peered":"2024-04-22T01:14:53.915513+0000","last_clean":"2024-04-22T01:14:53.915513+0000","last_became_active":"2024-04-22T01:14:45.689150+0000","last_became_peered":"2024-04-22T01:14:45.689150+0000","last_unstale":"2024-04-22T01:14:53.915513+0000","last_undegraded":"2024-04-22T01:14:53.915513+0000","last_fullsized":"2024-04-22T01:14:53.915513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805716+0000","last_change":"2024-04-22T01:14:46.688092+0000","last_active":"2024-04-22T01:14:52.805716+0000","last_peered":"2024-04-22T01:14:52.805716+0000","last_clean":"2024-04-22T01:14:52.805716+0000","last_became_active":"2024-04-22T01:14:46.687812+0000","last_became_peered":"2024-04-22T01:14:46.687812+0000","last_unstale":"2024-04-22T01:14:52.805716+0000","last_undegraded":"2024-04-22T01:14:52.805716+0000","last_fullsized":"2024-04-22T01:14:52.805716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745650+0000","last_change":"2024-04-22T01:14:50.778199+0000","last_active":"2024-04-22T01:14:51.745650+0000","last_peered":"2024-04-22T01:14:51.745650+0000","last_clean":"2024-04-22T01:14:51.745650+0000","last_became_active":"2024-04-22T01:14:50.773891+0000","last_became_peered":"2024-04-22T01:14:50.773891+0000","last_unstale":"2024-04-22T01:14:51.745650+0000","last_undegraded":"2024-04-22T01:14:51.745650+0000","last_fullsized":"2024-04-22T01:14:51.745650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:50.558971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445520+0000","last_change":"2024-04-22T01:14:45.702628+0000","last_active":"2024-04-22T01:14:53.445520+0000","last_peered":"2024-04-22T01:14:53.445520+0000","last_clean":"2024-04-22T01:14:53.445520+0000","last_became_active":"2024-04-22T01:14:45.701650+0000","last_became_peered":"2024-04-22T01:14:45.701650+0000","last_unstale":"2024-04-22T01:14:53.445520+0000","last_undegraded":"2024-04-22T01:14:53.445520+0000","last_fullsized":"2024-04-22T01:14:53.445520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749682+0000","last_change":"2024-04-22T01:14:46.694700+0000","last_active":"2024-04-22T01:14:51.749682+0000","last_peered":"2024-04-22T01:14:51.749682+0000","last_clean":"2024-04-22T01:14:51.749682+0000","last_became_active":"2024-04-22T01:14:46.694050+0000","last_became_peered":"2024-04-22T01:14:46.694050+0000","last_unstale":"2024-04-22T01:14:51.749682+0000","last_undegraded":"2024-04-22T01:14:51.749682+0000","last_fullsized":"2024-04-22T01:14:51.749682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748089+0000","last_change":"2024-04-22T01:14:50.782409+0000","last_active":"2024-04-22T01:14:51.748089+0000","last_peered":"2024-04-22T01:14:51.748089+0000","last_clean":"2024-04-22T01:14:51.748089+0000","last_became_active":"2024-04-22T01:14:50.780084+0000","last_became_peered":"2024-04-22T01:14:50.780084+0000","last_unstale":"2024-04-22T01:14:51.748089+0000","last_undegraded":"2024-04-22T01:14:51.748089+0000","last_fullsized":"2024-04-22T01:14:51.748089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444880+0000","last_change":"2024-04-22T01:14:45.708166+0000","last_active":"2024-04-22T01:14:53.444880+0000","last_peered":"2024-04-22T01:14:53.444880+0000","last_clean":"2024-04-22T01:14:53.444880+0000","last_became_active":"2024-04-22T01:14:45.707823+0000","last_became_peered":"2024-04-22T01:14:45.707823+0000","last_unstale":"2024-04-22T01:14:53.444880+0000","last_undegraded":"2024-04-22T01:14:53.444880+0000","last_fullsized":"2024-04-22T01:14:53.444880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750004+0000","last_change":"2024-04-22T01:14:46.692519+0000","last_active":"2024-04-22T01:14:51.750004+0000","last_peered":"2024-04-22T01:14:51.750004+0000","last_clean":"2024-04-22T01:14:51.750004+0000","last_became_active":"2024-04-22T01:14:46.691673+0000","last_became_peered":"2024-04-22T01:14:46.691673+0000","last_unstale":"2024-04-22T01:14:51.750004+0000","last_undegraded":"2024-04-22T01:14:51.750004+0000","last_fullsized":"2024-04-22T01:14:51.750004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745176+0000","last_change":"2024-04-22T01:14:50.778402+0000","last_active":"2024-04-22T01:14:51.745176+0000","last_peered":"2024-04-22T01:14:51.745176+0000","last_clean":"2024-04-22T01:14:51.745176+0000","last_became_active":"2024-04-22T01:14:50.775401+0000","last_became_peered":"2024-04-22T01:14:50.775401+0000","last_unstale":"2024-04-22T01:14:51.745176+0000","last_undegraded":"2024-04-22T01:14:51.745176+0000","last_fullsized":"2024-04-22T01:14:51.745176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:05.161322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940679+0000","last_change":"2024-04-22T01:14:46.717563+0000","last_active":"2024-04-22T01:14:53.940679+0000","last_peered":"2024-04-22T01:14:53.940679+0000","last_clean":"2024-04-22T01:14:53.940679+0000","last_became_active":"2024-04-22T01:14:46.715882+0000","last_became_peered":"2024-04-22T01:14:46.715882+0000","last_unstale":"2024-04-22T01:14:53.940679+0000","last_undegraded":"2024-04-22T01:14:53.940679+0000","last_fullsized":"2024-04-22T01:14:53.940679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445124+0000","last_change":"2024-04-22T01:14:45.704690+0000","last_active":"2024-04-22T01:14:53.445124+0000","last_peered":"2024-04-22T01:14:53.445124+0000","last_clean":"2024-04-22T01:14:53.445124+0000","last_became_active":"2024-04-22T01:14:45.704066+0000","last_became_peered":"2024-04-22T01:14:45.704066+0000","last_unstale":"2024-04-22T01:14:53.445124+0000","last_undegraded":"2024-04-22T01:14:53.445124+0000","last_fullsized":"2024-04-22T01:14:53.445124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:15.442187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747783+0000","last_change":"2024-04-22T01:14:50.785203+0000","last_active":"2024-04-22T01:14:51.747783+0000","last_peered":"2024-04-22T01:14:51.747783+0000","last_clean":"2024-04-22T01:14:51.747783+0000","last_became_active":"2024-04-22T01:14:50.783868+0000","last_became_peered":"2024-04-22T01:14:50.783868+0000","last_unstale":"2024-04-22T01:14:51.747783+0000","last_undegraded":"2024-04-22T01:14:51.747783+0000","last_fullsized":"2024-04-22T01:14:51.747783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917056+0000","last_change":"2024-04-22T01:14:46.695499+0000","last_active":"2024-04-22T01:14:53.917056+0000","last_peered":"2024-04-22T01:14:53.917056+0000","last_clean":"2024-04-22T01:14:53.917056+0000","last_became_active":"2024-04-22T01:14:46.695144+0000","last_became_peered":"2024-04-22T01:14:46.695144+0000","last_unstale":"2024-04-22T01:14:53.917056+0000","last_undegraded":"2024-04-22T01:14:53.917056+0000","last_fullsized":"2024-04-22T01:14:53.917056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865258+0000","last_change":"2024-04-22T01:14:45.689968+0000","last_active":"2024-04-22T01:14:53.865258+0000","last_peered":"2024-04-22T01:14:53.865258+0000","last_clean":"2024-04-22T01:14:53.865258+0000","last_became_active":"2024-04-22T01:14:45.689627+0000","last_became_peered":"2024-04-22T01:14:45.689627+0000","last_unstale":"2024-04-22T01:14:53.865258+0000","last_undegraded":"2024-04-22T01:14:53.865258+0000","last_fullsized":"2024-04-22T01:14:53.865258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746331+0000","last_change":"2024-04-22T01:14:50.777923+0000","last_active":"2024-04-22T01:14:51.746331+0000","last_peered":"2024-04-22T01:14:51.746331+0000","last_clean":"2024-04-22T01:14:51.746331+0000","last_became_active":"2024-04-22T01:14:50.774329+0000","last_became_peered":"2024-04-22T01:14:50.774329+0000","last_unstale":"2024-04-22T01:14:51.746331+0000","last_undegraded":"2024-04-22T01:14:51.746331+0000","last_fullsized":"2024-04-22T01:14:51.746331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:10.629355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916652+0000","last_change":"2024-04-22T01:14:46.698998+0000","last_active":"2024-04-22T01:14:53.916652+0000","last_peered":"2024-04-22T01:14:53.916652+0000","last_clean":"2024-04-22T01:14:53.916652+0000","last_became_active":"2024-04-22T01:14:46.695438+0000","last_became_peered":"2024-04-22T01:14:46.695438+0000","last_unstale":"2024-04-22T01:14:53.916652+0000","last_undegraded":"2024-04-22T01:14:53.916652+0000","last_fullsized":"2024-04-22T01:14:53.916652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863231+0000","last_change":"2024-04-22T01:14:45.699027+0000","last_active":"2024-04-22T01:14:53.863231+0000","last_peered":"2024-04-22T01:14:53.863231+0000","last_clean":"2024-04-22T01:14:53.863231+0000","last_became_active":"2024-04-22T01:14:45.698655+0000","last_became_peered":"2024-04-22T01:14:45.698655+0000","last_unstale":"2024-04-22T01:14:53.863231+0000","last_undegraded":"2024-04-22T01:14:53.863231+0000","last_fullsized":"2024-04-22T01:14:53.863231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865839+0000","last_change":"2024-04-22T01:14:50.930489+0000","last_active":"2024-04-22T01:14:53.865839+0000","last_peered":"2024-04-22T01:14:53.865839+0000","last_clean":"2024-04-22T01:14:53.865839+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.865839+0000","last_undegraded":"2024-04-22T01:14:53.865839+0000","last_fullsized":"2024-04-22T01:14:53.865839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746136+0000","last_change":"2024-04-22T01:14:46.686147+0000","last_active":"2024-04-22T01:14:51.746136+0000","last_peered":"2024-04-22T01:14:51.746136+0000","last_clean":"2024-04-22T01:14:51.746136+0000","last_became_active":"2024-04-22T01:14:46.680316+0000","last_became_peered":"2024-04-22T01:14:46.680316+0000","last_unstale":"2024-04-22T01:14:51.746136+0000","last_undegraded":"2024-04-22T01:14:51.746136+0000","last_fullsized":"2024-04-22T01:14:51.746136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:48.053524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444098+0000","last_change":"2024-04-22T01:14:45.696794+0000","last_active":"2024-04-22T01:14:53.444098+0000","last_peered":"2024-04-22T01:14:53.444098+0000","last_clean":"2024-04-22T01:14:53.444098+0000","last_became_active":"2024-04-22T01:14:45.688246+0000","last_became_peered":"2024-04-22T01:14:45.688246+0000","last_unstale":"2024-04-22T01:14:53.444098+0000","last_undegraded":"2024-04-22T01:14:53.444098+0000","last_fullsized":"2024-04-22T01:14:53.444098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746641+0000","last_change":"2024-04-22T01:14:50.778437+0000","last_active":"2024-04-22T01:14:51.746641+0000","last_peered":"2024-04-22T01:14:51.746641+0000","last_clean":"2024-04-22T01:14:51.746641+0000","last_became_active":"2024-04-22T01:14:50.774011+0000","last_became_peered":"2024-04-22T01:14:50.774011+0000","last_unstale":"2024-04-22T01:14:51.746641+0000","last_undegraded":"2024-04-22T01:14:51.746641+0000","last_fullsized":"2024-04-22T01:14:51.746641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:35.973709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806149+0000","last_change":"2024-04-22T01:14:46.683735+0000","last_active":"2024-04-22T01:14:52.806149+0000","last_peered":"2024-04-22T01:14:52.806149+0000","last_clean":"2024-04-22T01:14:52.806149+0000","last_became_active":"2024-04-22T01:14:46.683395+0000","last_became_peered":"2024-04-22T01:14:46.683395+0000","last_unstale":"2024-04-22T01:14:52.806149+0000","last_undegraded":"2024-04-22T01:14:52.806149+0000","last_fullsized":"2024-04-22T01:14:52.806149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:32:35.242494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939744+0000","last_change":"2024-04-22T01:14:45.690873+0000","last_active":"2024-04-22T01:14:53.939744+0000","last_peered":"2024-04-22T01:14:53.939744+0000","last_clean":"2024-04-22T01:14:53.939744+0000","last_became_active":"2024-04-22T01:14:45.689980+0000","last_became_peered":"2024-04-22T01:14:45.689980+0000","last_unstale":"2024-04-22T01:14:53.939744+0000","last_undegraded":"2024-04-22T01:14:53.939744+0000","last_fullsized":"2024-04-22T01:14:53.939744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745320+0000","last_change":"2024-04-22T01:14:50.934553+0000","last_active":"2024-04-22T01:14:51.745320+0000","last_peered":"2024-04-22T01:14:51.745320+0000","last_clean":"2024-04-22T01:14:51.745320+0000","last_became_active":"2024-04-22T01:14:50.933460+0000","last_became_peered":"2024-04-22T01:14:50.933460+0000","last_unstale":"2024-04-22T01:14:51.745320+0000","last_undegraded":"2024-04-22T01:14:51.745320+0000","last_fullsized":"2024-04-22T01:14:51.745320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:14:00.624653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918403+0000","last_change":"2024-04-22T01:14:46.736500+0000","last_active":"2024-04-22T01:14:53.918403+0000","last_peered":"2024-04-22T01:14:53.918403+0000","last_clean":"2024-04-22T01:14:53.918403+0000","last_became_active":"2024-04-22T01:14:46.736199+0000","last_became_peered":"2024-04-22T01:14:46.736199+0000","last_unstale":"2024-04-22T01:14:53.918403+0000","last_undegraded":"2024-04-22T01:14:53.918403+0000","last_fullsized":"2024-04-22T01:14:53.918403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865748+0000","last_change":"2024-04-22T01:14:45.691876+0000","last_active":"2024-04-22T01:14:53.865748+0000","last_peered":"2024-04-22T01:14:53.865748+0000","last_clean":"2024-04-22T01:14:53.865748+0000","last_became_active":"2024-04-22T01:14:45.691301+0000","last_became_peered":"2024-04-22T01:14:45.691301+0000","last_unstale":"2024-04-22T01:14:53.865748+0000","last_undegraded":"2024-04-22T01:14:53.865748+0000","last_fullsized":"2024-04-22T01:14:53.865748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915922+0000","last_change":"2024-04-22T01:14:50.935981+0000","last_active":"2024-04-22T01:14:53.915922+0000","last_peered":"2024-04-22T01:14:53.915922+0000","last_clean":"2024-04-22T01:14:53.915922+0000","last_became_active":"2024-04-22T01:14:50.935496+0000","last_became_peered":"2024-04-22T01:14:50.935496+0000","last_unstale":"2024-04-22T01:14:53.915922+0000","last_undegraded":"2024-04-22T01:14:53.915922+0000","last_fullsized":"2024-04-22T01:14:53.915922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:24:46.014214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865369+0000","last_change":"2024-04-22T01:14:46.690549+0000","last_active":"2024-04-22T01:14:53.865369+0000","last_peered":"2024-04-22T01:14:53.865369+0000","last_clean":"2024-04-22T01:14:53.865369+0000","last_became_active":"2024-04-22T01:14:46.689835+0000","last_became_peered":"2024-04-22T01:14:46.689835+0000","last_unstale":"2024-04-22T01:14:53.865369+0000","last_undegraded":"2024-04-22T01:14:53.865369+0000","last_fullsized":"2024-04-22T01:14:53.865369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445817+0000","last_change":"2024-04-22T01:14:45.702632+0000","last_active":"2024-04-22T01:14:53.445817+0000","last_peered":"2024-04-22T01:14:53.445817+0000","last_clean":"2024-04-22T01:14:53.445817+0000","last_became_active":"2024-04-22T01:14:45.698621+0000","last_became_peered":"2024-04-22T01:14:45.698621+0000","last_unstale":"2024-04-22T01:14:53.445817+0000","last_undegraded":"2024-04-22T01:14:53.445817+0000","last_fullsized":"2024-04-22T01:14:53.445817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805742+0000","last_change":"2024-04-22T01:14:50.932335+0000","last_active":"2024-04-22T01:14:52.805742+0000","last_peered":"2024-04-22T01:14:52.805742+0000","last_clean":"2024-04-22T01:14:52.805742+0000","last_became_active":"2024-04-22T01:14:50.931895+0000","last_became_peered":"2024-04-22T01:14:50.931895+0000","last_unstale":"2024-04-22T01:14:52.805742+0000","last_undegraded":"2024-04-22T01:14:52.805742+0000","last_fullsized":"2024-04-22T01:14:52.805742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750227+0000","last_change":"2024-04-22T01:14:46.704411+0000","last_active":"2024-04-22T01:14:51.750227+0000","last_peered":"2024-04-22T01:14:51.750227+0000","last_clean":"2024-04-22T01:14:51.750227+0000","last_became_active":"2024-04-22T01:14:46.703696+0000","last_became_peered":"2024-04-22T01:14:46.703696+0000","last_unstale":"2024-04-22T01:14:51.750227+0000","last_undegraded":"2024-04-22T01:14:51.750227+0000","last_fullsized":"2024-04-22T01:14:51.750227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863380+0000","last_change":"2024-04-22T01:14:45.685209+0000","last_active":"2024-04-22T01:14:53.863380+0000","last_peered":"2024-04-22T01:14:53.863380+0000","last_clean":"2024-04-22T01:14:53.863380+0000","last_became_active":"2024-04-22T01:14:45.684801+0000","last_became_peered":"2024-04-22T01:14:45.684801+0000","last_unstale":"2024-04-22T01:14:53.863380+0000","last_undegraded":"2024-04-22T01:14:53.863380+0000","last_fullsized":"2024-04-22T01:14:53.863380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864672+0000","last_change":"2024-04-22T01:14:50.788944+0000","last_active":"2024-04-22T01:14:53.864672+0000","last_peered":"2024-04-22T01:14:53.864672+0000","last_clean":"2024-04-22T01:14:53.864672+0000","last_became_active":"2024-04-22T01:14:50.788405+0000","last_became_peered":"2024-04-22T01:14:50.788405+0000","last_unstale":"2024-04-22T01:14:53.864672+0000","last_undegraded":"2024-04-22T01:14:53.864672+0000","last_fullsized":"2024-04-22T01:14:53.864672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747528+0000","last_change":"2024-04-22T01:14:46.680069+0000","last_active":"2024-04-22T01:14:51.747528+0000","last_peered":"2024-04-22T01:14:51.747528+0000","last_clean":"2024-04-22T01:14:51.747528+0000","last_became_active":"2024-04-22T01:14:46.679311+0000","last_became_peered":"2024-04-22T01:14:46.679311+0000","last_unstale":"2024-04-22T01:14:51.747528+0000","last_undegraded":"2024-04-22T01:14:51.747528+0000","last_fullsized":"2024-04-22T01:14:51.747528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748679+0000","last_change":"2024-04-22T01:14:45.701963+0000","last_active":"2024-04-22T01:14:51.748679+0000","last_peered":"2024-04-22T01:14:51.748679+0000","last_clean":"2024-04-22T01:14:51.748679+0000","last_became_active":"2024-04-22T01:14:45.700246+0000","last_became_peered":"2024-04-22T01:14:45.700246+0000","last_unstale":"2024-04-22T01:14:51.748679+0000","last_undegraded":"2024-04-22T01:14:51.748679+0000","last_fullsized":"2024-04-22T01:14:51.748679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865444+0000","last_change":"2024-04-22T01:14:50.783908+0000","last_active":"2024-04-22T01:14:53.865444+0000","last_peered":"2024-04-22T01:14:53.865444+0000","last_clean":"2024-04-22T01:14:53.865444+0000","last_became_active":"2024-04-22T01:14:50.782537+0000","last_became_peered":"2024-04-22T01:14:50.782537+0000","last_unstale":"2024-04-22T01:14:53.865444+0000","last_undegraded":"2024-04-22T01:14:53.865444+0000","last_fullsized":"2024-04-22T01:14:53.865444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864361+0000","last_change":"2024-04-22T01:14:46.681168+0000","last_active":"2024-04-22T01:14:53.864361+0000","last_peered":"2024-04-22T01:14:53.864361+0000","last_clean":"2024-04-22T01:14:53.864361+0000","last_became_active":"2024-04-22T01:14:46.680888+0000","last_became_peered":"2024-04-22T01:14:46.680888+0000","last_unstale":"2024-04-22T01:14:53.864361+0000","last_undegraded":"2024-04-22T01:14:53.864361+0000","last_fullsized":"2024-04-22T01:14:53.864361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744510+0000","last_change":"2024-04-22T01:14:45.677217+0000","last_active":"2024-04-22T01:14:51.744510+0000","last_peered":"2024-04-22T01:14:51.744510+0000","last_clean":"2024-04-22T01:14:51.744510+0000","last_became_active":"2024-04-22T01:14:45.676630+0000","last_became_peered":"2024-04-22T01:14:45.676630+0000","last_unstale":"2024-04-22T01:14:51.744510+0000","last_undegraded":"2024-04-22T01:14:51.744510+0000","last_fullsized":"2024-04-22T01:14:51.744510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865873+0000","last_change":"2024-04-22T01:14:50.778827+0000","last_active":"2024-04-22T01:14:53.865873+0000","last_peered":"2024-04-22T01:14:53.865873+0000","last_clean":"2024-04-22T01:14:53.865873+0000","last_became_active":"2024-04-22T01:14:50.773642+0000","last_became_peered":"2024-04-22T01:14:50.773642+0000","last_unstale":"2024-04-22T01:14:53.865873+0000","last_undegraded":"2024-04-22T01:14:53.865873+0000","last_fullsized":"2024-04-22T01:14:53.865873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.804659+0000","last_change":"2024-04-22T01:14:46.688078+0000","last_active":"2024-04-22T01:14:52.804659+0000","last_peered":"2024-04-22T01:14:52.804659+0000","last_clean":"2024-04-22T01:14:52.804659+0000","last_became_active":"2024-04-22T01:14:46.687685+0000","last_became_peered":"2024-04-22T01:14:46.687685+0000","last_unstale":"2024-04-22T01:14:52.804659+0000","last_undegraded":"2024-04-22T01:14:52.804659+0000","last_fullsized":"2024-04-22T01:14:52.804659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444194+0000","last_change":"2024-04-22T01:14:45.687324+0000","last_active":"2024-04-22T01:14:53.444194+0000","last_peered":"2024-04-22T01:14:53.444194+0000","last_clean":"2024-04-22T01:14:53.444194+0000","last_became_active":"2024-04-22T01:14:45.686553+0000","last_became_peered":"2024-04-22T01:14:45.686553+0000","last_unstale":"2024-04-22T01:14:53.444194+0000","last_undegraded":"2024-04-22T01:14:53.444194+0000","last_fullsized":"2024-04-22T01:14:53.444194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806076+0000","last_change":"2024-04-22T01:14:50.931597+0000","last_active":"2024-04-22T01:14:52.806076+0000","last_peered":"2024-04-22T01:14:52.806076+0000","last_clean":"2024-04-22T01:14:52.806076+0000","last_became_active":"2024-04-22T01:14:50.931200+0000","last_became_peered":"2024-04-22T01:14:50.931200+0000","last_unstale":"2024-04-22T01:14:52.806076+0000","last_undegraded":"2024-04-22T01:14:52.806076+0000","last_fullsized":"2024-04-22T01:14:52.806076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916581+0000","last_change":"2024-04-22T01:14:46.694352+0000","last_active":"2024-04-22T01:14:53.916581+0000","last_peered":"2024-04-22T01:14:53.916581+0000","last_clean":"2024-04-22T01:14:53.916581+0000","last_became_active":"2024-04-22T01:14:46.693993+0000","last_became_peered":"2024-04-22T01:14:46.693993+0000","last_unstale":"2024-04-22T01:14:53.916581+0000","last_undegraded":"2024-04-22T01:14:53.916581+0000","last_fullsized":"2024-04-22T01:14:53.916581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749278+0000","last_change":"2024-04-22T01:14:45.692182+0000","last_active":"2024-04-22T01:14:51.749278+0000","last_peered":"2024-04-22T01:14:51.749278+0000","last_clean":"2024-04-22T01:14:51.749278+0000","last_became_active":"2024-04-22T01:14:45.670363+0000","last_became_peered":"2024-04-22T01:14:45.670363+0000","last_unstale":"2024-04-22T01:14:51.749278+0000","last_undegraded":"2024-04-22T01:14:51.749278+0000","last_fullsized":"2024-04-22T01:14:51.749278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444406+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:14:53.444406+0000","last_peered":"2024-04-22T01:14:53.444406+0000","last_clean":"2024-04-22T01:14:53.444406+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:14:53.444406+0000","last_undegraded":"2024-04-22T01:14:53.444406+0000","last_fullsized":"2024-04-22T01:14:53.444406+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941257+0000","last_change":"2024-04-22T01:14:50.933466+0000","last_active":"2024-04-22T01:14:53.941257+0000","last_peered":"2024-04-22T01:14:53.941257+0000","last_clean":"2024-04-22T01:14:53.941257+0000","last_became_active":"2024-04-22T01:14:50.932162+0000","last_became_peered":"2024-04-22T01:14:50.932162+0000","last_unstale":"2024-04-22T01:14:53.941257+0000","last_undegraded":"2024-04-22T01:14:53.941257+0000","last_fullsized":"2024-04-22T01:14:53.941257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445273+0000","last_change":"2024-04-22T01:14:46.726237+0000","last_active":"2024-04-22T01:14:53.445273+0000","last_peered":"2024-04-22T01:14:53.445273+0000","last_clean":"2024-04-22T01:14:53.445273+0000","last_became_active":"2024-04-22T01:14:46.725538+0000","last_became_peered":"2024-04-22T01:14:46.725538+0000","last_unstale":"2024-04-22T01:14:53.445273+0000","last_undegraded":"2024-04-22T01:14:53.445273+0000","last_fullsized":"2024-04-22T01:14:53.445273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744593+0000","last_change":"2024-04-22T01:14:45.695208+0000","last_active":"2024-04-22T01:14:51.744593+0000","last_peered":"2024-04-22T01:14:51.744593+0000","last_clean":"2024-04-22T01:14:51.744593+0000","last_became_active":"2024-04-22T01:14:45.694829+0000","last_became_peered":"2024-04-22T01:14:45.694829+0000","last_unstale":"2024-04-22T01:14:51.744593+0000","last_undegraded":"2024-04-22T01:14:51.744593+0000","last_fullsized":"2024-04-22T01:14:51.744593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940759+0000","last_change":"2024-04-22T01:14:50.925049+0000","last_active":"2024-04-22T01:14:53.940759+0000","last_peered":"2024-04-22T01:14:53.940759+0000","last_clean":"2024-04-22T01:14:53.940759+0000","last_became_active":"2024-04-22T01:14:50.923916+0000","last_became_peered":"2024-04-22T01:14:50.923916+0000","last_unstale":"2024-04-22T01:14:53.940759+0000","last_undegraded":"2024-04-22T01:14:53.940759+0000","last_fullsized":"2024-04-22T01:14:53.940759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749772+0000","last_change":"2024-04-22T01:14:46.705928+0000","last_active":"2024-04-22T01:14:51.749772+0000","last_peered":"2024-04-22T01:14:51.749772+0000","last_clean":"2024-04-22T01:14:51.749772+0000","last_became_active":"2024-04-22T01:14:46.703511+0000","last_became_peered":"2024-04-22T01:14:46.703511+0000","last_unstale":"2024-04-22T01:14:51.749772+0000","last_undegraded":"2024-04-22T01:14:51.749772+0000","last_fullsized":"2024-04-22T01:14:51.749772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863430+0000","last_change":"2024-04-22T01:14:45.690308+0000","last_active":"2024-04-22T01:14:53.863430+0000","last_peered":"2024-04-22T01:14:53.863430+0000","last_clean":"2024-04-22T01:14:53.863430+0000","last_became_active":"2024-04-22T01:14:45.689948+0000","last_became_peered":"2024-04-22T01:14:45.689948+0000","last_unstale":"2024-04-22T01:14:53.863430+0000","last_undegraded":"2024-04-22T01:14:53.863430+0000","last_fullsized":"2024-04-22T01:14:53.863430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805877+0000","last_change":"2024-04-22T01:14:50.789559+0000","last_active":"2024-04-22T01:14:52.805877+0000","last_peered":"2024-04-22T01:14:52.805877+0000","last_clean":"2024-04-22T01:14:52.805877+0000","last_became_active":"2024-04-22T01:14:50.789026+0000","last_became_peered":"2024-04-22T01:14:50.789026+0000","last_unstale":"2024-04-22T01:14:52.805877+0000","last_undegraded":"2024-04-22T01:14:52.805877+0000","last_fullsized":"2024-04-22T01:14:52.805877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.446055+0000","last_change":"2024-04-22T01:14:46.734875+0000","last_active":"2024-04-22T01:14:53.446055+0000","last_peered":"2024-04-22T01:14:53.446055+0000","last_clean":"2024-04-22T01:14:53.446055+0000","last_became_active":"2024-04-22T01:14:46.733035+0000","last_became_peered":"2024-04-22T01:14:46.733035+0000","last_unstale":"2024-04-22T01:14:53.446055+0000","last_undegraded":"2024-04-22T01:14:53.446055+0000","last_fullsized":"2024-04-22T01:14:53.446055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744490+0000","last_change":"2024-04-22T01:14:45.692886+0000","last_active":"2024-04-22T01:14:51.744490+0000","last_peered":"2024-04-22T01:14:51.744490+0000","last_clean":"2024-04-22T01:14:51.744490+0000","last_became_active":"2024-04-22T01:14:45.692578+0000","last_became_peered":"2024-04-22T01:14:45.692578+0000","last_unstale":"2024-04-22T01:14:51.744490+0000","last_undegraded":"2024-04-22T01:14:51.744490+0000","last_fullsized":"2024-04-22T01:14:51.744490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745112+0000","last_change":"2024-04-22T01:14:50.925456+0000","last_active":"2024-04-22T01:14:51.745112+0000","last_peered":"2024-04-22T01:14:51.745112+0000","last_clean":"2024-04-22T01:14:51.745112+0000","last_became_active":"2024-04-22T01:14:50.924327+0000","last_became_peered":"2024-04-22T01:14:50.924327+0000","last_unstale":"2024-04-22T01:14:51.745112+0000","last_undegraded":"2024-04-22T01:14:51.745112+0000","last_fullsized":"2024-04-22T01:14:51.745112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445557+0000","last_change":"2024-04-22T01:14:46.711269+0000","last_active":"2024-04-22T01:14:53.445557+0000","last_peered":"2024-04-22T01:14:53.445557+0000","last_clean":"2024-04-22T01:14:53.445557+0000","last_became_active":"2024-04-22T01:14:46.710502+0000","last_became_peered":"2024-04-22T01:14:46.710502+0000","last_unstale":"2024-04-22T01:14:53.445557+0000","last_undegraded":"2024-04-22T01:14:53.445557+0000","last_fullsized":"2024-04-22T01:14:53.445557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746774+0000","last_change":"2024-04-22T01:14:45.679532+0000","last_active":"2024-04-22T01:14:51.746774+0000","last_peered":"2024-04-22T01:14:51.746774+0000","last_clean":"2024-04-22T01:14:51.746774+0000","last_became_active":"2024-04-22T01:14:45.678724+0000","last_became_peered":"2024-04-22T01:14:45.678724+0000","last_unstale":"2024-04-22T01:14:51.746774+0000","last_undegraded":"2024-04-22T01:14:51.746774+0000","last_fullsized":"2024-04-22T01:14:51.746774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:09.261855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744892+0000","last_change":"2024-04-22T01:14:50.780337+0000","last_active":"2024-04-22T01:14:51.744892+0000","last_peered":"2024-04-22T01:14:51.744892+0000","last_clean":"2024-04-22T01:14:51.744892+0000","last_became_active":"2024-04-22T01:14:50.779283+0000","last_became_peered":"2024-04-22T01:14:50.779283+0000","last_unstale":"2024-04-22T01:14:51.744892+0000","last_undegraded":"2024-04-22T01:14:51.744892+0000","last_fullsized":"2024-04-22T01:14:51.744892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:34.466322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444639+0000","last_change":"2024-04-22T01:14:46.719676+0000","last_active":"2024-04-22T01:14:53.444639+0000","last_peered":"2024-04-22T01:14:53.444639+0000","last_clean":"2024-04-22T01:14:53.444639+0000","last_became_active":"2024-04-22T01:14:46.712569+0000","last_became_peered":"2024-04-22T01:14:46.712569+0000","last_unstale":"2024-04-22T01:14:53.444639+0000","last_undegraded":"2024-04-22T01:14:53.444639+0000","last_fullsized":"2024-04-22T01:14:53.444639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744016+0000","last_change":"2024-04-22T01:14:45.681046+0000","last_active":"2024-04-22T01:14:51.744016+0000","last_peered":"2024-04-22T01:14:51.744016+0000","last_clean":"2024-04-22T01:14:51.744016+0000","last_became_active":"2024-04-22T01:14:45.680666+0000","last_became_peered":"2024-04-22T01:14:45.680666+0000","last_unstale":"2024-04-22T01:14:51.744016+0000","last_undegraded":"2024-04-22T01:14:51.744016+0000","last_fullsized":"2024-04-22T01:14:51.744016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805846+0000","last_change":"2024-04-22T01:14:50.767393+0000","last_active":"2024-04-22T01:14:52.805846+0000","last_peered":"2024-04-22T01:14:52.805846+0000","last_clean":"2024-04-22T01:14:52.805846+0000","last_became_active":"2024-04-22T01:14:50.766951+0000","last_became_peered":"2024-04-22T01:14:50.766951+0000","last_unstale":"2024-04-22T01:14:52.805846+0000","last_undegraded":"2024-04-22T01:14:52.805846+0000","last_fullsized":"2024-04-22T01:14:52.805846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940415+0000","last_change":"2024-04-22T01:14:46.706644+0000","last_active":"2024-04-22T01:14:53.940415+0000","last_peered":"2024-04-22T01:14:53.940415+0000","last_clean":"2024-04-22T01:14:53.940415+0000","last_became_active":"2024-04-22T01:14:46.705447+0000","last_became_peered":"2024-04-22T01:14:46.705447+0000","last_unstale":"2024-04-22T01:14:53.940415+0000","last_undegraded":"2024-04-22T01:14:53.940415+0000","last_fullsized":"2024-04-22T01:14:53.940415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939587+0000","last_change":"2024-04-22T01:14:45.692258+0000","last_active":"2024-04-22T01:14:53.939587+0000","last_peered":"2024-04-22T01:14:53.939587+0000","last_clean":"2024-04-22T01:14:53.939587+0000","last_became_active":"2024-04-22T01:14:45.691375+0000","last_became_peered":"2024-04-22T01:14:45.691375+0000","last_unstale":"2024-04-22T01:14:53.939587+0000","last_undegraded":"2024-04-22T01:14:53.939587+0000","last_fullsized":"2024-04-22T01:14:53.939587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918239+0000","last_change":"2024-04-22T01:14:50.932861+0000","last_active":"2024-04-22T01:14:53.918239+0000","last_peered":"2024-04-22T01:14:53.918239+0000","last_clean":"2024-04-22T01:14:53.918239+0000","last_became_active":"2024-04-22T01:14:50.932500+0000","last_became_peered":"2024-04-22T01:14:50.932500+0000","last_unstale":"2024-04-22T01:14:53.918239+0000","last_undegraded":"2024-04-22T01:14:53.918239+0000","last_fullsized":"2024-04-22T01:14:53.918239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:13.452468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445087+0000","last_change":"2024-04-22T01:14:46.733784+0000","last_active":"2024-04-22T01:14:53.445087+0000","last_peered":"2024-04-22T01:14:53.445087+0000","last_clean":"2024-04-22T01:14:53.445087+0000","last_became_active":"2024-04-22T01:14:46.732753+0000","last_became_peered":"2024-04-22T01:14:46.732753+0000","last_unstale":"2024-04-22T01:14:53.445087+0000","last_undegraded":"2024-04-22T01:14:53.445087+0000","last_fullsized":"2024-04-22T01:14:53.445087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750181+0000","last_change":"2024-04-22T01:14:45.706819+0000","last_active":"2024-04-22T01:14:51.750181+0000","last_peered":"2024-04-22T01:14:51.750181+0000","last_clean":"2024-04-22T01:14:51.750181+0000","last_became_active":"2024-04-22T01:14:45.705704+0000","last_became_peered":"2024-04-22T01:14:45.705704+0000","last_unstale":"2024-04-22T01:14:51.750181+0000","last_undegraded":"2024-04-22T01:14:51.750181+0000","last_fullsized":"2024-04-22T01:14:51.750181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917603+0000","last_change":"2024-04-22T01:14:50.925247+0000","last_active":"2024-04-22T01:14:53.917603+0000","last_peered":"2024-04-22T01:14:53.917603+0000","last_clean":"2024-04-22T01:14:53.917603+0000","last_became_active":"2024-04-22T01:14:50.924739+0000","last_became_peered":"2024-04-22T01:14:50.924739+0000","last_unstale":"2024-04-22T01:14:53.917603+0000","last_undegraded":"2024-04-22T01:14:53.917603+0000","last_fullsized":"2024-04-22T01:14:53.917603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:36.206609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805145+0000","last_change":"2024-04-22T01:14:46.686331+0000","last_active":"2024-04-22T01:14:52.805145+0000","last_peered":"2024-04-22T01:14:52.805145+0000","last_clean":"2024-04-22T01:14:52.805145+0000","last_became_active":"2024-04-22T01:14:46.686010+0000","last_became_peered":"2024-04-22T01:14:46.686010+0000","last_unstale":"2024-04-22T01:14:52.805145+0000","last_undegraded":"2024-04-22T01:14:52.805145+0000","last_fullsized":"2024-04-22T01:14:52.805145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806145+0000","last_change":"2024-04-22T01:14:45.686652+0000","last_active":"2024-04-22T01:14:52.806145+0000","last_peered":"2024-04-22T01:14:52.806145+0000","last_clean":"2024-04-22T01:14:52.806145+0000","last_became_active":"2024-04-22T01:14:45.686365+0000","last_became_peered":"2024-04-22T01:14:45.686365+0000","last_unstale":"2024-04-22T01:14:52.806145+0000","last_undegraded":"2024-04-22T01:14:52.806145+0000","last_fullsized":"2024-04-22T01:14:52.806145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749848+0000","last_change":"2024-04-22T01:14:50.925756+0000","last_active":"2024-04-22T01:14:51.749848+0000","last_peered":"2024-04-22T01:14:51.749848+0000","last_clean":"2024-04-22T01:14:51.749848+0000","last_became_active":"2024-04-22T01:14:50.924521+0000","last_became_peered":"2024-04-22T01:14:50.924521+0000","last_unstale":"2024-04-22T01:14:51.749848+0000","last_undegraded":"2024-04-22T01:14:51.749848+0000","last_fullsized":"2024-04-22T01:14:51.749848+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444945+0000","last_change":"2024-04-22T01:14:46.717218+0000","last_active":"2024-04-22T01:14:53.444945+0000","last_peered":"2024-04-22T01:14:53.444945+0000","last_clean":"2024-04-22T01:14:53.444945+0000","last_became_active":"2024-04-22T01:14:46.716342+0000","last_became_peered":"2024-04-22T01:14:46.716342+0000","last_unstale":"2024-04-22T01:14:53.444945+0000","last_undegraded":"2024-04-22T01:14:53.444945+0000","last_fullsized":"2024-04-22T01:14:53.444945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:57.350819+0000","last_change":"2024-04-22T01:14:45.688781+0000","last_active":"2024-04-22T01:14:57.350819+0000","last_peered":"2024-04-22T01:14:57.350819+0000","last_clean":"2024-04-22T01:14:57.350819+0000","last_became_active":"2024-04-22T01:14:45.688153+0000","last_became_peered":"2024-04-22T01:14:45.688153+0000","last_unstale":"2024-04-22T01:14:57.350819+0000","last_undegraded":"2024-04-22T01:14:57.350819+0000","last_fullsized":"2024-04-22T01:14:57.350819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864909+0000","last_change":"2024-04-22T01:14:50.778877+0000","last_active":"2024-04-22T01:14:53.864909+0000","last_peered":"2024-04-22T01:14:53.864909+0000","last_clean":"2024-04-22T01:14:53.864909+0000","last_became_active":"2024-04-22T01:14:50.773623+0000","last_became_peered":"2024-04-22T01:14:50.773623+0000","last_unstale":"2024-04-22T01:14:53.864909+0000","last_undegraded":"2024-04-22T01:14:53.864909+0000","last_fullsized":"2024-04-22T01:14:53.864909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746481+0000","last_change":"2024-04-22T01:14:46.727618+0000","last_active":"2024-04-22T01:14:51.746481+0000","last_peered":"2024-04-22T01:14:51.746481+0000","last_clean":"2024-04-22T01:14:51.746481+0000","last_became_active":"2024-04-22T01:14:46.727055+0000","last_became_peered":"2024-04-22T01:14:46.727055+0000","last_unstale":"2024-04-22T01:14:51.746481+0000","last_undegraded":"2024-04-22T01:14:51.746481+0000","last_fullsized":"2024-04-22T01:14:51.746481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:15.551380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863451+0000","last_change":"2024-04-22T01:14:45.695088+0000","last_active":"2024-04-22T01:14:53.863451+0000","last_peered":"2024-04-22T01:14:53.863451+0000","last_clean":"2024-04-22T01:14:53.863451+0000","last_became_active":"2024-04-22T01:14:45.694765+0000","last_became_peered":"2024-04-22T01:14:45.694765+0000","last_unstale":"2024-04-22T01:14:53.863451+0000","last_undegraded":"2024-04-22T01:14:53.863451+0000","last_fullsized":"2024-04-22T01:14:53.863451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747096+0000","last_change":"2024-04-22T01:14:50.782280+0000","last_active":"2024-04-22T01:14:51.747096+0000","last_peered":"2024-04-22T01:14:51.747096+0000","last_clean":"2024-04-22T01:14:51.747096+0000","last_became_active":"2024-04-22T01:14:50.779944+0000","last_became_peered":"2024-04-22T01:14:50.779944+0000","last_unstale":"2024-04-22T01:14:51.747096+0000","last_undegraded":"2024-04-22T01:14:51.747096+0000","last_fullsized":"2024-04-22T01:14:51.747096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805792+0000","last_change":"2024-04-22T01:14:46.688527+0000","last_active":"2024-04-22T01:14:52.805792+0000","last_peered":"2024-04-22T01:14:52.805792+0000","last_clean":"2024-04-22T01:14:52.805792+0000","last_became_active":"2024-04-22T01:14:46.687989+0000","last_became_peered":"2024-04-22T01:14:46.687989+0000","last_unstale":"2024-04-22T01:14:52.805792+0000","last_undegraded":"2024-04-22T01:14:52.805792+0000","last_fullsized":"2024-04-22T01:14:52.805792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749455+0000","last_change":"2024-04-22T01:14:45.699988+0000","last_active":"2024-04-22T01:14:51.749455+0000","last_peered":"2024-04-22T01:14:51.749455+0000","last_clean":"2024-04-22T01:14:51.749455+0000","last_became_active":"2024-04-22T01:14:45.692830+0000","last_became_peered":"2024-04-22T01:14:45.692830+0000","last_unstale":"2024-04-22T01:14:51.749455+0000","last_undegraded":"2024-04-22T01:14:51.749455+0000","last_fullsized":"2024-04-22T01:14:51.749455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916418+0000","last_change":"2024-04-22T01:14:50.781549+0000","last_active":"2024-04-22T01:14:53.916418+0000","last_peered":"2024-04-22T01:14:53.916418+0000","last_clean":"2024-04-22T01:14:53.916418+0000","last_became_active":"2024-04-22T01:14:50.780774+0000","last_became_peered":"2024-04-22T01:14:50.780774+0000","last_unstale":"2024-04-22T01:14:53.916418+0000","last_undegraded":"2024-04-22T01:14:53.916418+0000","last_fullsized":"2024-04-22T01:14:53.916418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749318+0000","last_change":"2024-04-22T01:14:46.686118+0000","last_active":"2024-04-22T01:14:51.749318+0000","last_peered":"2024-04-22T01:14:51.749318+0000","last_clean":"2024-04-22T01:14:51.749318+0000","last_became_active":"2024-04-22T01:14:46.685329+0000","last_became_peered":"2024-04-22T01:14:46.685329+0000","last_unstale":"2024-04-22T01:14:51.749318+0000","last_undegraded":"2024-04-22T01:14:51.749318+0000","last_fullsized":"2024-04-22T01:14:51.749318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939780+0000","last_change":"2024-04-22T01:14:45.690257+0000","last_active":"2024-04-22T01:14:53.939780+0000","last_peered":"2024-04-22T01:14:53.939780+0000","last_clean":"2024-04-22T01:14:53.939780+0000","last_became_active":"2024-04-22T01:14:45.688794+0000","last_became_peered":"2024-04-22T01:14:45.688794+0000","last_unstale":"2024-04-22T01:14:53.939780+0000","last_undegraded":"2024-04-22T01:14:53.939780+0000","last_fullsized":"2024-04-22T01:14:53.939780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916549+0000","last_change":"2024-04-22T01:14:50.930535+0000","last_active":"2024-04-22T01:14:53.916549+0000","last_peered":"2024-04-22T01:14:53.916549+0000","last_clean":"2024-04-22T01:14:53.916549+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.916549+0000","last_undegraded":"2024-04-22T01:14:53.916549+0000","last_fullsized":"2024-04-22T01:14:53.916549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805736+0000","last_change":"2024-04-22T01:14:50.932710+0000","last_active":"2024-04-22T01:14:52.805736+0000","last_peered":"2024-04-22T01:14:52.805736+0000","last_clean":"2024-04-22T01:14:52.805736+0000","last_became_active":"2024-04-22T01:14:50.931957+0000","last_became_peered":"2024-04-22T01:14:50.931957+0000","last_unstale":"2024-04-22T01:14:52.805736+0000","last_undegraded":"2024-04-22T01:14:52.805736+0000","last_fullsized":"2024-04-22T01:14:52.805736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864231+0000","last_change":"2024-04-22T01:14:46.697192+0000","last_active":"2024-04-22T01:14:53.864231+0000","last_peered":"2024-04-22T01:14:53.864231+0000","last_clean":"2024-04-22T01:14:53.864231+0000","last_became_active":"2024-04-22T01:14:46.696894+0000","last_became_peered":"2024-04-22T01:14:46.696894+0000","last_unstale":"2024-04-22T01:14:53.864231+0000","last_undegraded":"2024-04-22T01:14:53.864231+0000","last_fullsized":"2024-04-22T01:14:53.864231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745866+0000","last_change":"2024-04-22T01:14:45.701127+0000","last_active":"2024-04-22T01:14:51.745866+0000","last_peered":"2024-04-22T01:14:51.745866+0000","last_clean":"2024-04-22T01:14:51.745866+0000","last_became_active":"2024-04-22T01:14:45.699950+0000","last_became_peered":"2024-04-22T01:14:45.699950+0000","last_unstale":"2024-04-22T01:14:51.745866+0000","last_undegraded":"2024-04-22T01:14:51.745866+0000","last_fullsized":"2024-04-22T01:14:51.745866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941786+0000","last_change":"2024-04-22T01:14:50.781112+0000","last_active":"2024-04-22T01:14:53.941786+0000","last_peered":"2024-04-22T01:14:53.941786+0000","last_clean":"2024-04-22T01:14:53.941786+0000","last_became_active":"2024-04-22T01:14:50.780333+0000","last_became_peered":"2024-04-22T01:14:50.780333+0000","last_unstale":"2024-04-22T01:14:53.941786+0000","last_undegraded":"2024-04-22T01:14:53.941786+0000","last_fullsized":"2024-04-22T01:14:53.941786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916408+0000","last_change":"2024-04-22T01:14:46.699240+0000","last_active":"2024-04-22T01:14:53.916408+0000","last_peered":"2024-04-22T01:14:53.916408+0000","last_clean":"2024-04-22T01:14:53.916408+0000","last_became_active":"2024-04-22T01:14:46.698319+0000","last_became_peered":"2024-04-22T01:14:46.698319+0000","last_unstale":"2024-04-22T01:14:53.916408+0000","last_undegraded":"2024-04-22T01:14:53.916408+0000","last_fullsized":"2024-04-22T01:14:53.916408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916909+0000","last_change":"2024-04-22T01:14:45.692605+0000","last_active":"2024-04-22T01:14:53.916909+0000","last_peered":"2024-04-22T01:14:53.916909+0000","last_clean":"2024-04-22T01:14:53.916909+0000","last_became_active":"2024-04-22T01:14:45.692201+0000","last_became_peered":"2024-04-22T01:14:45.692201+0000","last_unstale":"2024-04-22T01:14:53.916909+0000","last_undegraded":"2024-04-22T01:14:53.916909+0000","last_fullsized":"2024-04-22T01:14:53.916909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939598+0000","last_change":"2024-04-22T01:14:45.690896+0000","last_active":"2024-04-22T01:14:53.939598+0000","last_peered":"2024-04-22T01:14:53.939598+0000","last_clean":"2024-04-22T01:14:53.939598+0000","last_became_active":"2024-04-22T01:14:45.689871+0000","last_became_peered":"2024-04-22T01:14:45.689871+0000","last_unstale":"2024-04-22T01:14:53.939598+0000","last_undegraded":"2024-04-22T01:14:53.939598+0000","last_fullsized":"2024-04-22T01:14:53.939598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806029+0000","last_change":"2024-04-22T01:14:46.685575+0000","last_active":"2024-04-22T01:14:52.806029+0000","last_peered":"2024-04-22T01:14:52.806029+0000","last_clean":"2024-04-22T01:14:52.806029+0000","last_became_active":"2024-04-22T01:14:46.685301+0000","last_became_peered":"2024-04-22T01:14:46.685301+0000","last_unstale":"2024-04-22T01:14:52.806029+0000","last_undegraded":"2024-04-22T01:14:52.806029+0000","last_fullsized":"2024-04-22T01:14:52.806029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744699+0000","last_change":"2024-04-22T01:14:50.775646+0000","last_active":"2024-04-22T01:14:51.744699+0000","last_peered":"2024-04-22T01:14:51.744699+0000","last_clean":"2024-04-22T01:14:51.744699+0000","last_became_active":"2024-04-22T01:14:50.775153+0000","last_became_peered":"2024-04-22T01:14:50.775153+0000","last_unstale":"2024-04-22T01:14:51.744699+0000","last_undegraded":"2024-04-22T01:14:51.744699+0000","last_fullsized":"2024-04-22T01:14:51.744699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:46.711296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915399+0000","last_change":"2024-04-22T01:14:45.700287+0000","last_active":"2024-04-22T01:14:53.915399+0000","last_peered":"2024-04-22T01:14:53.915399+0000","last_clean":"2024-04-22T01:14:53.915399+0000","last_became_active":"2024-04-22T01:14:45.700063+0000","last_became_peered":"2024-04-22T01:14:45.700063+0000","last_unstale":"2024-04-22T01:14:53.915399+0000","last_undegraded":"2024-04-22T01:14:53.915399+0000","last_fullsized":"2024-04-22T01:14:53.915399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745633+0000","last_change":"2024-04-22T01:14:46.694633+0000","last_active":"2024-04-22T01:14:51.745633+0000","last_peered":"2024-04-22T01:14:51.745633+0000","last_clean":"2024-04-22T01:14:51.745633+0000","last_became_active":"2024-04-22T01:14:46.690904+0000","last_became_peered":"2024-04-22T01:14:46.690904+0000","last_unstale":"2024-04-22T01:14:51.745633+0000","last_undegraded":"2024-04-22T01:14:51.745633+0000","last_fullsized":"2024-04-22T01:14:51.745633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:35.052516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917858+0000","last_change":"2024-04-22T01:14:50.775487+0000","last_active":"2024-04-22T01:14:53.917858+0000","last_peered":"2024-04-22T01:14:53.917858+0000","last_clean":"2024-04-22T01:14:53.917858+0000","last_became_active":"2024-04-22T01:14:50.774770+0000","last_became_peered":"2024-04-22T01:14:50.774770+0000","last_unstale":"2024-04-22T01:14:53.917858+0000","last_undegraded":"2024-04-22T01:14:53.917858+0000","last_fullsized":"2024-04-22T01:14:53.917858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:59:18.378937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940165+0000","last_change":"2024-04-22T01:14:46.715466+0000","last_active":"2024-04-22T01:14:53.940165+0000","last_peered":"2024-04-22T01:14:53.940165+0000","last_clean":"2024-04-22T01:14:53.940165+0000","last_became_active":"2024-04-22T01:14:46.706194+0000","last_became_peered":"2024-04-22T01:14:46.706194+0000","last_unstale":"2024-04-22T01:14:53.940165+0000","last_undegraded":"2024-04-22T01:14:53.940165+0000","last_fullsized":"2024-04-22T01:14:53.940165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744256+0000","last_change":"2024-04-22T01:14:46.695116+0000","last_active":"2024-04-22T01:14:51.744256+0000","last_peered":"2024-04-22T01:14:51.744256+0000","last_clean":"2024-04-22T01:14:51.744256+0000","last_became_active":"2024-04-22T01:14:46.694817+0000","last_became_peered":"2024-04-22T01:14:46.694817+0000","last_unstale":"2024-04-22T01:14:51.744256+0000","last_undegraded":"2024-04-22T01:14:51.744256+0000","last_fullsized":"2024-04-22T01:14:51.744256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748407+0000","last_change":"2024-04-22T01:14:46.697812+0000","last_active":"2024-04-22T01:14:51.748407+0000","last_peered":"2024-04-22T01:14:51.748407+0000","last_clean":"2024-04-22T01:14:51.748407+0000","last_became_active":"2024-04-22T01:14:46.697056+0000","last_became_peered":"2024-04-22T01:14:46.697056+0000","last_unstale":"2024-04-22T01:14:51.748407+0000","last_undegraded":"2024-04-22T01:14:51.748407+0000","last_fullsized":"2024-04-22T01:14:51.748407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805883+0000","last_change":"2024-04-22T01:14:46.696293+0000","last_active":"2024-04-22T01:14:52.805883+0000","last_peered":"2024-04-22T01:14:52.805883+0000","last_clean":"2024-04-22T01:14:52.805883+0000","last_became_active":"2024-04-22T01:14:46.695950+0000","last_became_peered":"2024-04-22T01:14:46.695950+0000","last_unstale":"2024-04-22T01:14:52.805883+0000","last_undegraded":"2024-04-22T01:14:52.805883+0000","last_fullsized":"2024-04-22T01:14:52.805883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917964+0000","last_change":"2024-04-22T01:14:46.705052+0000","last_active":"2024-04-22T01:14:53.917964+0000","last_peered":"2024-04-22T01:14:53.917964+0000","last_clean":"2024-04-22T01:14:53.917964+0000","last_became_active":"2024-04-22T01:14:46.704239+0000","last_became_peered":"2024-04-22T01:14:46.704239+0000","last_unstale":"2024-04-22T01:14:53.917964+0000","last_undegraded":"2024-04-22T01:14:53.917964+0000","last_fullsized":"2024-04-22T01:14:53.917964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865529+0000","last_change":"2024-04-22T01:14:46.702587+0000","last_active":"2024-04-22T01:14:53.865529+0000","last_peered":"2024-04-22T01:14:53.865529+0000","last_clean":"2024-04-22T01:14:53.865529+0000","last_became_active":"2024-04-22T01:14:46.702257+0000","last_became_peered":"2024-04-22T01:14:46.702257+0000","last_unstale":"2024-04-22T01:14:53.865529+0000","last_undegraded":"2024-04-22T01:14:53.865529+0000","last_fullsized":"2024-04-22T01:14:53.865529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941652+0000","last_change":"2024-04-22T01:14:46.729745+0000","last_active":"2024-04-22T01:14:53.941652+0000","last_peered":"2024-04-22T01:14:53.941652+0000","last_clean":"2024-04-22T01:14:53.941652+0000","last_became_active":"2024-04-22T01:14:46.729165+0000","last_became_peered":"2024-04-22T01:14:46.729165+0000","last_unstale":"2024-04-22T01:14:53.941652+0000","last_undegraded":"2024-04-22T01:14:53.941652+0000","last_fullsized":"2024-04-22T01:14:53.941652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918240+0000","last_change":"2024-04-22T01:14:46.707058+0000","last_active":"2024-04-22T01:14:53.918240+0000","last_peered":"2024-04-22T01:14:53.918240+0000","last_clean":"2024-04-22T01:14:53.918240+0000","last_became_active":"2024-04-22T01:14:46.706772+0000","last_became_peered":"2024-04-22T01:14:46.706772+0000","last_unstale":"2024-04-22T01:14:53.918240+0000","last_undegraded":"2024-04-22T01:14:53.918240+0000","last_fullsized":"2024-04-22T01:14:53.918240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745635+0000","last_change":"2024-04-22T01:14:46.702309+0000","last_active":"2024-04-22T01:14:51.745635+0000","last_peered":"2024-04-22T01:14:51.745635+0000","last_clean":"2024-04-22T01:14:51.745635+0000","last_became_active":"2024-04-22T01:14:46.701437+0000","last_became_peered":"2024-04-22T01:14:46.701437+0000","last_unstale":"2024-04-22T01:14:51.745635+0000","last_undegraded":"2024-04-22T01:14:51.745635+0000","last_fullsized":"2024-04-22T01:14:51.745635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916675+0000","last_change":"2024-04-22T01:14:46.706939+0000","last_active":"2024-04-22T01:14:53.916675+0000","last_peered":"2024-04-22T01:14:53.916675+0000","last_clean":"2024-04-22T01:14:53.916675+0000","last_became_active":"2024-04-22T01:14:46.706649+0000","last_became_peered":"2024-04-22T01:14:46.706649+0000","last_unstale":"2024-04-22T01:14:53.916675+0000","last_undegraded":"2024-04-22T01:14:53.916675+0000","last_fullsized":"2024-04-22T01:14:53.916675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748360+0000","last_change":"2024-04-22T01:14:46.708136+0000","last_active":"2024-04-22T01:14:51.748360+0000","last_peered":"2024-04-22T01:14:51.748360+0000","last_clean":"2024-04-22T01:14:51.748360+0000","last_became_active":"2024-04-22T01:14:46.704481+0000","last_became_peered":"2024-04-22T01:14:46.704481+0000","last_unstale":"2024-04-22T01:14:51.748360+0000","last_undegraded":"2024-04-22T01:14:51.748360+0000","last_fullsized":"2024-04-22T01:14:51.748360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805288+0000","last_change":"2024-04-22T01:14:46.730861+0000","last_active":"2024-04-22T01:14:52.805288+0000","last_peered":"2024-04-22T01:14:52.805288+0000","last_clean":"2024-04-22T01:14:52.805288+0000","last_became_active":"2024-04-22T01:14:46.730520+0000","last_became_peered":"2024-04-22T01:14:46.730520+0000","last_unstale":"2024-04-22T01:14:52.805288+0000","last_undegraded":"2024-04-22T01:14:52.805288+0000","last_fullsized":"2024-04-22T01:14:52.805288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:42.634594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941258+0000","last_change":"2024-04-22T01:14:46.724974+0000","last_active":"2024-04-22T01:14:53.941258+0000","last_peered":"2024-04-22T01:14:53.941258+0000","last_clean":"2024-04-22T01:14:53.941258+0000","last_became_active":"2024-04-22T01:14:46.717427+0000","last_became_peered":"2024-04-22T01:14:46.717427+0000","last_unstale":"2024-04-22T01:14:53.941258+0000","last_undegraded":"2024-04-22T01:14:53.941258+0000","last_fullsized":"2024-04-22T01:14:53.941258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445528+0000","last_change":"2024-04-22T01:14:46.724990+0000","last_active":"2024-04-22T01:14:53.445528+0000","last_peered":"2024-04-22T01:14:53.445528+0000","last_clean":"2024-04-22T01:14:53.445528+0000","last_became_active":"2024-04-22T01:14:46.724307+0000","last_became_peered":"2024-04-22T01:14:46.724307+0000","last_unstale":"2024-04-22T01:14:53.445528+0000","last_undegraded":"2024-04-22T01:14:53.445528+0000","last_fullsized":"2024-04-22T01:14:53.445528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744480+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.744480+0000","last_peered":"2024-04-22T01:14:51.744480+0000","last_clean":"2024-04-22T01:14:51.744480+0000","last_became_active":"2024-04-22T01:14:46.719087+0000","last_became_peered":"2024-04-22T01:14:46.719087+0000","last_unstale":"2024-04-22T01:14:51.744480+0000","last_undegraded":"2024-04-22T01:14:51.744480+0000","last_fullsized":"2024-04-22T01:14:51.744480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749914+0000","last_change":"2024-04-22T01:14:46.726617+0000","last_active":"2024-04-22T01:14:51.749914+0000","last_peered":"2024-04-22T01:14:51.749914+0000","last_clean":"2024-04-22T01:14:51.749914+0000","last_became_active":"2024-04-22T01:14:46.725959+0000","last_became_peered":"2024-04-22T01:14:46.725959+0000","last_unstale":"2024-04-22T01:14:51.749914+0000","last_undegraded":"2024-04-22T01:14:51.749914+0000","last_fullsized":"2024-04-22T01:14:51.749914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745404+0000","last_change":"2024-04-22T01:14:46.678143+0000","last_active":"2024-04-22T01:14:51.745404+0000","last_peered":"2024-04-22T01:14:51.745404+0000","last_clean":"2024-04-22T01:14:51.745404+0000","last_became_active":"2024-04-22T01:14:46.677398+0000","last_became_peered":"2024-04-22T01:14:46.677398+0000","last_unstale":"2024-04-22T01:14:51.745404+0000","last_undegraded":"2024-04-22T01:14:51.745404+0000","last_fullsized":"2024-04-22T01:14:51.745404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864510+0000","last_change":"2024-04-22T01:14:46.697838+0000","last_active":"2024-04-22T01:14:53.864510+0000","last_peered":"2024-04-22T01:14:53.864510+0000","last_clean":"2024-04-22T01:14:53.864510+0000","last_became_active":"2024-04-22T01:14:46.697564+0000","last_became_peered":"2024-04-22T01:14:46.697564+0000","last_unstale":"2024-04-22T01:14:53.864510+0000","last_undegraded":"2024-04-22T01:14:53.864510+0000","last_fullsized":"2024-04-22T01:14:53.864510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805023+0000","last_change":"2024-04-22T01:14:46.679516+0000","last_active":"2024-04-22T01:14:52.805023+0000","last_peered":"2024-04-22T01:14:52.805023+0000","last_clean":"2024-04-22T01:14:52.805023+0000","last_became_active":"2024-04-22T01:14:46.679223+0000","last_became_peered":"2024-04-22T01:14:46.679223+0000","last_unstale":"2024-04-22T01:14:52.805023+0000","last_undegraded":"2024-04-22T01:14:52.805023+0000","last_fullsized":"2024-04-22T01:14:52.805023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916981+0000","last_change":"2024-04-22T01:14:46.709145+0000","last_active":"2024-04-22T01:14:53.916981+0000","last_peered":"2024-04-22T01:14:53.916981+0000","last_clean":"2024-04-22T01:14:53.916981+0000","last_became_active":"2024-04-22T01:14:46.708806+0000","last_became_peered":"2024-04-22T01:14:46.708806+0000","last_unstale":"2024-04-22T01:14:53.916981+0000","last_undegraded":"2024-04-22T01:14:53.916981+0000","last_fullsized":"2024-04-22T01:14:53.916981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746371+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.746371+0000","last_peered":"2024-04-22T01:14:51.746371+0000","last_clean":"2024-04-22T01:14:51.746371+0000","last_became_active":"2024-04-22T01:14:46.718623+0000","last_became_peered":"2024-04-22T01:14:46.718623+0000","last_unstale":"2024-04-22T01:14:51.746371+0000","last_undegraded":"2024-04-22T01:14:51.746371+0000","last_fullsized":"2024-04-22T01:14:51.746371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941253+0000","last_change":"2024-04-22T01:14:46.705553+0000","last_active":"2024-04-22T01:14:53.941253+0000","last_peered":"2024-04-22T01:14:53.941253+0000","last_clean":"2024-04-22T01:14:53.941253+0000","last_became_active":"2024-04-22T01:14:46.704621+0000","last_became_peered":"2024-04-22T01:14:46.704621+0000","last_unstale":"2024-04-22T01:14:53.941253+0000","last_undegraded":"2024-04-22T01:14:53.941253+0000","last_fullsized":"2024-04-22T01:14:53.941253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941576+0000","last_change":"2024-04-22T01:14:46.705656+0000","last_active":"2024-04-22T01:14:53.941576+0000","last_peered":"2024-04-22T01:14:53.941576+0000","last_clean":"2024-04-22T01:14:53.941576+0000","last_became_active":"2024-04-22T01:14:46.704857+0000","last_became_peered":"2024-04-22T01:14:46.704857+0000","last_unstale":"2024-04-22T01:14:53.941576+0000","last_undegraded":"2024-04-22T01:14:53.941576+0000","last_fullsized":"2024-04-22T01:14:53.941576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445983+0000","last_change":"2024-04-22T01:14:46.735811+0000","last_active":"2024-04-22T01:14:53.445983+0000","last_peered":"2024-04-22T01:14:53.445983+0000","last_clean":"2024-04-22T01:14:53.445983+0000","last_became_active":"2024-04-22T01:14:46.727833+0000","last_became_peered":"2024-04-22T01:14:46.727833+0000","last_unstale":"2024-04-22T01:14:53.445983+0000","last_undegraded":"2024-04-22T01:14:53.445983+0000","last_fullsized":"2024-04-22T01:14:53.445983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916197+0000","last_change":"2024-04-22T01:14:46.699497+0000","last_active":"2024-04-22T01:14:53.916197+0000","last_peered":"2024-04-22T01:14:53.916197+0000","last_clean":"2024-04-22T01:14:53.916197+0000","last_became_active":"2024-04-22T01:14:46.698632+0000","last_became_peered":"2024-04-22T01:14:46.698632+0000","last_unstale":"2024-04-22T01:14:53.916197+0000","last_undegraded":"2024-04-22T01:14:53.916197+0000","last_fullsized":"2024-04-22T01:14:53.916197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444562+0000","last_change":"2024-04-22T01:14:46.699917+0000","last_active":"2024-04-22T01:14:53.444562+0000","last_peered":"2024-04-22T01:14:53.444562+0000","last_clean":"2024-04-22T01:14:53.444562+0000","last_became_active":"2024-04-22T01:14:46.684087+0000","last_became_peered":"2024-04-22T01:14:46.684087+0000","last_unstale":"2024-04-22T01:14:53.444562+0000","last_undegraded":"2024-04-22T01:14:53.444562+0000","last_fullsized":"2024-04-22T01:14:53.444562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748148+0000","last_change":"2024-04-22T01:14:46.683759+0000","last_active":"2024-04-22T01:14:51.748148+0000","last_peered":"2024-04-22T01:14:51.748148+0000","last_clean":"2024-04-22T01:14:51.748148+0000","last_became_active":"2024-04-22T01:14:46.682687+0000","last_became_peered":"2024-04-22T01:14:46.682687+0000","last_unstale":"2024-04-22T01:14:51.748148+0000","last_undegraded":"2024-04-22T01:14:51.748148+0000","last_fullsized":"2024-04-22T01:14:51.748148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444815+0000","last_change":"2024-04-22T01:14:46.732970+0000","last_active":"2024-04-22T01:14:53.444815+0000","last_peered":"2024-04-22T01:14:53.444815+0000","last_clean":"2024-04-22T01:14:53.444815+0000","last_became_active":"2024-04-22T01:14:46.727597+0000","last_became_peered":"2024-04-22T01:14:46.727597+0000","last_unstale":"2024-04-22T01:14:53.444815+0000","last_undegraded":"2024-04-22T01:14:53.444815+0000","last_fullsized":"2024-04-22T01:14:53.444815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749390+0000","last_change":"2024-04-22T01:14:46.705305+0000","last_active":"2024-04-22T01:14:51.749390+0000","last_peered":"2024-04-22T01:14:51.749390+0000","last_clean":"2024-04-22T01:14:51.749390+0000","last_became_active":"2024-04-22T01:14:46.704267+0000","last_became_peered":"2024-04-22T01:14:46.704267+0000","last_unstale":"2024-04-22T01:14:51.749390+0000","last_undegraded":"2024-04-22T01:14:51.749390+0000","last_fullsized":"2024-04-22T01:14:51.749390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805327+0000","last_change":"2024-04-22T01:14:46.678217+0000","last_active":"2024-04-22T01:14:52.805327+0000","last_peered":"2024-04-22T01:14:52.805327+0000","last_clean":"2024-04-22T01:14:52.805327+0000","last_became_active":"2024-04-22T01:14:46.677881+0000","last_became_peered":"2024-04-22T01:14:46.677881+0000","last_unstale":"2024-04-22T01:14:52.805327+0000","last_undegraded":"2024-04-22T01:14:52.805327+0000","last_fullsized":"2024-04-22T01:14:52.805327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865661+0000","last_change":"2024-04-22T01:14:46.682640+0000","last_active":"2024-04-22T01:14:53.865661+0000","last_peered":"2024-04-22T01:14:53.865661+0000","last_clean":"2024-04-22T01:14:53.865661+0000","last_became_active":"2024-04-22T01:14:46.682389+0000","last_became_peered":"2024-04-22T01:14:46.682389+0000","last_unstale":"2024-04-22T01:14:53.865661+0000","last_undegraded":"2024-04-22T01:14:53.865661+0000","last_fullsized":"2024-04-22T01:14:53.865661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445838+0000","last_change":"2024-04-22T01:14:46.727060+0000","last_active":"2024-04-22T01:14:53.445838+0000","last_peered":"2024-04-22T01:14:53.445838+0000","last_clean":"2024-04-22T01:14:53.445838+0000","last_became_active":"2024-04-22T01:14:46.726485+0000","last_became_peered":"2024-04-22T01:14:46.726485+0000","last_unstale":"2024-04-22T01:14:53.445838+0000","last_undegraded":"2024-04-22T01:14:53.445838+0000","last_fullsized":"2024-04-22T01:14:53.445838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:15:10.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:15:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.002+0000 7f0c53734640 1 Processor -- start 2024-04-22T01:15:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.002+0000 7f0c53734640 1 -- start start 2024-04-22T01:15:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c53734640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c4c15a550 0x7f0c4c0aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c53734640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4c0cf4a0 con 0x7f0c4c0aeae0 2024-04-22T01:15:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c53734640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4c06f260 con 0x7f0c4c071b20 2024-04-22T01:15:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c53734640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0c4c06f010 con 0x7f0c4c15a550 2024-04-22T01:15:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c514d0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c4c15a550 0x7f0c4c0aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c50ccf640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0c4c0aeae0 0x7f0c4c0cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40718/0 (socket says 172.21.15.70:40718) 2024-04-22T01:15:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c50ccf640 1 -- 172.21.15.70:0/3815124050 learned_addr learned my addr 172.21.15.70:0/3815124050 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c514d0640 1 -- 172.21.15.70:0/3815124050 >> v1:172.21.15.98:6789/0 conn(0x7f0c4c071b20 legacy=0x7f0c4c0b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c514d0640 1 -- 172.21.15.70:0/3815124050 >> v1:172.21.15.70:6789/0 conn(0x7f0c4c0aeae0 legacy=0x7f0c4c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c514d0640 1 -- 172.21.15.70:0/3815124050 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4c06f540 con 0x7f0c4c15a550 2024-04-22T01:15:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c43fff640 1 -- 172.21.15.70:0/3815124050 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2244148329 0 0) 0x7f0c4c0cf4a0 con 0x7f0c4c0aeae0 2024-04-22T01:15:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c514d0640 1 --2- 172.21.15.70:0/3815124050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c0aef50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0c480097c0 tx=0x7f0c4801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728e0b40ccd3418a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c43fff640 1 -- 172.21.15.70:0/3815124050 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480b0070 con 0x7f0c4c15a550 2024-04-22T01:15:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c43fff640 1 -- 172.21.15.70:0/3815124050 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c480ab600 con 0x7f0c4c15a550 2024-04-22T01:15:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.006+0000 7f0c43fff640 1 -- 172.21.15.70:0/3815124050 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480ab9d0 con 0x7f0c4c15a550 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/3815124050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 msgr2=0x7f0c4c0aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 --2- 172.21.15.70:0/3815124050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c0aef50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0c480097c0 tx=0x7f0c4801c460 comp rx=0 tx=0).stop 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/3815124050 shutdown_connections 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 --2- 172.21.15.70:0/3815124050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c0aef50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/3815124050 >> 172.21.15.70:0/3815124050 conn(0x7f0c4c06b1f0 msgr2=0x7f0c4c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/3815124050 shutdown_connections 2024-04-22T01:15:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/3815124050 wait complete. 2024-04-22T01:15:11.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 Processor -- start 2024-04-22T01:15:11.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- start start 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c4c071b20 0x7f0c4c0aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4c0aeae0 0x7f0c4c0af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c14c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4c1680e0 con 0x7f0c4c15a550 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4c167f60 con 0x7f0c4c0aeae0 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4c168260 con 0x7f0c4c071b20 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c14c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c14c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54548/0 (socket says 172.21.15.70:54548) 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 -- 172.21.15.70:0/1556996092 learned_addr learned my addr 172.21.15.70:0/1556996092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c514d0640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c4c071b20 0x7f0c4c0aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c50ccf640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4c0aeae0 0x7f0c4c0af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 -- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c4c071b20 msgr2=0x7f0c4c0aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c4c071b20 0x7f0c4c0aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 -- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4c0aeae0 msgr2=0x7f0c4c0af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4c0aeae0 0x7f0c4c0af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 -- 172.21.15.70:0/1556996092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4c150a70 con 0x7f0c4c15a550 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c50ccf640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4c0aeae0 0x7f0c4c0af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c51cd1640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c14c950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400b970 tx=0x7f0c4400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44019070 con 0x7f0c4c15a550 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4400cb40 con 0x7f0c4c15a550 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44014440 con 0x7f0c4c15a550 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c4c14fcb0 con 0x7f0c4c15a550 2024-04-22T01:15:11.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.010+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c4c150270 con 0x7f0c4c15a550 2024-04-22T01:15:11.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.014+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c4c068210 con 0x7f0c4c15a550 2024-04-22T01:15:11.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.014+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0c4401c070 con 0x7f0c4c15a550 2024-04-22T01:15:11.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.014+0000 7f0c41ffb640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0c300678f0 0x7f0c30069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.014+0000 7f0c514d0640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0c300678f0 0x7f0c30069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.014+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0c44085c80 con 0x7f0c4c15a550 2024-04-22T01:15:11.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.014+0000 7f0c514d0640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0c300678f0 0x7f0c30069db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c480ad000 tx=0x7f0c48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.018+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0c4c150a70 con 0x7f0c4c15a550 2024-04-22T01:15:11.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.174+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c4c06b950 con 0x7f0c300678f0 2024-04-22T01:15:11.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.186+0000 7f0c41ffb640 1 -- 172.21.15.70:0/1556996092 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f0c4c06b950 con 0x7f0c300678f0 2024-04-22T01:15:11.188 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:11.191 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0c300678f0 msgr2=0x7f0c30069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0c300678f0 0x7f0c30069db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c480ad000 tx=0x7f0c48008040 comp rx=0 tx=0).stop 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 msgr2=0x7f0c4c14c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c14c950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400b970 tx=0x7f0c4400be30 comp rx=0 tx=0).stop 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 shutdown_connections 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0c300678f0 0x7f0c30069db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c4c15a550 0x7f0c4c14c950 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4c0aeae0 0x7f0c4c0af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 --2- 172.21.15.70:0/1556996092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c4c071b20 0x7f0c4c0aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 >> 172.21.15.70:0/1556996092 conn(0x7f0c4c06b1f0 msgr2=0x7f0c4c0b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 shutdown_connections 2024-04-22T01:15:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.190+0000 7f0c53734640 1 -- 172.21.15.70:0/1556996092 wait complete. 2024-04-22T01:15:11.208 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-22T01:15:10.593713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236020,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738700,"statfs":{"total":773094113280,"available":772852428800,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004689"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805910+0000","last_change":"2024-04-22T01:14:45.690356+0000","last_active":"2024-04-22T01:14:52.805910+0000","last_peered":"2024-04-22T01:14:52.805910+0000","last_clean":"2024-04-22T01:14:52.805910+0000","last_became_active":"2024-04-22T01:14:45.690080+0000","last_became_peered":"2024-04-22T01:14:45.690080+0000","last_unstale":"2024-04-22T01:14:52.805910+0000","last_undegraded":"2024-04-22T01:14:52.805910+0000","last_fullsized":"2024-04-22T01:14:52.805910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917875+0000","last_change":"2024-04-22T01:14:46.720257+0000","last_active":"2024-04-22T01:14:53.917875+0000","last_peered":"2024-04-22T01:14:53.917875+0000","last_clean":"2024-04-22T01:14:53.917875+0000","last_became_active":"2024-04-22T01:14:46.720047+0000","last_became_peered":"2024-04-22T01:14:46.720047+0000","last_unstale":"2024-04-22T01:14:53.917875+0000","last_undegraded":"2024-04-22T01:14:53.917875+0000","last_fullsized":"2024-04-22T01:14:53.917875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916284+0000","last_change":"2024-04-22T01:14:50.933593+0000","last_active":"2024-04-22T01:14:53.916284+0000","last_peered":"2024-04-22T01:14:53.916284+0000","last_clean":"2024-04-22T01:14:53.916284+0000","last_became_active":"2024-04-22T01:14:50.933273+0000","last_became_peered":"2024-04-22T01:14:50.933273+0000","last_unstale":"2024-04-22T01:14:53.916284+0000","last_undegraded":"2024-04-22T01:14:53.916284+0000","last_fullsized":"2024-04-22T01:14:53.916284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:46.804215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915513+0000","last_change":"2024-04-22T01:14:45.689486+0000","last_active":"2024-04-22T01:14:53.915513+0000","last_peered":"2024-04-22T01:14:53.915513+0000","last_clean":"2024-04-22T01:14:53.915513+0000","last_became_active":"2024-04-22T01:14:45.689150+0000","last_became_peered":"2024-04-22T01:14:45.689150+0000","last_unstale":"2024-04-22T01:14:53.915513+0000","last_undegraded":"2024-04-22T01:14:53.915513+0000","last_fullsized":"2024-04-22T01:14:53.915513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805716+0000","last_change":"2024-04-22T01:14:46.688092+0000","last_active":"2024-04-22T01:14:52.805716+0000","last_peered":"2024-04-22T01:14:52.805716+0000","last_clean":"2024-04-22T01:14:52.805716+0000","last_became_active":"2024-04-22T01:14:46.687812+0000","last_became_peered":"2024-04-22T01:14:46.687812+0000","last_unstale":"2024-04-22T01:14:52.805716+0000","last_undegraded":"2024-04-22T01:14:52.805716+0000","last_fullsized":"2024-04-22T01:14:52.805716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745650+0000","last_change":"2024-04-22T01:14:50.778199+0000","last_active":"2024-04-22T01:14:51.745650+0000","last_peered":"2024-04-22T01:14:51.745650+0000","last_clean":"2024-04-22T01:14:51.745650+0000","last_became_active":"2024-04-22T01:14:50.773891+0000","last_became_peered":"2024-04-22T01:14:50.773891+0000","last_unstale":"2024-04-22T01:14:51.745650+0000","last_undegraded":"2024-04-22T01:14:51.745650+0000","last_fullsized":"2024-04-22T01:14:51.745650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:50.558971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445520+0000","last_change":"2024-04-22T01:14:45.702628+0000","last_active":"2024-04-22T01:14:53.445520+0000","last_peered":"2024-04-22T01:14:53.445520+0000","last_clean":"2024-04-22T01:14:53.445520+0000","last_became_active":"2024-04-22T01:14:45.701650+0000","last_became_peered":"2024-04-22T01:14:45.701650+0000","last_unstale":"2024-04-22T01:14:53.445520+0000","last_undegraded":"2024-04-22T01:14:53.445520+0000","last_fullsized":"2024-04-22T01:14:53.445520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749682+0000","last_change":"2024-04-22T01:14:46.694700+0000","last_active":"2024-04-22T01:14:51.749682+0000","last_peered":"2024-04-22T01:14:51.749682+0000","last_clean":"2024-04-22T01:14:51.749682+0000","last_became_active":"2024-04-22T01:14:46.694050+0000","last_became_peered":"2024-04-22T01:14:46.694050+0000","last_unstale":"2024-04-22T01:14:51.749682+0000","last_undegraded":"2024-04-22T01:14:51.749682+0000","last_fullsized":"2024-04-22T01:14:51.749682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748089+0000","last_change":"2024-04-22T01:14:50.782409+0000","last_active":"2024-04-22T01:14:51.748089+0000","last_peered":"2024-04-22T01:14:51.748089+0000","last_clean":"2024-04-22T01:14:51.748089+0000","last_became_active":"2024-04-22T01:14:50.780084+0000","last_became_peered":"2024-04-22T01:14:50.780084+0000","last_unstale":"2024-04-22T01:14:51.748089+0000","last_undegraded":"2024-04-22T01:14:51.748089+0000","last_fullsized":"2024-04-22T01:14:51.748089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444880+0000","last_change":"2024-04-22T01:14:45.708166+0000","last_active":"2024-04-22T01:14:53.444880+0000","last_peered":"2024-04-22T01:14:53.444880+0000","last_clean":"2024-04-22T01:14:53.444880+0000","last_became_active":"2024-04-22T01:14:45.707823+0000","last_became_peered":"2024-04-22T01:14:45.707823+0000","last_unstale":"2024-04-22T01:14:53.444880+0000","last_undegraded":"2024-04-22T01:14:53.444880+0000","last_fullsized":"2024-04-22T01:14:53.444880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750004+0000","last_change":"2024-04-22T01:14:46.692519+0000","last_active":"2024-04-22T01:14:51.750004+0000","last_peered":"2024-04-22T01:14:51.750004+0000","last_clean":"2024-04-22T01:14:51.750004+0000","last_became_active":"2024-04-22T01:14:46.691673+0000","last_became_peered":"2024-04-22T01:14:46.691673+0000","last_unstale":"2024-04-22T01:14:51.750004+0000","last_undegraded":"2024-04-22T01:14:51.750004+0000","last_fullsized":"2024-04-22T01:14:51.750004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745176+0000","last_change":"2024-04-22T01:14:50.778402+0000","last_active":"2024-04-22T01:14:51.745176+0000","last_peered":"2024-04-22T01:14:51.745176+0000","last_clean":"2024-04-22T01:14:51.745176+0000","last_became_active":"2024-04-22T01:14:50.775401+0000","last_became_peered":"2024-04-22T01:14:50.775401+0000","last_unstale":"2024-04-22T01:14:51.745176+0000","last_undegraded":"2024-04-22T01:14:51.745176+0000","last_fullsized":"2024-04-22T01:14:51.745176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:05.161322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940679+0000","last_change":"2024-04-22T01:14:46.717563+0000","last_active":"2024-04-22T01:14:53.940679+0000","last_peered":"2024-04-22T01:14:53.940679+0000","last_clean":"2024-04-22T01:14:53.940679+0000","last_became_active":"2024-04-22T01:14:46.715882+0000","last_became_peered":"2024-04-22T01:14:46.715882+0000","last_unstale":"2024-04-22T01:14:53.940679+0000","last_undegraded":"2024-04-22T01:14:53.940679+0000","last_fullsized":"2024-04-22T01:14:53.940679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445124+0000","last_change":"2024-04-22T01:14:45.704690+0000","last_active":"2024-04-22T01:14:53.445124+0000","last_peered":"2024-04-22T01:14:53.445124+0000","last_clean":"2024-04-22T01:14:53.445124+0000","last_became_active":"2024-04-22T01:14:45.704066+0000","last_became_peered":"2024-04-22T01:14:45.704066+0000","last_unstale":"2024-04-22T01:14:53.445124+0000","last_undegraded":"2024-04-22T01:14:53.445124+0000","last_fullsized":"2024-04-22T01:14:53.445124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:15.442187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747783+0000","last_change":"2024-04-22T01:14:50.785203+0000","last_active":"2024-04-22T01:14:51.747783+0000","last_peered":"2024-04-22T01:14:51.747783+0000","last_clean":"2024-04-22T01:14:51.747783+0000","last_became_active":"2024-04-22T01:14:50.783868+0000","last_became_peered":"2024-04-22T01:14:50.783868+0000","last_unstale":"2024-04-22T01:14:51.747783+0000","last_undegraded":"2024-04-22T01:14:51.747783+0000","last_fullsized":"2024-04-22T01:14:51.747783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917056+0000","last_change":"2024-04-22T01:14:46.695499+0000","last_active":"2024-04-22T01:14:53.917056+0000","last_peered":"2024-04-22T01:14:53.917056+0000","last_clean":"2024-04-22T01:14:53.917056+0000","last_became_active":"2024-04-22T01:14:46.695144+0000","last_became_peered":"2024-04-22T01:14:46.695144+0000","last_unstale":"2024-04-22T01:14:53.917056+0000","last_undegraded":"2024-04-22T01:14:53.917056+0000","last_fullsized":"2024-04-22T01:14:53.917056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865258+0000","last_change":"2024-04-22T01:14:45.689968+0000","last_active":"2024-04-22T01:14:53.865258+0000","last_peered":"2024-04-22T01:14:53.865258+0000","last_clean":"2024-04-22T01:14:53.865258+0000","last_became_active":"2024-04-22T01:14:45.689627+0000","last_became_peered":"2024-04-22T01:14:45.689627+0000","last_unstale":"2024-04-22T01:14:53.865258+0000","last_undegraded":"2024-04-22T01:14:53.865258+0000","last_fullsized":"2024-04-22T01:14:53.865258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746331+0000","last_change":"2024-04-22T01:14:50.777923+0000","last_active":"2024-04-22T01:14:51.746331+0000","last_peered":"2024-04-22T01:14:51.746331+0000","last_clean":"2024-04-22T01:14:51.746331+0000","last_became_active":"2024-04-22T01:14:50.774329+0000","last_became_peered":"2024-04-22T01:14:50.774329+0000","last_unstale":"2024-04-22T01:14:51.746331+0000","last_undegraded":"2024-04-22T01:14:51.746331+0000","last_fullsized":"2024-04-22T01:14:51.746331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:10.629355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916652+0000","last_change":"2024-04-22T01:14:46.698998+0000","last_active":"2024-04-22T01:14:53.916652+0000","last_peered":"2024-04-22T01:14:53.916652+0000","last_clean":"2024-04-22T01:14:53.916652+0000","last_became_active":"2024-04-22T01:14:46.695438+0000","last_became_peered":"2024-04-22T01:14:46.695438+0000","last_unstale":"2024-04-22T01:14:53.916652+0000","last_undegraded":"2024-04-22T01:14:53.916652+0000","last_fullsized":"2024-04-22T01:14:53.916652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863231+0000","last_change":"2024-04-22T01:14:45.699027+0000","last_active":"2024-04-22T01:14:53.863231+0000","last_peered":"2024-04-22T01:14:53.863231+0000","last_clean":"2024-04-22T01:14:53.863231+0000","last_became_active":"2024-04-22T01:14:45.698655+0000","last_became_peered":"2024-04-22T01:14:45.698655+0000","last_unstale":"2024-04-22T01:14:53.863231+0000","last_undegraded":"2024-04-22T01:14:53.863231+0000","last_fullsized":"2024-04-22T01:14:53.863231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865839+0000","last_change":"2024-04-22T01:14:50.930489+0000","last_active":"2024-04-22T01:14:53.865839+0000","last_peered":"2024-04-22T01:14:53.865839+0000","last_clean":"2024-04-22T01:14:53.865839+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.865839+0000","last_undegraded":"2024-04-22T01:14:53.865839+0000","last_fullsized":"2024-04-22T01:14:53.865839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746136+0000","last_change":"2024-04-22T01:14:46.686147+0000","last_active":"2024-04-22T01:14:51.746136+0000","last_peered":"2024-04-22T01:14:51.746136+0000","last_clean":"2024-04-22T01:14:51.746136+0000","last_became_active":"2024-04-22T01:14:46.680316+0000","last_became_peered":"2024-04-22T01:14:46.680316+0000","last_unstale":"2024-04-22T01:14:51.746136+0000","last_undegraded":"2024-04-22T01:14:51.746136+0000","last_fullsized":"2024-04-22T01:14:51.746136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:48.053524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444098+0000","last_change":"2024-04-22T01:14:45.696794+0000","last_active":"2024-04-22T01:14:53.444098+0000","last_peered":"2024-04-22T01:14:53.444098+0000","last_clean":"2024-04-22T01:14:53.444098+0000","last_became_active":"2024-04-22T01:14:45.688246+0000","last_became_peered":"2024-04-22T01:14:45.688246+0000","last_unstale":"2024-04-22T01:14:53.444098+0000","last_undegraded":"2024-04-22T01:14:53.444098+0000","last_fullsized":"2024-04-22T01:14:53.444098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746641+0000","last_change":"2024-04-22T01:14:50.778437+0000","last_active":"2024-04-22T01:14:51.746641+0000","last_peered":"2024-04-22T01:14:51.746641+0000","last_clean":"2024-04-22T01:14:51.746641+0000","last_became_active":"2024-04-22T01:14:50.774011+0000","last_became_peered":"2024-04-22T01:14:50.774011+0000","last_unstale":"2024-04-22T01:14:51.746641+0000","last_undegraded":"2024-04-22T01:14:51.746641+0000","last_fullsized":"2024-04-22T01:14:51.746641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:35.973709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806149+0000","last_change":"2024-04-22T01:14:46.683735+0000","last_active":"2024-04-22T01:14:52.806149+0000","last_peered":"2024-04-22T01:14:52.806149+0000","last_clean":"2024-04-22T01:14:52.806149+0000","last_became_active":"2024-04-22T01:14:46.683395+0000","last_became_peered":"2024-04-22T01:14:46.683395+0000","last_unstale":"2024-04-22T01:14:52.806149+0000","last_undegraded":"2024-04-22T01:14:52.806149+0000","last_fullsized":"2024-04-22T01:14:52.806149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:32:35.242494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939744+0000","last_change":"2024-04-22T01:14:45.690873+0000","last_active":"2024-04-22T01:14:53.939744+0000","last_peered":"2024-04-22T01:14:53.939744+0000","last_clean":"2024-04-22T01:14:53.939744+0000","last_became_active":"2024-04-22T01:14:45.689980+0000","last_became_peered":"2024-04-22T01:14:45.689980+0000","last_unstale":"2024-04-22T01:14:53.939744+0000","last_undegraded":"2024-04-22T01:14:53.939744+0000","last_fullsized":"2024-04-22T01:14:53.939744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745320+0000","last_change":"2024-04-22T01:14:50.934553+0000","last_active":"2024-04-22T01:14:51.745320+0000","last_peered":"2024-04-22T01:14:51.745320+0000","last_clean":"2024-04-22T01:14:51.745320+0000","last_became_active":"2024-04-22T01:14:50.933460+0000","last_became_peered":"2024-04-22T01:14:50.933460+0000","last_unstale":"2024-04-22T01:14:51.745320+0000","last_undegraded":"2024-04-22T01:14:51.745320+0000","last_fullsized":"2024-04-22T01:14:51.745320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:14:00.624653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918403+0000","last_change":"2024-04-22T01:14:46.736500+0000","last_active":"2024-04-22T01:14:53.918403+0000","last_peered":"2024-04-22T01:14:53.918403+0000","last_clean":"2024-04-22T01:14:53.918403+0000","last_became_active":"2024-04-22T01:14:46.736199+0000","last_became_peered":"2024-04-22T01:14:46.736199+0000","last_unstale":"2024-04-22T01:14:53.918403+0000","last_undegraded":"2024-04-22T01:14:53.918403+0000","last_fullsized":"2024-04-22T01:14:53.918403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865748+0000","last_change":"2024-04-22T01:14:45.691876+0000","last_active":"2024-04-22T01:14:53.865748+0000","last_peered":"2024-04-22T01:14:53.865748+0000","last_clean":"2024-04-22T01:14:53.865748+0000","last_became_active":"2024-04-22T01:14:45.691301+0000","last_became_peered":"2024-04-22T01:14:45.691301+0000","last_unstale":"2024-04-22T01:14:53.865748+0000","last_undegraded":"2024-04-22T01:14:53.865748+0000","last_fullsized":"2024-04-22T01:14:53.865748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915922+0000","last_change":"2024-04-22T01:14:50.935981+0000","last_active":"2024-04-22T01:14:53.915922+0000","last_peered":"2024-04-22T01:14:53.915922+0000","last_clean":"2024-04-22T01:14:53.915922+0000","last_became_active":"2024-04-22T01:14:50.935496+0000","last_became_peered":"2024-04-22T01:14:50.935496+0000","last_unstale":"2024-04-22T01:14:53.915922+0000","last_undegraded":"2024-04-22T01:14:53.915922+0000","last_fullsized":"2024-04-22T01:14:53.915922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:24:46.014214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865369+0000","last_change":"2024-04-22T01:14:46.690549+0000","last_active":"2024-04-22T01:14:53.865369+0000","last_peered":"2024-04-22T01:14:53.865369+0000","last_clean":"2024-04-22T01:14:53.865369+0000","last_became_active":"2024-04-22T01:14:46.689835+0000","last_became_peered":"2024-04-22T01:14:46.689835+0000","last_unstale":"2024-04-22T01:14:53.865369+0000","last_undegraded":"2024-04-22T01:14:53.865369+0000","last_fullsized":"2024-04-22T01:14:53.865369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445817+0000","last_change":"2024-04-22T01:14:45.702632+0000","last_active":"2024-04-22T01:14:53.445817+0000","last_peered":"2024-04-22T01:14:53.445817+0000","last_clean":"2024-04-22T01:14:53.445817+0000","last_became_active":"2024-04-22T01:14:45.698621+0000","last_became_peered":"2024-04-22T01:14:45.698621+0000","last_unstale":"2024-04-22T01:14:53.445817+0000","last_undegraded":"2024-04-22T01:14:53.445817+0000","last_fullsized":"2024-04-22T01:14:53.445817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805742+0000","last_change":"2024-04-22T01:14:50.932335+0000","last_active":"2024-04-22T01:14:52.805742+0000","last_peered":"2024-04-22T01:14:52.805742+0000","last_clean":"2024-04-22T01:14:52.805742+0000","last_became_active":"2024-04-22T01:14:50.931895+0000","last_became_peered":"2024-04-22T01:14:50.931895+0000","last_unstale":"2024-04-22T01:14:52.805742+0000","last_undegraded":"2024-04-22T01:14:52.805742+0000","last_fullsized":"2024-04-22T01:14:52.805742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750227+0000","last_change":"2024-04-22T01:14:46.704411+0000","last_active":"2024-04-22T01:14:51.750227+0000","last_peered":"2024-04-22T01:14:51.750227+0000","last_clean":"2024-04-22T01:14:51.750227+0000","last_became_active":"2024-04-22T01:14:46.703696+0000","last_became_peered":"2024-04-22T01:14:46.703696+0000","last_unstale":"2024-04-22T01:14:51.750227+0000","last_undegraded":"2024-04-22T01:14:51.750227+0000","last_fullsized":"2024-04-22T01:14:51.750227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863380+0000","last_change":"2024-04-22T01:14:45.685209+0000","last_active":"2024-04-22T01:14:53.863380+0000","last_peered":"2024-04-22T01:14:53.863380+0000","last_clean":"2024-04-22T01:14:53.863380+0000","last_became_active":"2024-04-22T01:14:45.684801+0000","last_became_peered":"2024-04-22T01:14:45.684801+0000","last_unstale":"2024-04-22T01:14:53.863380+0000","last_undegraded":"2024-04-22T01:14:53.863380+0000","last_fullsized":"2024-04-22T01:14:53.863380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864672+0000","last_change":"2024-04-22T01:14:50.788944+0000","last_active":"2024-04-22T01:14:53.864672+0000","last_peered":"2024-04-22T01:14:53.864672+0000","last_clean":"2024-04-22T01:14:53.864672+0000","last_became_active":"2024-04-22T01:14:50.788405+0000","last_became_peered":"2024-04-22T01:14:50.788405+0000","last_unstale":"2024-04-22T01:14:53.864672+0000","last_undegraded":"2024-04-22T01:14:53.864672+0000","last_fullsized":"2024-04-22T01:14:53.864672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747528+0000","last_change":"2024-04-22T01:14:46.680069+0000","last_active":"2024-04-22T01:14:51.747528+0000","last_peered":"2024-04-22T01:14:51.747528+0000","last_clean":"2024-04-22T01:14:51.747528+0000","last_became_active":"2024-04-22T01:14:46.679311+0000","last_became_peered":"2024-04-22T01:14:46.679311+0000","last_unstale":"2024-04-22T01:14:51.747528+0000","last_undegraded":"2024-04-22T01:14:51.747528+0000","last_fullsized":"2024-04-22T01:14:51.747528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748679+0000","last_change":"2024-04-22T01:14:45.701963+0000","last_active":"2024-04-22T01:14:51.748679+0000","last_peered":"2024-04-22T01:14:51.748679+0000","last_clean":"2024-04-22T01:14:51.748679+0000","last_became_active":"2024-04-22T01:14:45.700246+0000","last_became_peered":"2024-04-22T01:14:45.700246+0000","last_unstale":"2024-04-22T01:14:51.748679+0000","last_undegraded":"2024-04-22T01:14:51.748679+0000","last_fullsized":"2024-04-22T01:14:51.748679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865444+0000","last_change":"2024-04-22T01:14:50.783908+0000","last_active":"2024-04-22T01:14:53.865444+0000","last_peered":"2024-04-22T01:14:53.865444+0000","last_clean":"2024-04-22T01:14:53.865444+0000","last_became_active":"2024-04-22T01:14:50.782537+0000","last_became_peered":"2024-04-22T01:14:50.782537+0000","last_unstale":"2024-04-22T01:14:53.865444+0000","last_undegraded":"2024-04-22T01:14:53.865444+0000","last_fullsized":"2024-04-22T01:14:53.865444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864361+0000","last_change":"2024-04-22T01:14:46.681168+0000","last_active":"2024-04-22T01:14:53.864361+0000","last_peered":"2024-04-22T01:14:53.864361+0000","last_clean":"2024-04-22T01:14:53.864361+0000","last_became_active":"2024-04-22T01:14:46.680888+0000","last_became_peered":"2024-04-22T01:14:46.680888+0000","last_unstale":"2024-04-22T01:14:53.864361+0000","last_undegraded":"2024-04-22T01:14:53.864361+0000","last_fullsized":"2024-04-22T01:14:53.864361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744510+0000","last_change":"2024-04-22T01:14:45.677217+0000","last_active":"2024-04-22T01:14:51.744510+0000","last_peered":"2024-04-22T01:14:51.744510+0000","last_clean":"2024-04-22T01:14:51.744510+0000","last_became_active":"2024-04-22T01:14:45.676630+0000","last_became_peered":"2024-04-22T01:14:45.676630+0000","last_unstale":"2024-04-22T01:14:51.744510+0000","last_undegraded":"2024-04-22T01:14:51.744510+0000","last_fullsized":"2024-04-22T01:14:51.744510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865873+0000","last_change":"2024-04-22T01:14:50.778827+0000","last_active":"2024-04-22T01:14:53.865873+0000","last_peered":"2024-04-22T01:14:53.865873+0000","last_clean":"2024-04-22T01:14:53.865873+0000","last_became_active":"2024-04-22T01:14:50.773642+0000","last_became_peered":"2024-04-22T01:14:50.773642+0000","last_unstale":"2024-04-22T01:14:53.865873+0000","last_undegraded":"2024-04-22T01:14:53.865873+0000","last_fullsized":"2024-04-22T01:14:53.865873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.804659+0000","last_change":"2024-04-22T01:14:46.688078+0000","last_active":"2024-04-22T01:14:52.804659+0000","last_peered":"2024-04-22T01:14:52.804659+0000","last_clean":"2024-04-22T01:14:52.804659+0000","last_became_active":"2024-04-22T01:14:46.687685+0000","last_became_peered":"2024-04-22T01:14:46.687685+0000","last_unstale":"2024-04-22T01:14:52.804659+0000","last_undegraded":"2024-04-22T01:14:52.804659+0000","last_fullsized":"2024-04-22T01:14:52.804659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444194+0000","last_change":"2024-04-22T01:14:45.687324+0000","last_active":"2024-04-22T01:14:53.444194+0000","last_peered":"2024-04-22T01:14:53.444194+0000","last_clean":"2024-04-22T01:14:53.444194+0000","last_became_active":"2024-04-22T01:14:45.686553+0000","last_became_peered":"2024-04-22T01:14:45.686553+0000","last_unstale":"2024-04-22T01:14:53.444194+0000","last_undegraded":"2024-04-22T01:14:53.444194+0000","last_fullsized":"2024-04-22T01:14:53.444194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806076+0000","last_change":"2024-04-22T01:14:50.931597+0000","last_active":"2024-04-22T01:14:52.806076+0000","last_peered":"2024-04-22T01:14:52.806076+0000","last_clean":"2024-04-22T01:14:52.806076+0000","last_became_active":"2024-04-22T01:14:50.931200+0000","last_became_peered":"2024-04-22T01:14:50.931200+0000","last_unstale":"2024-04-22T01:14:52.806076+0000","last_undegraded":"2024-04-22T01:14:52.806076+0000","last_fullsized":"2024-04-22T01:14:52.806076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916581+0000","last_change":"2024-04-22T01:14:46.694352+0000","last_active":"2024-04-22T01:14:53.916581+0000","last_peered":"2024-04-22T01:14:53.916581+0000","last_clean":"2024-04-22T01:14:53.916581+0000","last_became_active":"2024-04-22T01:14:46.693993+0000","last_became_peered":"2024-04-22T01:14:46.693993+0000","last_unstale":"2024-04-22T01:14:53.916581+0000","last_undegraded":"2024-04-22T01:14:53.916581+0000","last_fullsized":"2024-04-22T01:14:53.916581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749278+0000","last_change":"2024-04-22T01:14:45.692182+0000","last_active":"2024-04-22T01:14:51.749278+0000","last_peered":"2024-04-22T01:14:51.749278+0000","last_clean":"2024-04-22T01:14:51.749278+0000","last_became_active":"2024-04-22T01:14:45.670363+0000","last_became_peered":"2024-04-22T01:14:45.670363+0000","last_unstale":"2024-04-22T01:14:51.749278+0000","last_undegraded":"2024-04-22T01:14:51.749278+0000","last_fullsized":"2024-04-22T01:14:51.749278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444406+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:14:53.444406+0000","last_peered":"2024-04-22T01:14:53.444406+0000","last_clean":"2024-04-22T01:14:53.444406+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:14:53.444406+0000","last_undegraded":"2024-04-22T01:14:53.444406+0000","last_fullsized":"2024-04-22T01:14:53.444406+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941257+0000","last_change":"2024-04-22T01:14:50.933466+0000","last_active":"2024-04-22T01:14:53.941257+0000","last_peered":"2024-04-22T01:14:53.941257+0000","last_clean":"2024-04-22T01:14:53.941257+0000","last_became_active":"2024-04-22T01:14:50.932162+0000","last_became_peered":"2024-04-22T01:14:50.932162+0000","last_unstale":"2024-04-22T01:14:53.941257+0000","last_undegraded":"2024-04-22T01:14:53.941257+0000","last_fullsized":"2024-04-22T01:14:53.941257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445273+0000","last_change":"2024-04-22T01:14:46.726237+0000","last_active":"2024-04-22T01:14:53.445273+0000","last_peered":"2024-04-22T01:14:53.445273+0000","last_clean":"2024-04-22T01:14:53.445273+0000","last_became_active":"2024-04-22T01:14:46.725538+0000","last_became_peered":"2024-04-22T01:14:46.725538+0000","last_unstale":"2024-04-22T01:14:53.445273+0000","last_undegraded":"2024-04-22T01:14:53.445273+0000","last_fullsized":"2024-04-22T01:14:53.445273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744593+0000","last_change":"2024-04-22T01:14:45.695208+0000","last_active":"2024-04-22T01:14:51.744593+0000","last_peered":"2024-04-22T01:14:51.744593+0000","last_clean":"2024-04-22T01:14:51.744593+0000","last_became_active":"2024-04-22T01:14:45.694829+0000","last_became_peered":"2024-04-22T01:14:45.694829+0000","last_unstale":"2024-04-22T01:14:51.744593+0000","last_undegraded":"2024-04-22T01:14:51.744593+0000","last_fullsized":"2024-04-22T01:14:51.744593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940759+0000","last_change":"2024-04-22T01:14:50.925049+0000","last_active":"2024-04-22T01:14:53.940759+0000","last_peered":"2024-04-22T01:14:53.940759+0000","last_clean":"2024-04-22T01:14:53.940759+0000","last_became_active":"2024-04-22T01:14:50.923916+0000","last_became_peered":"2024-04-22T01:14:50.923916+0000","last_unstale":"2024-04-22T01:14:53.940759+0000","last_undegraded":"2024-04-22T01:14:53.940759+0000","last_fullsized":"2024-04-22T01:14:53.940759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749772+0000","last_change":"2024-04-22T01:14:46.705928+0000","last_active":"2024-04-22T01:14:51.749772+0000","last_peered":"2024-04-22T01:14:51.749772+0000","last_clean":"2024-04-22T01:14:51.749772+0000","last_became_active":"2024-04-22T01:14:46.703511+0000","last_became_peered":"2024-04-22T01:14:46.703511+0000","last_unstale":"2024-04-22T01:14:51.749772+0000","last_undegraded":"2024-04-22T01:14:51.749772+0000","last_fullsized":"2024-04-22T01:14:51.749772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863430+0000","last_change":"2024-04-22T01:14:45.690308+0000","last_active":"2024-04-22T01:14:53.863430+0000","last_peered":"2024-04-22T01:14:53.863430+0000","last_clean":"2024-04-22T01:14:53.863430+0000","last_became_active":"2024-04-22T01:14:45.689948+0000","last_became_peered":"2024-04-22T01:14:45.689948+0000","last_unstale":"2024-04-22T01:14:53.863430+0000","last_undegraded":"2024-04-22T01:14:53.863430+0000","last_fullsized":"2024-04-22T01:14:53.863430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805877+0000","last_change":"2024-04-22T01:14:50.789559+0000","last_active":"2024-04-22T01:14:52.805877+0000","last_peered":"2024-04-22T01:14:52.805877+0000","last_clean":"2024-04-22T01:14:52.805877+0000","last_became_active":"2024-04-22T01:14:50.789026+0000","last_became_peered":"2024-04-22T01:14:50.789026+0000","last_unstale":"2024-04-22T01:14:52.805877+0000","last_undegraded":"2024-04-22T01:14:52.805877+0000","last_fullsized":"2024-04-22T01:14:52.805877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.446055+0000","last_change":"2024-04-22T01:14:46.734875+0000","last_active":"2024-04-22T01:14:53.446055+0000","last_peered":"2024-04-22T01:14:53.446055+0000","last_clean":"2024-04-22T01:14:53.446055+0000","last_became_active":"2024-04-22T01:14:46.733035+0000","last_became_peered":"2024-04-22T01:14:46.733035+0000","last_unstale":"2024-04-22T01:14:53.446055+0000","last_undegraded":"2024-04-22T01:14:53.446055+0000","last_fullsized":"2024-04-22T01:14:53.446055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744490+0000","last_change":"2024-04-22T01:14:45.692886+0000","last_active":"2024-04-22T01:14:51.744490+0000","last_peered":"2024-04-22T01:14:51.744490+0000","last_clean":"2024-04-22T01:14:51.744490+0000","last_became_active":"2024-04-22T01:14:45.692578+0000","last_became_peered":"2024-04-22T01:14:45.692578+0000","last_unstale":"2024-04-22T01:14:51.744490+0000","last_undegraded":"2024-04-22T01:14:51.744490+0000","last_fullsized":"2024-04-22T01:14:51.744490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745112+0000","last_change":"2024-04-22T01:14:50.925456+0000","last_active":"2024-04-22T01:14:51.745112+0000","last_peered":"2024-04-22T01:14:51.745112+0000","last_clean":"2024-04-22T01:14:51.745112+0000","last_became_active":"2024-04-22T01:14:50.924327+0000","last_became_peered":"2024-04-22T01:14:50.924327+0000","last_unstale":"2024-04-22T01:14:51.745112+0000","last_undegraded":"2024-04-22T01:14:51.745112+0000","last_fullsized":"2024-04-22T01:14:51.745112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445557+0000","last_change":"2024-04-22T01:14:46.711269+0000","last_active":"2024-04-22T01:14:53.445557+0000","last_peered":"2024-04-22T01:14:53.445557+0000","last_clean":"2024-04-22T01:14:53.445557+0000","last_became_active":"2024-04-22T01:14:46.710502+0000","last_became_peered":"2024-04-22T01:14:46.710502+0000","last_unstale":"2024-04-22T01:14:53.445557+0000","last_undegraded":"2024-04-22T01:14:53.445557+0000","last_fullsized":"2024-04-22T01:14:53.445557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746774+0000","last_change":"2024-04-22T01:14:45.679532+0000","last_active":"2024-04-22T01:14:51.746774+0000","last_peered":"2024-04-22T01:14:51.746774+0000","last_clean":"2024-04-22T01:14:51.746774+0000","last_became_active":"2024-04-22T01:14:45.678724+0000","last_became_peered":"2024-04-22T01:14:45.678724+0000","last_unstale":"2024-04-22T01:14:51.746774+0000","last_undegraded":"2024-04-22T01:14:51.746774+0000","last_fullsized":"2024-04-22T01:14:51.746774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:09.261855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744892+0000","last_change":"2024-04-22T01:14:50.780337+0000","last_active":"2024-04-22T01:14:51.744892+0000","last_peered":"2024-04-22T01:14:51.744892+0000","last_clean":"2024-04-22T01:14:51.744892+0000","last_became_active":"2024-04-22T01:14:50.779283+0000","last_became_peered":"2024-04-22T01:14:50.779283+0000","last_unstale":"2024-04-22T01:14:51.744892+0000","last_undegraded":"2024-04-22T01:14:51.744892+0000","last_fullsized":"2024-04-22T01:14:51.744892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:34.466322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444639+0000","last_change":"2024-04-22T01:14:46.719676+0000","last_active":"2024-04-22T01:14:53.444639+0000","last_peered":"2024-04-22T01:14:53.444639+0000","last_clean":"2024-04-22T01:14:53.444639+0000","last_became_active":"2024-04-22T01:14:46.712569+0000","last_became_peered":"2024-04-22T01:14:46.712569+0000","last_unstale":"2024-04-22T01:14:53.444639+0000","last_undegraded":"2024-04-22T01:14:53.444639+0000","last_fullsized":"2024-04-22T01:14:53.444639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744016+0000","last_change":"2024-04-22T01:14:45.681046+0000","last_active":"2024-04-22T01:14:51.744016+0000","last_peered":"2024-04-22T01:14:51.744016+0000","last_clean":"2024-04-22T01:14:51.744016+0000","last_became_active":"2024-04-22T01:14:45.680666+0000","last_became_peered":"2024-04-22T01:14:45.680666+0000","last_unstale":"2024-04-22T01:14:51.744016+0000","last_undegraded":"2024-04-22T01:14:51.744016+0000","last_fullsized":"2024-04-22T01:14:51.744016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805846+0000","last_change":"2024-04-22T01:14:50.767393+0000","last_active":"2024-04-22T01:14:52.805846+0000","last_peered":"2024-04-22T01:14:52.805846+0000","last_clean":"2024-04-22T01:14:52.805846+0000","last_became_active":"2024-04-22T01:14:50.766951+0000","last_became_peered":"2024-04-22T01:14:50.766951+0000","last_unstale":"2024-04-22T01:14:52.805846+0000","last_undegraded":"2024-04-22T01:14:52.805846+0000","last_fullsized":"2024-04-22T01:14:52.805846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940415+0000","last_change":"2024-04-22T01:14:46.706644+0000","last_active":"2024-04-22T01:14:53.940415+0000","last_peered":"2024-04-22T01:14:53.940415+0000","last_clean":"2024-04-22T01:14:53.940415+0000","last_became_active":"2024-04-22T01:14:46.705447+0000","last_became_peered":"2024-04-22T01:14:46.705447+0000","last_unstale":"2024-04-22T01:14:53.940415+0000","last_undegraded":"2024-04-22T01:14:53.940415+0000","last_fullsized":"2024-04-22T01:14:53.940415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939587+0000","last_change":"2024-04-22T01:14:45.692258+0000","last_active":"2024-04-22T01:14:53.939587+0000","last_peered":"2024-04-22T01:14:53.939587+0000","last_clean":"2024-04-22T01:14:53.939587+0000","last_became_active":"2024-04-22T01:14:45.691375+0000","last_became_peered":"2024-04-22T01:14:45.691375+0000","last_unstale":"2024-04-22T01:14:53.939587+0000","last_undegraded":"2024-04-22T01:14:53.939587+0000","last_fullsized":"2024-04-22T01:14:53.939587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918239+0000","last_change":"2024-04-22T01:14:50.932861+0000","last_active":"2024-04-22T01:14:53.918239+0000","last_peered":"2024-04-22T01:14:53.918239+0000","last_clean":"2024-04-22T01:14:53.918239+0000","last_became_active":"2024-04-22T01:14:50.932500+0000","last_became_peered":"2024-04-22T01:14:50.932500+0000","last_unstale":"2024-04-22T01:14:53.918239+0000","last_undegraded":"2024-04-22T01:14:53.918239+0000","last_fullsized":"2024-04-22T01:14:53.918239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:13.452468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445087+0000","last_change":"2024-04-22T01:14:46.733784+0000","last_active":"2024-04-22T01:14:53.445087+0000","last_peered":"2024-04-22T01:14:53.445087+0000","last_clean":"2024-04-22T01:14:53.445087+0000","last_became_active":"2024-04-22T01:14:46.732753+0000","last_became_peered":"2024-04-22T01:14:46.732753+0000","last_unstale":"2024-04-22T01:14:53.445087+0000","last_undegraded":"2024-04-22T01:14:53.445087+0000","last_fullsized":"2024-04-22T01:14:53.445087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750181+0000","last_change":"2024-04-22T01:14:45.706819+0000","last_active":"2024-04-22T01:14:51.750181+0000","last_peered":"2024-04-22T01:14:51.750181+0000","last_clean":"2024-04-22T01:14:51.750181+0000","last_became_active":"2024-04-22T01:14:45.705704+0000","last_became_peered":"2024-04-22T01:14:45.705704+0000","last_unstale":"2024-04-22T01:14:51.750181+0000","last_undegraded":"2024-04-22T01:14:51.750181+0000","last_fullsized":"2024-04-22T01:14:51.750181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917603+0000","last_change":"2024-04-22T01:14:50.925247+0000","last_active":"2024-04-22T01:14:53.917603+0000","last_peered":"2024-04-22T01:14:53.917603+0000","last_clean":"2024-04-22T01:14:53.917603+0000","last_became_active":"2024-04-22T01:14:50.924739+0000","last_became_peered":"2024-04-22T01:14:50.924739+0000","last_unstale":"2024-04-22T01:14:53.917603+0000","last_undegraded":"2024-04-22T01:14:53.917603+0000","last_fullsized":"2024-04-22T01:14:53.917603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:36.206609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805145+0000","last_change":"2024-04-22T01:14:46.686331+0000","last_active":"2024-04-22T01:14:52.805145+0000","last_peered":"2024-04-22T01:14:52.805145+0000","last_clean":"2024-04-22T01:14:52.805145+0000","last_became_active":"2024-04-22T01:14:46.686010+0000","last_became_peered":"2024-04-22T01:14:46.686010+0000","last_unstale":"2024-04-22T01:14:52.805145+0000","last_undegraded":"2024-04-22T01:14:52.805145+0000","last_fullsized":"2024-04-22T01:14:52.805145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806145+0000","last_change":"2024-04-22T01:14:45.686652+0000","last_active":"2024-04-22T01:14:52.806145+0000","last_peered":"2024-04-22T01:14:52.806145+0000","last_clean":"2024-04-22T01:14:52.806145+0000","last_became_active":"2024-04-22T01:14:45.686365+0000","last_became_peered":"2024-04-22T01:14:45.686365+0000","last_unstale":"2024-04-22T01:14:52.806145+0000","last_undegraded":"2024-04-22T01:14:52.806145+0000","last_fullsized":"2024-04-22T01:14:52.806145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749848+0000","last_change":"2024-04-22T01:14:50.925756+0000","last_active":"2024-04-22T01:14:51.749848+0000","last_peered":"2024-04-22T01:14:51.749848+0000","last_clean":"2024-04-22T01:14:51.749848+0000","last_became_active":"2024-04-22T01:14:50.924521+0000","last_became_peered":"2024-04-22T01:14:50.924521+0000","last_unstale":"2024-04-22T01:14:51.749848+0000","last_undegraded":"2024-04-22T01:14:51.749848+0000","last_fullsized":"2024-04-22T01:14:51.749848+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444945+0000","last_change":"2024-04-22T01:14:46.717218+0000","last_active":"2024-04-22T01:14:53.444945+0000","last_peered":"2024-04-22T01:14:53.444945+0000","last_clean":"2024-04-22T01:14:53.444945+0000","last_became_active":"2024-04-22T01:14:46.716342+0000","last_became_peered":"2024-04-22T01:14:46.716342+0000","last_unstale":"2024-04-22T01:14:53.444945+0000","last_undegraded":"2024-04-22T01:14:53.444945+0000","last_fullsized":"2024-04-22T01:14:53.444945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:57.350819+0000","last_change":"2024-04-22T01:14:45.688781+0000","last_active":"2024-04-22T01:14:57.350819+0000","last_peered":"2024-04-22T01:14:57.350819+0000","last_clean":"2024-04-22T01:14:57.350819+0000","last_became_active":"2024-04-22T01:14:45.688153+0000","last_became_peered":"2024-04-22T01:14:45.688153+0000","last_unstale":"2024-04-22T01:14:57.350819+0000","last_undegraded":"2024-04-22T01:14:57.350819+0000","last_fullsized":"2024-04-22T01:14:57.350819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864909+0000","last_change":"2024-04-22T01:14:50.778877+0000","last_active":"2024-04-22T01:14:53.864909+0000","last_peered":"2024-04-22T01:14:53.864909+0000","last_clean":"2024-04-22T01:14:53.864909+0000","last_became_active":"2024-04-22T01:14:50.773623+0000","last_became_peered":"2024-04-22T01:14:50.773623+0000","last_unstale":"2024-04-22T01:14:53.864909+0000","last_undegraded":"2024-04-22T01:14:53.864909+0000","last_fullsized":"2024-04-22T01:14:53.864909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746481+0000","last_change":"2024-04-22T01:14:46.727618+0000","last_active":"2024-04-22T01:14:51.746481+0000","last_peered":"2024-04-22T01:14:51.746481+0000","last_clean":"2024-04-22T01:14:51.746481+0000","last_became_active":"2024-04-22T01:14:46.727055+0000","last_became_peered":"2024-04-22T01:14:46.727055+0000","last_unstale":"2024-04-22T01:14:51.746481+0000","last_undegraded":"2024-04-22T01:14:51.746481+0000","last_fullsized":"2024-04-22T01:14:51.746481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:15.551380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863451+0000","last_change":"2024-04-22T01:14:45.695088+0000","last_active":"2024-04-22T01:14:53.863451+0000","last_peered":"2024-04-22T01:14:53.863451+0000","last_clean":"2024-04-22T01:14:53.863451+0000","last_became_active":"2024-04-22T01:14:45.694765+0000","last_became_peered":"2024-04-22T01:14:45.694765+0000","last_unstale":"2024-04-22T01:14:53.863451+0000","last_undegraded":"2024-04-22T01:14:53.863451+0000","last_fullsized":"2024-04-22T01:14:53.863451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747096+0000","last_change":"2024-04-22T01:14:50.782280+0000","last_active":"2024-04-22T01:14:51.747096+0000","last_peered":"2024-04-22T01:14:51.747096+0000","last_clean":"2024-04-22T01:14:51.747096+0000","last_became_active":"2024-04-22T01:14:50.779944+0000","last_became_peered":"2024-04-22T01:14:50.779944+0000","last_unstale":"2024-04-22T01:14:51.747096+0000","last_undegraded":"2024-04-22T01:14:51.747096+0000","last_fullsized":"2024-04-22T01:14:51.747096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805792+0000","last_change":"2024-04-22T01:14:46.688527+0000","last_active":"2024-04-22T01:14:52.805792+0000","last_peered":"2024-04-22T01:14:52.805792+0000","last_clean":"2024-04-22T01:14:52.805792+0000","last_became_active":"2024-04-22T01:14:46.687989+0000","last_became_peered":"2024-04-22T01:14:46.687989+0000","last_unstale":"2024-04-22T01:14:52.805792+0000","last_undegraded":"2024-04-22T01:14:52.805792+0000","last_fullsized":"2024-04-22T01:14:52.805792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749455+0000","last_change":"2024-04-22T01:14:45.699988+0000","last_active":"2024-04-22T01:14:51.749455+0000","last_peered":"2024-04-22T01:14:51.749455+0000","last_clean":"2024-04-22T01:14:51.749455+0000","last_became_active":"2024-04-22T01:14:45.692830+0000","last_became_peered":"2024-04-22T01:14:45.692830+0000","last_unstale":"2024-04-22T01:14:51.749455+0000","last_undegraded":"2024-04-22T01:14:51.749455+0000","last_fullsized":"2024-04-22T01:14:51.749455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916418+0000","last_change":"2024-04-22T01:14:50.781549+0000","last_active":"2024-04-22T01:14:53.916418+0000","last_peered":"2024-04-22T01:14:53.916418+0000","last_clean":"2024-04-22T01:14:53.916418+0000","last_became_active":"2024-04-22T01:14:50.780774+0000","last_became_peered":"2024-04-22T01:14:50.780774+0000","last_unstale":"2024-04-22T01:14:53.916418+0000","last_undegraded":"2024-04-22T01:14:53.916418+0000","last_fullsized":"2024-04-22T01:14:53.916418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749318+0000","last_change":"2024-04-22T01:14:46.686118+0000","last_active":"2024-04-22T01:14:51.749318+0000","last_peered":"2024-04-22T01:14:51.749318+0000","last_clean":"2024-04-22T01:14:51.749318+0000","last_became_active":"2024-04-22T01:14:46.685329+0000","last_became_peered":"2024-04-22T01:14:46.685329+0000","last_unstale":"2024-04-22T01:14:51.749318+0000","last_undegraded":"2024-04-22T01:14:51.749318+0000","last_fullsized":"2024-04-22T01:14:51.749318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939780+0000","last_change":"2024-04-22T01:14:45.690257+0000","last_active":"2024-04-22T01:14:53.939780+0000","last_peered":"2024-04-22T01:14:53.939780+0000","last_clean":"2024-04-22T01:14:53.939780+0000","last_became_active":"2024-04-22T01:14:45.688794+0000","last_became_peered":"2024-04-22T01:14:45.688794+0000","last_unstale":"2024-04-22T01:14:53.939780+0000","last_undegraded":"2024-04-22T01:14:53.939780+0000","last_fullsized":"2024-04-22T01:14:53.939780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916549+0000","last_change":"2024-04-22T01:14:50.930535+0000","last_active":"2024-04-22T01:14:53.916549+0000","last_peered":"2024-04-22T01:14:53.916549+0000","last_clean":"2024-04-22T01:14:53.916549+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.916549+0000","last_undegraded":"2024-04-22T01:14:53.916549+0000","last_fullsized":"2024-04-22T01:14:53.916549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805736+0000","last_change":"2024-04-22T01:14:50.932710+0000","last_active":"2024-04-22T01:14:52.805736+0000","last_peered":"2024-04-22T01:14:52.805736+0000","last_clean":"2024-04-22T01:14:52.805736+0000","last_became_active":"2024-04-22T01:14:50.931957+0000","last_became_peered":"2024-04-22T01:14:50.931957+0000","last_unstale":"2024-04-22T01:14:52.805736+0000","last_undegraded":"2024-04-22T01:14:52.805736+0000","last_fullsized":"2024-04-22T01:14:52.805736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864231+0000","last_change":"2024-04-22T01:14:46.697192+0000","last_active":"2024-04-22T01:14:53.864231+0000","last_peered":"2024-04-22T01:14:53.864231+0000","last_clean":"2024-04-22T01:14:53.864231+0000","last_became_active":"2024-04-22T01:14:46.696894+0000","last_became_peered":"2024-04-22T01:14:46.696894+0000","last_unstale":"2024-04-22T01:14:53.864231+0000","last_undegraded":"2024-04-22T01:14:53.864231+0000","last_fullsized":"2024-04-22T01:14:53.864231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745866+0000","last_change":"2024-04-22T01:14:45.701127+0000","last_active":"2024-04-22T01:14:51.745866+0000","last_peered":"2024-04-22T01:14:51.745866+0000","last_clean":"2024-04-22T01:14:51.745866+0000","last_became_active":"2024-04-22T01:14:45.699950+0000","last_became_peered":"2024-04-22T01:14:45.699950+0000","last_unstale":"2024-04-22T01:14:51.745866+0000","last_undegraded":"2024-04-22T01:14:51.745866+0000","last_fullsized":"2024-04-22T01:14:51.745866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941786+0000","last_change":"2024-04-22T01:14:50.781112+0000","last_active":"2024-04-22T01:14:53.941786+0000","last_peered":"2024-04-22T01:14:53.941786+0000","last_clean":"2024-04-22T01:14:53.941786+0000","last_became_active":"2024-04-22T01:14:50.780333+0000","last_became_peered":"2024-04-22T01:14:50.780333+0000","last_unstale":"2024-04-22T01:14:53.941786+0000","last_undegraded":"2024-04-22T01:14:53.941786+0000","last_fullsized":"2024-04-22T01:14:53.941786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916408+0000","last_change":"2024-04-22T01:14:46.699240+0000","last_active":"2024-04-22T01:14:53.916408+0000","last_peered":"2024-04-22T01:14:53.916408+0000","last_clean":"2024-04-22T01:14:53.916408+0000","last_became_active":"2024-04-22T01:14:46.698319+0000","last_became_peered":"2024-04-22T01:14:46.698319+0000","last_unstale":"2024-04-22T01:14:53.916408+0000","last_undegraded":"2024-04-22T01:14:53.916408+0000","last_fullsized":"2024-04-22T01:14:53.916408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916909+0000","last_change":"2024-04-22T01:14:45.692605+0000","last_active":"2024-04-22T01:14:53.916909+0000","last_peered":"2024-04-22T01:14:53.916909+0000","last_clean":"2024-04-22T01:14:53.916909+0000","last_became_active":"2024-04-22T01:14:45.692201+0000","last_became_peered":"2024-04-22T01:14:45.692201+0000","last_unstale":"2024-04-22T01:14:53.916909+0000","last_undegraded":"2024-04-22T01:14:53.916909+0000","last_fullsized":"2024-04-22T01:14:53.916909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939598+0000","last_change":"2024-04-22T01:14:45.690896+0000","last_active":"2024-04-22T01:14:53.939598+0000","last_peered":"2024-04-22T01:14:53.939598+0000","last_clean":"2024-04-22T01:14:53.939598+0000","last_became_active":"2024-04-22T01:14:45.689871+0000","last_became_peered":"2024-04-22T01:14:45.689871+0000","last_unstale":"2024-04-22T01:14:53.939598+0000","last_undegraded":"2024-04-22T01:14:53.939598+0000","last_fullsized":"2024-04-22T01:14:53.939598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806029+0000","last_change":"2024-04-22T01:14:46.685575+0000","last_active":"2024-04-22T01:14:52.806029+0000","last_peered":"2024-04-22T01:14:52.806029+0000","last_clean":"2024-04-22T01:14:52.806029+0000","last_became_active":"2024-04-22T01:14:46.685301+0000","last_became_peered":"2024-04-22T01:14:46.685301+0000","last_unstale":"2024-04-22T01:14:52.806029+0000","last_undegraded":"2024-04-22T01:14:52.806029+0000","last_fullsized":"2024-04-22T01:14:52.806029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744699+0000","last_change":"2024-04-22T01:14:50.775646+0000","last_active":"2024-04-22T01:14:51.744699+0000","last_peered":"2024-04-22T01:14:51.744699+0000","last_clean":"2024-04-22T01:14:51.744699+0000","last_became_active":"2024-04-22T01:14:50.775153+0000","last_became_peered":"2024-04-22T01:14:50.775153+0000","last_unstale":"2024-04-22T01:14:51.744699+0000","last_undegraded":"2024-04-22T01:14:51.744699+0000","last_fullsized":"2024-04-22T01:14:51.744699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:46.711296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915399+0000","last_change":"2024-04-22T01:14:45.700287+0000","last_active":"2024-04-22T01:14:53.915399+0000","last_peered":"2024-04-22T01:14:53.915399+0000","last_clean":"2024-04-22T01:14:53.915399+0000","last_became_active":"2024-04-22T01:14:45.700063+0000","last_became_peered":"2024-04-22T01:14:45.700063+0000","last_unstale":"2024-04-22T01:14:53.915399+0000","last_undegraded":"2024-04-22T01:14:53.915399+0000","last_fullsized":"2024-04-22T01:14:53.915399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745633+0000","last_change":"2024-04-22T01:14:46.694633+0000","last_active":"2024-04-22T01:14:51.745633+0000","last_peered":"2024-04-22T01:14:51.745633+0000","last_clean":"2024-04-22T01:14:51.745633+0000","last_became_active":"2024-04-22T01:14:46.690904+0000","last_became_peered":"2024-04-22T01:14:46.690904+0000","last_unstale":"2024-04-22T01:14:51.745633+0000","last_undegraded":"2024-04-22T01:14:51.745633+0000","last_fullsized":"2024-04-22T01:14:51.745633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:35.052516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917858+0000","last_change":"2024-04-22T01:14:50.775487+0000","last_active":"2024-04-22T01:14:53.917858+0000","last_peered":"2024-04-22T01:14:53.917858+0000","last_clean":"2024-04-22T01:14:53.917858+0000","last_became_active":"2024-04-22T01:14:50.774770+0000","last_became_peered":"2024-04-22T01:14:50.774770+0000","last_unstale":"2024-04-22T01:14:53.917858+0000","last_undegraded":"2024-04-22T01:14:53.917858+0000","last_fullsized":"2024-04-22T01:14:53.917858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:59:18.378937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940165+0000","last_change":"2024-04-22T01:14:46.715466+0000","last_active":"2024-04-22T01:14:53.940165+0000","last_peered":"2024-04-22T01:14:53.940165+0000","last_clean":"2024-04-22T01:14:53.940165+0000","last_became_active":"2024-04-22T01:14:46.706194+0000","last_became_peered":"2024-04-22T01:14:46.706194+0000","last_unstale":"2024-04-22T01:14:53.940165+0000","last_undegraded":"2024-04-22T01:14:53.940165+0000","last_fullsized":"2024-04-22T01:14:53.940165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744256+0000","last_change":"2024-04-22T01:14:46.695116+0000","last_active":"2024-04-22T01:14:51.744256+0000","last_peered":"2024-04-22T01:14:51.744256+0000","last_clean":"2024-04-22T01:14:51.744256+0000","last_became_active":"2024-04-22T01:14:46.694817+0000","last_became_peered":"2024-04-22T01:14:46.694817+0000","last_unstale":"2024-04-22T01:14:51.744256+0000","last_undegraded":"2024-04-22T01:14:51.744256+0000","last_fullsized":"2024-04-22T01:14:51.744256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748407+0000","last_change":"2024-04-22T01:14:46.697812+0000","last_active":"2024-04-22T01:14:51.748407+0000","last_peered":"2024-04-22T01:14:51.748407+0000","last_clean":"2024-04-22T01:14:51.748407+0000","last_became_active":"2024-04-22T01:14:46.697056+0000","last_became_peered":"2024-04-22T01:14:46.697056+0000","last_unstale":"2024-04-22T01:14:51.748407+0000","last_undegraded":"2024-04-22T01:14:51.748407+0000","last_fullsized":"2024-04-22T01:14:51.748407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805883+0000","last_change":"2024-04-22T01:14:46.696293+0000","last_active":"2024-04-22T01:14:52.805883+0000","last_peered":"2024-04-22T01:14:52.805883+0000","last_clean":"2024-04-22T01:14:52.805883+0000","last_became_active":"2024-04-22T01:14:46.695950+0000","last_became_peered":"2024-04-22T01:14:46.695950+0000","last_unstale":"2024-04-22T01:14:52.805883+0000","last_undegraded":"2024-04-22T01:14:52.805883+0000","last_fullsized":"2024-04-22T01:14:52.805883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917964+0000","last_change":"2024-04-22T01:14:46.705052+0000","last_active":"2024-04-22T01:14:53.917964+0000","last_peered":"2024-04-22T01:14:53.917964+0000","last_clean":"2024-04-22T01:14:53.917964+0000","last_became_active":"2024-04-22T01:14:46.704239+0000","last_became_peered":"2024-04-22T01:14:46.704239+0000","last_unstale":"2024-04-22T01:14:53.917964+0000","last_undegraded":"2024-04-22T01:14:53.917964+0000","last_fullsized":"2024-04-22T01:14:53.917964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865529+0000","last_change":"2024-04-22T01:14:46.702587+0000","last_active":"2024-04-22T01:14:53.865529+0000","last_peered":"2024-04-22T01:14:53.865529+0000","last_clean":"2024-04-22T01:14:53.865529+0000","last_became_active":"2024-04-22T01:14:46.702257+0000","last_became_peered":"2024-04-22T01:14:46.702257+0000","last_unstale":"2024-04-22T01:14:53.865529+0000","last_undegraded":"2024-04-22T01:14:53.865529+0000","last_fullsized":"2024-04-22T01:14:53.865529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941652+0000","last_change":"2024-04-22T01:14:46.729745+0000","last_active":"2024-04-22T01:14:53.941652+0000","last_peered":"2024-04-22T01:14:53.941652+0000","last_clean":"2024-04-22T01:14:53.941652+0000","last_became_active":"2024-04-22T01:14:46.729165+0000","last_became_peered":"2024-04-22T01:14:46.729165+0000","last_unstale":"2024-04-22T01:14:53.941652+0000","last_undegraded":"2024-04-22T01:14:53.941652+0000","last_fullsized":"2024-04-22T01:14:53.941652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918240+0000","last_change":"2024-04-22T01:14:46.707058+0000","last_active":"2024-04-22T01:14:53.918240+0000","last_peered":"2024-04-22T01:14:53.918240+0000","last_clean":"2024-04-22T01:14:53.918240+0000","last_became_active":"2024-04-22T01:14:46.706772+0000","last_became_peered":"2024-04-22T01:14:46.706772+0000","last_unstale":"2024-04-22T01:14:53.918240+0000","last_undegraded":"2024-04-22T01:14:53.918240+0000","last_fullsized":"2024-04-22T01:14:53.918240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745635+0000","last_change":"2024-04-22T01:14:46.702309+0000","last_active":"2024-04-22T01:14:51.745635+0000","last_peered":"2024-04-22T01:14:51.745635+0000","last_clean":"2024-04-22T01:14:51.745635+0000","last_became_active":"2024-04-22T01:14:46.701437+0000","last_became_peered":"2024-04-22T01:14:46.701437+0000","last_unstale":"2024-04-22T01:14:51.745635+0000","last_undegraded":"2024-04-22T01:14:51.745635+0000","last_fullsized":"2024-04-22T01:14:51.745635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916675+0000","last_change":"2024-04-22T01:14:46.706939+0000","last_active":"2024-04-22T01:14:53.916675+0000","last_peered":"2024-04-22T01:14:53.916675+0000","last_clean":"2024-04-22T01:14:53.916675+0000","last_became_active":"2024-04-22T01:14:46.706649+0000","last_became_peered":"2024-04-22T01:14:46.706649+0000","last_unstale":"2024-04-22T01:14:53.916675+0000","last_undegraded":"2024-04-22T01:14:53.916675+0000","last_fullsized":"2024-04-22T01:14:53.916675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748360+0000","last_change":"2024-04-22T01:14:46.708136+0000","last_active":"2024-04-22T01:14:51.748360+0000","last_peered":"2024-04-22T01:14:51.748360+0000","last_clean":"2024-04-22T01:14:51.748360+0000","last_became_active":"2024-04-22T01:14:46.704481+0000","last_became_peered":"2024-04-22T01:14:46.704481+0000","last_unstale":"2024-04-22T01:14:51.748360+0000","last_undegraded":"2024-04-22T01:14:51.748360+0000","last_fullsized":"2024-04-22T01:14:51.748360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805288+0000","last_change":"2024-04-22T01:14:46.730861+0000","last_active":"2024-04-22T01:14:52.805288+0000","last_peered":"2024-04-22T01:14:52.805288+0000","last_clean":"2024-04-22T01:14:52.805288+0000","last_became_active":"2024-04-22T01:14:46.730520+0000","last_became_peered":"2024-04-22T01:14:46.730520+0000","last_unstale":"2024-04-22T01:14:52.805288+0000","last_undegraded":"2024-04-22T01:14:52.805288+0000","last_fullsized":"2024-04-22T01:14:52.805288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:42.634594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941258+0000","last_change":"2024-04-22T01:14:46.724974+0000","last_active":"2024-04-22T01:14:53.941258+0000","last_peered":"2024-04-22T01:14:53.941258+0000","last_clean":"2024-04-22T01:14:53.941258+0000","last_became_active":"2024-04-22T01:14:46.717427+0000","last_became_peered":"2024-04-22T01:14:46.717427+0000","last_unstale":"2024-04-22T01:14:53.941258+0000","last_undegraded":"2024-04-22T01:14:53.941258+0000","last_fullsized":"2024-04-22T01:14:53.941258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445528+0000","last_change":"2024-04-22T01:14:46.724990+0000","last_active":"2024-04-22T01:14:53.445528+0000","last_peered":"2024-04-22T01:14:53.445528+0000","last_clean":"2024-04-22T01:14:53.445528+0000","last_became_active":"2024-04-22T01:14:46.724307+0000","last_became_peered":"2024-04-22T01:14:46.724307+0000","last_unstale":"2024-04-22T01:14:53.445528+0000","last_undegraded":"2024-04-22T01:14:53.445528+0000","last_fullsized":"2024-04-22T01:14:53.445528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744480+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.744480+0000","last_peered":"2024-04-22T01:14:51.744480+0000","last_clean":"2024-04-22T01:14:51.744480+0000","last_became_active":"2024-04-22T01:14:46.719087+0000","last_became_peered":"2024-04-22T01:14:46.719087+0000","last_unstale":"2024-04-22T01:14:51.744480+0000","last_undegraded":"2024-04-22T01:14:51.744480+0000","last_fullsized":"2024-04-22T01:14:51.744480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749914+0000","last_change":"2024-04-22T01:14:46.726617+0000","last_active":"2024-04-22T01:14:51.749914+0000","last_peered":"2024-04-22T01:14:51.749914+0000","last_clean":"2024-04-22T01:14:51.749914+0000","last_became_active":"2024-04-22T01:14:46.725959+0000","last_became_peered":"2024-04-22T01:14:46.725959+0000","last_unstale":"2024-04-22T01:14:51.749914+0000","last_undegraded":"2024-04-22T01:14:51.749914+0000","last_fullsized":"2024-04-22T01:14:51.749914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745404+0000","last_change":"2024-04-22T01:14:46.678143+0000","last_active":"2024-04-22T01:14:51.745404+0000","last_peered":"2024-04-22T01:14:51.745404+0000","last_clean":"2024-04-22T01:14:51.745404+0000","last_became_active":"2024-04-22T01:14:46.677398+0000","last_became_peered":"2024-04-22T01:14:46.677398+0000","last_unstale":"2024-04-22T01:14:51.745404+0000","last_undegraded":"2024-04-22T01:14:51.745404+0000","last_fullsized":"2024-04-22T01:14:51.745404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864510+0000","last_change":"2024-04-22T01:14:46.697838+0000","last_active":"2024-04-22T01:14:53.864510+0000","last_peered":"2024-04-22T01:14:53.864510+0000","last_clean":"2024-04-22T01:14:53.864510+0000","last_became_active":"2024-04-22T01:14:46.697564+0000","last_became_peered":"2024-04-22T01:14:46.697564+0000","last_unstale":"2024-04-22T01:14:53.864510+0000","last_undegraded":"2024-04-22T01:14:53.864510+0000","last_fullsized":"2024-04-22T01:14:53.864510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805023+0000","last_change":"2024-04-22T01:14:46.679516+0000","last_active":"2024-04-22T01:14:52.805023+0000","last_peered":"2024-04-22T01:14:52.805023+0000","last_clean":"2024-04-22T01:14:52.805023+0000","last_became_active":"2024-04-22T01:14:46.679223+0000","last_became_peered":"2024-04-22T01:14:46.679223+0000","last_unstale":"2024-04-22T01:14:52.805023+0000","last_undegraded":"2024-04-22T01:14:52.805023+0000","last_fullsized":"2024-04-22T01:14:52.805023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916981+0000","last_change":"2024-04-22T01:14:46.709145+0000","last_active":"2024-04-22T01:14:53.916981+0000","last_peered":"2024-04-22T01:14:53.916981+0000","last_clean":"2024-04-22T01:14:53.916981+0000","last_became_active":"2024-04-22T01:14:46.708806+0000","last_became_peered":"2024-04-22T01:14:46.708806+0000","last_unstale":"2024-04-22T01:14:53.916981+0000","last_undegraded":"2024-04-22T01:14:53.916981+0000","last_fullsized":"2024-04-22T01:14:53.916981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746371+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.746371+0000","last_peered":"2024-04-22T01:14:51.746371+0000","last_clean":"2024-04-22T01:14:51.746371+0000","last_became_active":"2024-04-22T01:14:46.718623+0000","last_became_peered":"2024-04-22T01:14:46.718623+0000","last_unstale":"2024-04-22T01:14:51.746371+0000","last_undegraded":"2024-04-22T01:14:51.746371+0000","last_fullsized":"2024-04-22T01:14:51.746371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941253+0000","last_change":"2024-04-22T01:14:46.705553+0000","last_active":"2024-04-22T01:14:53.941253+0000","last_peered":"2024-04-22T01:14:53.941253+0000","last_clean":"2024-04-22T01:14:53.941253+0000","last_became_active":"2024-04-22T01:14:46.704621+0000","last_became_peered":"2024-04-22T01:14:46.704621+0000","last_unstale":"2024-04-22T01:14:53.941253+0000","last_undegraded":"2024-04-22T01:14:53.941253+0000","last_fullsized":"2024-04-22T01:14:53.941253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941576+0000","last_change":"2024-04-22T01:14:46.705656+0000","last_active":"2024-04-22T01:14:53.941576+0000","last_peered":"2024-04-22T01:14:53.941576+0000","last_clean":"2024-04-22T01:14:53.941576+0000","last_became_active":"2024-04-22T01:14:46.704857+0000","last_became_peered":"2024-04-22T01:14:46.704857+0000","last_unstale":"2024-04-22T01:14:53.941576+0000","last_undegraded":"2024-04-22T01:14:53.941576+0000","last_fullsized":"2024-04-22T01:14:53.941576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445983+0000","last_change":"2024-04-22T01:14:46.735811+0000","last_active":"2024-04-22T01:14:53.445983+0000","last_peered":"2024-04-22T01:14:53.445983+0000","last_clean":"2024-04-22T01:14:53.445983+0000","last_became_active":"2024-04-22T01:14:46.727833+0000","last_became_peered":"2024-04-22T01:14:46.727833+0000","last_unstale":"2024-04-22T01:14:53.445983+0000","last_undegraded":"2024-04-22T01:14:53.445983+0000","last_fullsized":"2024-04-22T01:14:53.445983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916197+0000","last_change":"2024-04-22T01:14:46.699497+0000","last_active":"2024-04-22T01:14:53.916197+0000","last_peered":"2024-04-22T01:14:53.916197+0000","last_clean":"2024-04-22T01:14:53.916197+0000","last_became_active":"2024-04-22T01:14:46.698632+0000","last_became_peered":"2024-04-22T01:14:46.698632+0000","last_unstale":"2024-04-22T01:14:53.916197+0000","last_undegraded":"2024-04-22T01:14:53.916197+0000","last_fullsized":"2024-04-22T01:14:53.916197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444562+0000","last_change":"2024-04-22T01:14:46.699917+0000","last_active":"2024-04-22T01:14:53.444562+0000","last_peered":"2024-04-22T01:14:53.444562+0000","last_clean":"2024-04-22T01:14:53.444562+0000","last_became_active":"2024-04-22T01:14:46.684087+0000","last_became_peered":"2024-04-22T01:14:46.684087+0000","last_unstale":"2024-04-22T01:14:53.444562+0000","last_undegraded":"2024-04-22T01:14:53.444562+0000","last_fullsized":"2024-04-22T01:14:53.444562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748148+0000","last_change":"2024-04-22T01:14:46.683759+0000","last_active":"2024-04-22T01:14:51.748148+0000","last_peered":"2024-04-22T01:14:51.748148+0000","last_clean":"2024-04-22T01:14:51.748148+0000","last_became_active":"2024-04-22T01:14:46.682687+0000","last_became_peered":"2024-04-22T01:14:46.682687+0000","last_unstale":"2024-04-22T01:14:51.748148+0000","last_undegraded":"2024-04-22T01:14:51.748148+0000","last_fullsized":"2024-04-22T01:14:51.748148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444815+0000","last_change":"2024-04-22T01:14:46.732970+0000","last_active":"2024-04-22T01:14:53.444815+0000","last_peered":"2024-04-22T01:14:53.444815+0000","last_clean":"2024-04-22T01:14:53.444815+0000","last_became_active":"2024-04-22T01:14:46.727597+0000","last_became_peered":"2024-04-22T01:14:46.727597+0000","last_unstale":"2024-04-22T01:14:53.444815+0000","last_undegraded":"2024-04-22T01:14:53.444815+0000","last_fullsized":"2024-04-22T01:14:53.444815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749390+0000","last_change":"2024-04-22T01:14:46.705305+0000","last_active":"2024-04-22T01:14:51.749390+0000","last_peered":"2024-04-22T01:14:51.749390+0000","last_clean":"2024-04-22T01:14:51.749390+0000","last_became_active":"2024-04-22T01:14:46.704267+0000","last_became_peered":"2024-04-22T01:14:46.704267+0000","last_unstale":"2024-04-22T01:14:51.749390+0000","last_undegraded":"2024-04-22T01:14:51.749390+0000","last_fullsized":"2024-04-22T01:14:51.749390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805327+0000","last_change":"2024-04-22T01:14:46.678217+0000","last_active":"2024-04-22T01:14:52.805327+0000","last_peered":"2024-04-22T01:14:52.805327+0000","last_clean":"2024-04-22T01:14:52.805327+0000","last_became_active":"2024-04-22T01:14:46.677881+0000","last_became_peered":"2024-04-22T01:14:46.677881+0000","last_unstale":"2024-04-22T01:14:52.805327+0000","last_undegraded":"2024-04-22T01:14:52.805327+0000","last_fullsized":"2024-04-22T01:14:52.805327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865661+0000","last_change":"2024-04-22T01:14:46.682640+0000","last_active":"2024-04-22T01:14:53.865661+0000","last_peered":"2024-04-22T01:14:53.865661+0000","last_clean":"2024-04-22T01:14:53.865661+0000","last_became_active":"2024-04-22T01:14:46.682389+0000","last_became_peered":"2024-04-22T01:14:46.682389+0000","last_unstale":"2024-04-22T01:14:53.865661+0000","last_undegraded":"2024-04-22T01:14:53.865661+0000","last_fullsized":"2024-04-22T01:14:53.865661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445838+0000","last_change":"2024-04-22T01:14:46.727060+0000","last_active":"2024-04-22T01:14:53.445838+0000","last_peered":"2024-04-22T01:14:53.445838+0000","last_clean":"2024-04-22T01:14:53.445838+0000","last_became_active":"2024-04-22T01:14:46.726485+0000","last_became_peered":"2024-04-22T01:14:46.726485+0000","last_unstale":"2024-04-22T01:14:53.445838+0000","last_undegraded":"2024-04-22T01:14:53.445838+0000","last_fullsized":"2024-04-22T01:14:53.445838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:15:11.213 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:15:11.214 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T01:15:11.214 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T01:15:11.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T01:15:11.214 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T01:15:11.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.374+0000 7f3020afc640 1 Processor -- start 2024-04-22T01:15:11.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.374+0000 7f3020afc640 1 -- start start 2024-04-22T01:15:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3020afc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f301c0cc150 0x7f301c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3020afc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f301c072060 0x7f301c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3020afc640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f301c0cd7b0 con 0x7f301c1455b0 2024-04-22T01:15:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3020afc640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f301c072490 con 0x7f301c0cc150 2024-04-22T01:15:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3020afc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f301c0cc710 con 0x7f301c072060 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f301c0cc150 0x7f301c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f301c0cc150 0x7f301c071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54564/0 (socket says 172.21.15.70:54564) 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f301a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f301c1455b0 0x7f301c0cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40724/0 (socket says 172.21.15.70:40724) 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 -- 172.21.15.70:0/1762550010 learned_addr learned my addr 172.21.15.70:0/1762550010 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f301ad9d640 1 --2- 172.21.15.70:0/1762550010 >> v2:172.21.15.98:3300/0 conn(0x7f301c072060 0x7f301c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 -- 172.21.15.70:0/1762550010 >> v2:172.21.15.98:3300/0 conn(0x7f301c072060 msgr2=0x7f301c06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 --2- 172.21.15.70:0/1762550010 >> v2:172.21.15.98:3300/0 conn(0x7f301c072060 0x7f301c06e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 -- 172.21.15.70:0/1762550010 >> v1:172.21.15.70:6789/0 conn(0x7f301c1455b0 legacy=0x7f301c0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 -- 172.21.15.70:0/1762550010 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c0cc9f0 con 0x7f301c0cc150 2024-04-22T01:15:11.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f3019d9b640 1 --2- 172.21.15.70:0/1762550010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c0cc150 0x7f301c071b20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f301c06ffb0 tx=0x7f301001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64ab555cc65e2fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f301959a640 1 -- 172.21.15.70:0/1762550010 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100b0070 con 0x7f301c0cc150 2024-04-22T01:15:11.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f301959a640 1 -- 172.21.15.70:0/1762550010 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30100ab600 con 0x7f301c0cc150 2024-04-22T01:15:11.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.378+0000 7f301959a640 1 -- 172.21.15.70:0/1762550010 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100ab9d0 con 0x7f301c0cc150 2024-04-22T01:15:11.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/1762550010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c0cc150 msgr2=0x7f301c071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 --2- 172.21.15.70:0/1762550010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c0cc150 0x7f301c071b20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f301c06ffb0 tx=0x7f301001c400 comp rx=0 tx=0).stop 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/1762550010 shutdown_connections 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 --2- 172.21.15.70:0/1762550010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c0cc150 0x7f301c071b20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 --2- 172.21.15.70:0/1762550010 >> v2:172.21.15.98:3300/0 conn(0x7f301c072060 0x7f301c06e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/1762550010 >> 172.21.15.70:0/1762550010 conn(0x7f301c06b1f0 msgr2=0x7f301c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/1762550010 shutdown_connections 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/1762550010 wait complete. 2024-04-22T01:15:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 Processor -- start 2024-04-22T01:15:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- start start 2024-04-22T01:15:11.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f301c072060 0x7f301c1436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f301c0cc150 0x7f301c13d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 0x7f301c13dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f301c06f2e0 con 0x7f301c1455b0 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f301c06f160 con 0x7f301c072060 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f301c168b40 con 0x7f301c0cc150 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 0x7f301c13dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 0x7f301c13dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54570/0 (socket says 172.21.15.70:54570) 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 -- 172.21.15.70:0/3510826138 learned_addr learned my addr 172.21.15.70:0/3510826138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3019d9b640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f301c0cc150 0x7f301c13d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301a59c640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f301c072060 0x7f301c1436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 -- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f301c0cc150 msgr2=0x7f301c13d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f301c0cc150 0x7f301c13d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 -- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f301c072060 msgr2=0x7f301c1436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f301c072060 0x7f301c1436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 -- 172.21.15.70:0/3510826138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c13e9a0 con 0x7f301c1455b0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301a59c640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f301c072060 0x7f301c1436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f301ad9d640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 0x7f301c13dd50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3014007e90 tx=0x7f301400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014014420 con 0x7f301c1455b0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3014019070 con 0x7f301c1455b0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014014800 con 0x7f301c1455b0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301c13ec80 con 0x7f301c1455b0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f301c1407a0 con 0x7f301c1455b0 2024-04-22T01:15:11.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.382+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f301c068210 con 0x7f301c1455b0 2024-04-22T01:15:11.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.386+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f30140149a0 con 0x7f301c1455b0 2024-04-22T01:15:11.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.386+0000 7f300affd640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ffc0678c0 0x7f2ffc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.386+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3014086830 con 0x7f301c1455b0 2024-04-22T01:15:11.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.386+0000 7f301a59c640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ffc0678c0 0x7f2ffc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.386+0000 7f301a59c640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ffc0678c0 0x7f2ffc069d80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f300c007fd0 tx=0x7f300c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.390+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3014011260 con 0x7f301c1455b0 2024-04-22T01:15:11.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.586+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f301c13f1a0 con 0x7f301c1455b0 2024-04-22T01:15:11.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f300affd640 1 -- 172.21.15.70:0/3510826138 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+166887 (secure 0 0 0) 0x7f3014057030 con 0x7f301c1455b0 2024-04-22T01:15:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:11.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ffc0678c0 msgr2=0x7f2ffc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ffc0678c0 0x7f2ffc069d80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f300c007fd0 tx=0x7f300c009040 comp rx=0 tx=0).stop 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 msgr2=0x7f301c13dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 0x7f301c13dd50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3014007e90 tx=0x7f301400e4b0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 shutdown_connections 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ffc0678c0 0x7f2ffc069d80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f301c1455b0 0x7f301c13dd50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f301c0cc150 0x7f301c13d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 --2- 172.21.15.70:0/3510826138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f301c072060 0x7f301c1436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 >> 172.21.15.70:0/3510826138 conn(0x7f301c06b1f0 msgr2=0x7f301c06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 shutdown_connections 2024-04-22T01:15:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.594+0000 7f3020afc640 1 -- 172.21.15.70:0/3510826138 wait complete. 2024-04-22T01:15:11.610 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4026573221},{"type":"v1","addr":"172.21.15.70:6833","nonce":4026573221}]},"active_addr":"172.21.15.70:6833/4026573221","active_change":"2024-04-22T01:14:26.574058+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4111,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3914964105}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":4029216471}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3278246881}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2861747254}]}]} 2024-04-22T01:15:11.614 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T01:15:11.614 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T01:15:11.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.730+0000 7f6d890d5640 1 Processor -- start 2024-04-22T01:15:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.730+0000 7f6d890d5640 1 -- start start 2024-04-22T01:15:11.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d890d5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6d841455b0 0x7f6d840cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d890d5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d840cc150 0x7f6d84071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d890d5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d84072470 con 0x7f6d84072060 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d890d5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6d840d0250 con 0x7f6d841455b0 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d890d5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6d840cd7b0 con 0x7f6d840cc150 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d82d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6d841455b0 0x7f6d840cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d8359e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6d84072060 0x7f6d8415f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40738/0 (socket says 172.21.15.70:40738) 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d8359e640 1 -- 172.21.15.70:0/2517780404 learned_addr learned my addr 172.21.15.70:0/2517780404 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d7a59c640 1 --2- 172.21.15.70:0/2517780404 >> v2:172.21.15.98:3300/0 conn(0x7f6d840cc150 0x7f6d84071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d82d9d640 1 -- 172.21.15.70:0/2517780404 >> v2:172.21.15.98:3300/0 conn(0x7f6d840cc150 msgr2=0x7f6d84071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d82d9d640 1 --2- 172.21.15.70:0/2517780404 >> v2:172.21.15.98:3300/0 conn(0x7f6d840cc150 0x7f6d84071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d82d9d640 1 -- 172.21.15.70:0/2517780404 >> v1:172.21.15.70:6789/0 conn(0x7f6d84072060 legacy=0x7f6d8415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d82d9d640 1 -- 172.21.15.70:0/2517780404 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d840cc710 con 0x7f6d841455b0 2024-04-22T01:15:11.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d82d9d640 1 --2- 172.21.15.70:0/2517780404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d840cfd10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d840cbd20 tx=0x7f6d7401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f00786633e0f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d8259c640 1 -- 172.21.15.70:0/2517780404 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d740b0070 con 0x7f6d841455b0 2024-04-22T01:15:11.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d8259c640 1 -- 172.21.15.70:0/2517780404 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d740ab600 con 0x7f6d841455b0 2024-04-22T01:15:11.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.734+0000 7f6d8259c640 1 -- 172.21.15.70:0/2517780404 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d740ab9d0 con 0x7f6d841455b0 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/2517780404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 msgr2=0x7f6d840cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 --2- 172.21.15.70:0/2517780404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d840cfd10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d840cbd20 tx=0x7f6d7401c360 comp rx=0 tx=0).stop 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/2517780404 shutdown_connections 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 --2- 172.21.15.70:0/2517780404 >> v2:172.21.15.98:3300/0 conn(0x7f6d840cc150 0x7f6d84071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 --2- 172.21.15.70:0/2517780404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d840cfd10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/2517780404 >> 172.21.15.70:0/2517780404 conn(0x7f6d8406b1f0 msgr2=0x7f6d8406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:11.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/2517780404 shutdown_connections 2024-04-22T01:15:11.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/2517780404 wait complete. 2024-04-22T01:15:11.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 Processor -- start 2024-04-22T01:15:11.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- start start 2024-04-22T01:15:11.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d84072060 0x7f6d84158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d840cc150 0x7f6d840b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d8415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d84168b40 con 0x7f6d841455b0 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8406f210 con 0x7f6d84072060 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6d84168cc0 con 0x7f6d840cc150 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d8415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d8415adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54582/0 (socket says 172.21.15.70:54582) 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 -- 172.21.15.70:0/1117131042 learned_addr learned my addr 172.21.15.70:0/1117131042 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 -- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d840cc150 msgr2=0x7f6d840b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d840cc150 0x7f6d840b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 -- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d84072060 msgr2=0x7f6d84158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d82d9d640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d84072060 0x7f6d84158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d84072060 0x7f6d84158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 -- 172.21.15.70:0/1117131042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d84143bf0 con 0x7f6d841455b0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d82d9d640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d84072060 0x7f6d84158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d8359e640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d8415adc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00ca20 tx=0x7f6d7c00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c017070 con 0x7f6d841455b0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d7c00abd0 con 0x7f6d841455b0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d84143ed0 con 0x7f6d841455b0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c012400 con 0x7f6d841455b0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.738+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d84144490 con 0x7f6d841455b0 2024-04-22T01:15:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.742+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d84068210 con 0x7f6d841455b0 2024-04-22T01:15:11.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.742+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d7c01a070 con 0x7f6d841455b0 2024-04-22T01:15:11.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.742+0000 7f6d7bfff640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d640678c0 0x7f6d64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:11.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.742+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6d7c085bd0 con 0x7f6d841455b0 2024-04-22T01:15:11.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.742+0000 7f6d82d9d640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d640678c0 0x7f6d64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:11.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.742+0000 7f6d82d9d640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d640678c0 0x7f6d64069d80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d740ad000 tx=0x7f6d74008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.746+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6d84072470 con 0x7f6d841455b0 2024-04-22T01:15:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.906+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d84144660 con 0x7f6d841455b0 2024-04-22T01:15:11.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d7bfff640 1 -- 172.21.15.70:0/1117131042 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f6d7c056460 con 0x7f6d841455b0 2024-04-22T01:15:11.909 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:11.909 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d640678c0 msgr2=0x7f6d64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d640678c0 0x7f6d64069d80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d740ad000 tx=0x7f6d74008040 comp rx=0 tx=0).stop 2024-04-22T01:15:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 msgr2=0x7f6d8415adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d8415adc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00ca20 tx=0x7f6d7c00cee0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 shutdown_connections 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d640678c0 0x7f6d64069d80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d841455b0 0x7f6d8415adc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d840cc150 0x7f6d840b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 --2- 172.21.15.70:0/1117131042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d84072060 0x7f6d84158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 >> 172.21.15.70:0/1117131042 conn(0x7f6d8406b1f0 msgr2=0x7f6d840b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 shutdown_connections 2024-04-22T01:15:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:11.910+0000 7f6d890d5640 1 -- 172.21.15.70:0/1117131042 wait complete. 2024-04-22T01:15:11.928 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T01:15:11.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:12.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 Processor -- start 2024-04-22T01:15:12.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 -- start start 2024-04-22T01:15:12.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfd015bb40 0x7fcfd0072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd00b3cd0 0x7fcfd0161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd015bf20 con 0x7fcfd006ed40 2024-04-22T01:15:12.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcfd006e8b0 con 0x7fcfd015bb40 2024-04-22T01:15:12.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.046+0000 7fcfd5851640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd00aec30 con 0x7fcfd00b3cd0 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfd015bb40 0x7fcfd0072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfd015bb40 0x7fcfd0072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42798/0 (socket says 172.21.15.70:42798) 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 -- 172.21.15.70:0/2352822916 learned_addr learned my addr 172.21.15.70:0/2352822916 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfcf7fe640 1 --2- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd00b3cd0 0x7fcfd0161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 -- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd00b3cd0 msgr2=0x7fcfd0161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 --2- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd00b3cd0 0x7fcfd0161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 -- 172.21.15.70:0/2352822916 >> v1:172.21.15.98:6789/0 conn(0x7fcfd006ed40 legacy=0x7fcfd00b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 -- 172.21.15.70:0/2352822916 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd00aef10 con 0x7fcfd015bb40 2024-04-22T01:15:12.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfceffd640 1 --2- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd015bb40 0x7fcfd0072530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009870 tx=0x7fcfc001c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71294640d4e59d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfcdffb640 1 -- 172.21.15.70:0/2352822916 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00a95f0 con 0x7fcfd015bb40 2024-04-22T01:15:12.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfcdffb640 1 -- 172.21.15.70:0/2352822916 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc00a9790 con 0x7fcfd015bb40 2024-04-22T01:15:12.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfcdffb640 1 -- 172.21.15.70:0/2352822916 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00a9b60 con 0x7fcfd015bb40 2024-04-22T01:15:12.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 -- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd015bb40 msgr2=0x7fcfd0072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 --2- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd015bb40 0x7fcfd0072530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009870 tx=0x7fcfc001c530 comp rx=0 tx=0).stop 2024-04-22T01:15:12.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 -- 172.21.15.70:0/2352822916 shutdown_connections 2024-04-22T01:15:12.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 --2- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd00b3cd0 0x7fcfd0161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 --2- 172.21.15.70:0/2352822916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd015bb40 0x7fcfd0072530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 -- 172.21.15.70:0/2352822916 >> 172.21.15.70:0/2352822916 conn(0x7fcfd006b1f0 msgr2=0x7fcfd006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 -- 172.21.15.70:0/2352822916 shutdown_connections 2024-04-22T01:15:12.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.050+0000 7fcfd5851640 1 -- 172.21.15.70:0/2352822916 wait complete. 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 Processor -- start 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- start start 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd006ed40 0x7fcfd0147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd00b3cd0 0x7fcfd0149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 0x7fcfd0151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd006f740 con 0x7fcfd015bb40 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd006f5c0 con 0x7fcfd00b3cd0 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd016ae60 con 0x7fcfd006ed40 2024-04-22T01:15:12.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 0x7fcfd0151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 0x7fcfd0151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54594/0 (socket says 172.21.15.70:54594) 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 -- 172.21.15.70:0/4248593825 learned_addr learned my addr 172.21.15.70:0/4248593825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfce7fc640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd00b3cd0 0x7fcfd0149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfceffd640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd006ed40 0x7fcfd0147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 -- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd006ed40 msgr2=0x7fcfd0147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd006ed40 0x7fcfd0147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 -- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd00b3cd0 msgr2=0x7fcfd0149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd00b3cd0 0x7fcfd0149c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 -- 172.21.15.70:0/4248593825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd00b3700 con 0x7fcfd015bb40 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfcf7fe640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 0x7fcfd0151e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009e30 tx=0x7fcfc800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8018020 con 0x7fcfd015bb40 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc8014480 con 0x7fcfd015bb40 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8014870 con 0x7fcfd015bb40 2024-04-22T01:15:12.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd014a670 con 0x7fcfd015bb40 2024-04-22T01:15:12.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd014c160 con 0x7fcfd015bb40 2024-04-22T01:15:12.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.054+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd0068210 con 0x7fcfd015bb40 2024-04-22T01:15:12.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.058+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcfc801b020 con 0x7fcfd015bb40 2024-04-22T01:15:12.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.058+0000 7fcfb7fff640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfbc0678c0 0x7fcfbc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.058+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcfc80873e0 con 0x7fcfd015bb40 2024-04-22T01:15:12.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.058+0000 7fcfceffd640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfbc0678c0 0x7fcfbc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.058+0000 7fcfceffd640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfbc0678c0 0x7fcfbc069d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00ae000 tx=0x7fcfc00a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.062+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcfc8022030 con 0x7fcfd015bb40 2024-04-22T01:15:12.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.222+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcfd006e040 con 0x7fcfd015bb40 2024-04-22T01:15:12.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.226+0000 7fcfb7fff640 1 -- 172.21.15.70:0/4248593825 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7fcfc8010040 con 0x7fcfd015bb40 2024-04-22T01:15:12.227 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:12.228 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:12.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfbc0678c0 msgr2=0x7fcfbc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfbc0678c0 0x7fcfbc069d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00ae000 tx=0x7fcfc00a7000 comp rx=0 tx=0).stop 2024-04-22T01:15:12.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 msgr2=0x7fcfd0151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 0x7fcfd0151e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009e30 tx=0x7fcfc800c490 comp rx=0 tx=0).stop 2024-04-22T01:15:12.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 shutdown_connections 2024-04-22T01:15:12.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfbc0678c0 0x7fcfbc069d80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfd015bb40 0x7fcfd0151e80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd00b3cd0 0x7fcfd0149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 --2- 172.21.15.70:0/4248593825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd006ed40 0x7fcfd0147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 >> 172.21.15.70:0/4248593825 conn(0x7fcfd006b1f0 msgr2=0x7fcfd00af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 shutdown_connections 2024-04-22T01:15:12.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.230+0000 7fcfd5851640 1 -- 172.21.15.70:0/4248593825 wait complete. 2024-04-22T01:15:12.247 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T01:15:12.247 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T01:15:12.247 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T01:15:12.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-22T01:15:12.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-22T01:15:12.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-22T01:15:12.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-22T01:15:12.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-22T01:15:12.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:15:12.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:15:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.426+0000 7f165da03640 1 Processor -- start 2024-04-22T01:15:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.426+0000 7f64400bb640 1 Processor -- start 2024-04-22T01:15:12.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.426+0000 7f165da03640 1 -- start start 2024-04-22T01:15:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f165da03640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1658162350 0x7f165809a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f165da03640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f165809acb0 con 0x7f16580c5280 2024-04-22T01:15:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f165da03640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f165809af90 con 0x7f165806c650 2024-04-22T01:15:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f165da03640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f16580ae7a0 con 0x7f1658162350 2024-04-22T01:15:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 Processor -- start 2024-04-22T01:15:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f16577fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f16580c5280 0x7f165816be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40754/0 (socket says 172.21.15.70:40754) 2024-04-22T01:15:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f16577fe640 1 -- 172.21.15.70:0/1789397958 learned_addr learned my addr 172.21.15.70:0/1789397958 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1656ffd640 1 --2- 172.21.15.70:0/1789397958 >> v2:172.21.15.98:3300/0 conn(0x7f1658162350 0x7f165809a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 Processor -- start 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.426+0000 7f64400bb640 1 -- start start 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f64400bb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64380b0310 0x7f64380b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f64400bb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f64380c3a40 0x7f64380c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f64400bb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f6438162f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f64400bb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f64380c43f0 con 0x7f64380c3a40 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f64400bb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6438163880 con 0x7f64380b0310 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f64400bb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f64380b0a70 con 0x7f6438162b50 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f6438162f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f6438162f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41056/0 (socket says 172.21.15.70:41056) 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 -- 172.21.15.70:0/3867909759 learned_addr learned my addr 172.21.15.70:0/3867909759 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643de57640 1 --2- 172.21.15.70:0/3867909759 >> v2:172.21.15.98:3300/0 conn(0x7f64380b0310 0x7f64380b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3309548691 0 0) 0x7f165809acb0 con 0x7f16580c5280 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1640003530 con 0x7f16580c5280 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643d656640 1 --2- 172.21.15.70:0/3867909759 >> v2:172.21.15.70:3300/0 conn(0x7f64380c3a40 0x7f64380c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 -- 172.21.15.70:0/3867909759 >> v2:172.21.15.98:3300/0 conn(0x7f64380b0310 msgr2=0x7f64380b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 --2- 172.21.15.70:0/3867909759 >> v2:172.21.15.98:3300/0 conn(0x7f64380b0310 0x7f64380b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 -- 172.21.15.70:0/3867909759 >> v2:172.21.15.70:3300/0 conn(0x7f64380c3a40 msgr2=0x7f64380c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 --2- 172.21.15.70:0/3867909759 >> v2:172.21.15.70:3300/0 conn(0x7f64380c3a40 0x7f64380c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f643e658640 1 -- 172.21.15.70:0/3867909759 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64380b0d50 con 0x7f6438162b50 2024-04-22T01:15:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482132998 0 0) 0x7f165809af90 con 0x7f165806c650 2024-04-22T01:15:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f165809acb0 con 0x7f165806c650 2024-04-22T01:15:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 -- start start 2024-04-22T01:15:12.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538153973 0 0) 0x7f1640003530 con 0x7f16580c5280 2024-04-22T01:15:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 >> v2:172.21.15.98:3300/0 conn(0x7f1658162350 msgr2=0x7f165809a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff538009630 0x7ff5380519a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff538007c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53800ad20 con 0x7ff538009a10 2024-04-22T01:15:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff538052f00 con 0x7ff538009630 2024-04-22T01:15:12.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5472e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff5380522f0 con 0x7ff538055940 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5462df640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff538009630 0x7ff5380519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5462df640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff538009630 0x7ff5380519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54610/0 (socket says 172.21.15.70:54610) 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5462df640 1 -- 172.21.15.70:0/4243677535 learned_addr learned my addr 172.21.15.70:0/4243677535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff546ae0640 1 --2- 172.21.15.70:0/4243677535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff538007c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 -- start start 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80340b3e10 0x7f803415d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f80340b41f0 0x7f803415a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f803415a1a0 con 0x7f803415ae80 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f803415dcd0 con 0x7f80340b41f0 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803359e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f803415df50 con 0x7f80340b3e10 2024-04-22T01:15:12.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f80340b41f0 0x7f803415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f80340b41f0 0x7f803415a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54612/0 (socket says 172.21.15.70:54612) 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 -- 172.21.15.70:0/492644413 learned_addr learned my addr 172.21.15.70:0/492644413 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803259c640 1 --2- 172.21.15.70:0/492644413 >> v2:172.21.15.98:3300/0 conn(0x7f80340b3e10 0x7f803415d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 --2- 172.21.15.70:0/1789397958 >> v2:172.21.15.98:3300/0 conn(0x7f1658162350 0x7f165809a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 >> v1:172.21.15.98:6789/0 conn(0x7f165806c650 legacy=0x7f16580c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165809b270 con 0x7f16580c5280 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1650001a00 con 0x7f16580c5280 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 >> v1:172.21.15.70:6789/0 conn(0x7f16580c5280 legacy=0x7f165816be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640003bf0 0x7f1640003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1640004810 0x7f1640004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16400053b0 0x7f164000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16580ae7a0 con 0x7f1640004810 2024-04-22T01:15:12.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f164000c2f0 con 0x7f1640003bf0 2024-04-22T01:15:12.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f164000c4f0 con 0x7f16400053b0 2024-04-22T01:15:12.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f4d7cffa640 1 Processor -- start 2024-04-22T01:15:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f1656ffd640 1 --2- 172.21.15.70:0/1789397958 >> v2:172.21.15.98:3300/0 conn(0x7f1658162350 0x7f165809a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 -- 172.21.15.70:0/492644413 >> v2:172.21.15.98:3300/0 conn(0x7f80340b3e10 msgr2=0x7f803415d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 --2- 172.21.15.70:0/492644413 >> v2:172.21.15.98:3300/0 conn(0x7f80340b3e10 0x7f803415d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 -- 172.21.15.70:0/492644413 >> v1:172.21.15.70:6789/0 conn(0x7f803415ae80 legacy=0x7f80340b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f8031d9b640 1 -- 172.21.15.70:0/492644413 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803415e230 con 0x7f80340b41f0 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5452dd640 1 -- 172.21.15.70:0/4243677535 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614038656 0 0) 0x7ff53800ad20 con 0x7ff538009a10 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff5452dd640 1 -- 172.21.15.70:0/4243677535 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff52c003530 con 0x7ff538009a10 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff546ae0640 1 -- 172.21.15.70:0/4243677535 >> v2:172.21.15.70:3300/0 conn(0x7ff538009630 msgr2=0x7ff5380519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff546ae0640 1 --2- 172.21.15.70:0/4243677535 >> v2:172.21.15.70:3300/0 conn(0x7ff538009630 0x7ff5380519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff546ae0640 1 -- 172.21.15.70:0/4243677535 >> v1:172.21.15.98:6789/0 conn(0x7ff538009a10 legacy=0x7ff538055230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7ff546ae0640 1 -- 172.21.15.70:0/4243677535 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5380525d0 con 0x7ff538055940 2024-04-22T01:15:12.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.430+0000 7f803259c640 1 --2- 172.21.15.70:0/492644413 >> v2:172.21.15.98:3300/0 conn(0x7f80340b3e10 0x7f803415d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:12.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16567fc640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1640004810 0x7f1640004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f1656ffd640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640003bf0 0x7f1640003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01d1fae640 1 Processor -- start 2024-04-22T01:15:12.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01d1fae640 1 -- start start 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d7cffa640 1 -- start start 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16567fc640 1 -- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16400053b0 msgr2=0x7f164000ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16567fc640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16400053b0 0x7f164000ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16567fc640 1 -- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640003bf0 msgr2=0x7f1640003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16567fc640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640003bf0 0x7f1640003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16567fc640 1 -- 172.21.15.70:0/1789397958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1640003530 con 0x7f1640004810 2024-04-22T01:15:12.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d7cffa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d780b16a0 0x7f4d780ad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d7cffa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 0x7f4d780adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d7cffa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d780cf390 con 0x7f4d780c3830 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d7cffa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4d780cf670 con 0x7f4d780b16a0 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d7cffa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4d780cf950 con 0x7f4d780b0780 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f643e658640 1 --2- 172.21.15.70:0/3867909759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f6438162f80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64380d0cb0 tx=0x7f643001c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4986422af4da18e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f643ce55640 1 -- 172.21.15.70:0/3867909759 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300a57b0 con 0x7f6438162b50 2024-04-22T01:15:12.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f643ce55640 1 -- 172.21.15.70:0/3867909759 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64300a5950 con 0x7f6438162b50 2024-04-22T01:15:12.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f643ce55640 1 -- 172.21.15.70:0/3867909759 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300a5d20 con 0x7f6438162b50 2024-04-22T01:15:12.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f16577fe640 1 -- 172.21.15.70:0/1789397958 reap_dead start 2024-04-22T01:15:12.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 0x7f4d780adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 0x7f4d780adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41078/0 (socket says 172.21.15.70:41078) 2024-04-22T01:15:12.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 -- 172.21.15.70:0/2390725876 learned_addr learned my addr 172.21.15.70:0/2390725876 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d6ffff640 1 --2- 172.21.15.70:0/2390725876 >> v2:172.21.15.98:3300/0 conn(0x7f4d780b16a0 0x7f4d780ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01d1fae640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f01c40543d0 0x7f01c40df770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01d1fae640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4052860 con 0x7f01c4053620 2024-04-22T01:15:12.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01d1fae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4052b40 con 0x7f01c4005ef0 2024-04-22T01:15:12.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01d1fae640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f01c4050b10 con 0x7f01c40543d0 2024-04-22T01:15:12.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01cbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f01c4005ef0 0x7f01c4006340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:40772/0 (socket says 172.21.15.70:40772) 2024-04-22T01:15:12.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01cbfff640 1 -- 172.21.15.70:0/3896157037 learned_addr learned my addr 172.21.15.70:0/3896157037 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7fd27ab53640 1 Processor -- start 2024-04-22T01:15:12.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 -- 172.21.15.70:0/3867909759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 msgr2=0x7f6438162f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 --2- 172.21.15.70:0/3867909759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f6438162f80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64380d0cb0 tx=0x7f643001c930 comp rx=0 tx=0).stop 2024-04-22T01:15:12.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7fd27ab53640 1 -- start start 2024-04-22T01:15:12.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 -- 172.21.15.70:0/3867909759 shutdown_connections 2024-04-22T01:15:12.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 --2- 172.21.15.70:0/3867909759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f6438162f80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 --2- 172.21.15.70:0/3867909759 >> v2:172.21.15.70:3300/0 conn(0x7f64380c3a40 0x7f64380c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 --2- 172.21.15.70:0/3867909759 >> v2:172.21.15.98:3300/0 conn(0x7f64380b0310 0x7f64380b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 -- 172.21.15.70:0/3867909759 >> 172.21.15.70:0/3867909759 conn(0x7f6438099a60 msgr2=0x7f6438099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 -- 172.21.15.70:0/3867909759 shutdown_connections 2024-04-22T01:15:12.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701259167 0 0) 0x7f01c4052b40 con 0x7f01c4005ef0 2024-04-22T01:15:12.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01bc003530 con 0x7f01c4005ef0 2024-04-22T01:15:12.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01caffd640 1 --2- 172.21.15.70:0/3896157037 >> v2:172.21.15.70:3300/0 conn(0x7f01c40543d0 0x7f01c40df770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f8031d9b640 1 --2- 172.21.15.70:0/492644413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80340b41f0 0x7f803415a940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8034072880 tx=0x7f802c01df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13043e23ab0675eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7ff546ae0640 1 --2- 172.21.15.70:0/4243677535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff538007c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff5300097c0 tx=0x7ff53001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=262cf5dc0ad0da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7ff5452dd640 1 -- 172.21.15.70:0/4243677535 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5300aa600 con 0x7ff538055940 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7ff5452dd640 1 -- 172.21.15.70:0/4243677535 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5300af070 con 0x7ff538055940 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7ff5452dd640 1 -- 172.21.15.70:0/4243677535 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5300aa9a0 con 0x7ff538055940 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3035869520 0 0) 0x7f01bc003530 con 0x7f01c4005ef0 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 >> v2:172.21.15.70:3300/0 conn(0x7f01c40543d0 msgr2=0x7f01c40df770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 --2- 172.21.15.70:0/3896157037 >> v2:172.21.15.70:3300/0 conn(0x7f01c40543d0 0x7f01c40df770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 >> v1:172.21.15.70:6789/0 conn(0x7f01c4053620 legacy=0x7f01c4053cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c4052e20 con 0x7f01c4005ef0 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f01c0001680 con 0x7f01c4005ef0 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 >> v1:172.21.15.98:6789/0 conn(0x7f01c4005ef0 legacy=0x7f01c4006340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc003a10 0x7f01bc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc004810 0x7f01bc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01bc0053d0 0x7f01bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc00c200 con 0x7f01bc0053d0 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc00c3b0 con 0x7f01bc004810 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f01bc00c5b0 con 0x7f01bc003a10 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01caffd640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc004810 0x7f01bc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cb7fe640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc003a10 0x7f01bc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f803159a640 1 -- 172.21.15.70:0/492644413 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c0b2070 con 0x7f80340b41f0 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f803159a640 1 -- 172.21.15.70:0/492644413 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f802c0ad400 con 0x7f80340b41f0 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f803159a640 1 -- 172.21.15.70:0/492644413 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c0ad7d0 con 0x7f80340b41f0 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 -- 172.21.15.70:0/492644413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80340b41f0 msgr2=0x7f803415a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 --2- 172.21.15.70:0/492644413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80340b41f0 0x7f803415a940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8034072880 tx=0x7f802c01df20 comp rx=0 tx=0).stop 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 -- 172.21.15.70:0/4243677535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 msgr2=0x7ff538007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 --2- 172.21.15.70:0/4243677535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff538007c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff5300097c0 tx=0x7ff53001c4a0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f16567fc640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1640004810 0x7f1640004c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1648007f50 tx=0x7f164801ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b6944d32ae0619 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cbfff640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01bc0053d0 0x7f01bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 -- 172.21.15.70:0/4243677535 shutdown_connections 2024-04-22T01:15:12.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 --2- 172.21.15.70:0/4243677535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff538007c60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 --2- 172.21.15.70:0/4243677535 >> v2:172.21.15.70:3300/0 conn(0x7ff538009630 0x7ff5380519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 -- 172.21.15.70:0/4243677535 >> 172.21.15.70:0/4243677535 conn(0x7ff538053760 msgr2=0x7ff538053b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cb7fe640 1 -- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc004810 msgr2=0x7f01bc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cb7fe640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc004810 0x7f01bc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cb7fe640 1 -- 172.21.15.70:0/3896157037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01bc0053d0 msgr2=0x7f01bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f64400bb640 1 -- 172.21.15.70:0/3867909759 wait complete. 2024-04-22T01:15:12.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cb7fe640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01bc0053d0 0x7f01bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cb7fe640 1 -- 172.21.15.70:0/3896157037 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01bc003530 con 0x7f01bc003a10 2024-04-22T01:15:12.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 Processor -- start 2024-04-22T01:15:12.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 -- 172.21.15.70:0/2390725876 >> v2:172.21.15.98:3300/0 conn(0x7f4d780b16a0 msgr2=0x7f4d780ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 --2- 172.21.15.70:0/2390725876 >> v2:172.21.15.98:3300/0 conn(0x7f4d780b16a0 0x7f4d780ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 -- 172.21.15.70:0/2390725876 >> v1:172.21.15.98:6789/0 conn(0x7f4d780c3830 legacy=0x7f4d780c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.434+0000 7f4d77fff640 1 -- 172.21.15.70:0/2390725876 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d780cfc30 con 0x7f4d780b0780 2024-04-22T01:15:12.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f4d6ffff640 1 --2- 172.21.15.70:0/2390725876 >> v2:172.21.15.98:3300/0 conn(0x7f4d780b16a0 0x7f4d780ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:12.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 -- 172.21.15.70:0/492644413 shutdown_connections 2024-04-22T01:15:12.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 --2- 172.21.15.70:0/492644413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80340b41f0 0x7f803415a940 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 --2- 172.21.15.70:0/492644413 >> v2:172.21.15.98:3300/0 conn(0x7f80340b3e10 0x7f803415d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 -- 172.21.15.70:0/492644413 >> 172.21.15.70:0/492644413 conn(0x7f803406b1f0 msgr2=0x7f803406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 -- 172.21.15.70:0/492644413 shutdown_connections 2024-04-22T01:15:12.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f4d77fff640 1 --2- 172.21.15.70:0/2390725876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 0x7f4d780adad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009e50 tx=0x7f4d7001f250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e89cdec58747f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd27ab53640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 0x7fd27415e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd279b51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 0x7fd27415e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd279b51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 0x7fd27415e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41100/0 (socket says 172.21.15.70:41100) 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd279b51640 1 -- 172.21.15.70:0/3698632700 learned_addr learned my addr 172.21.15.70:0/3698632700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd279350640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd27406e320 0x7fd27415a250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40778/0 (socket says 172.21.15.70:40778) 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648007b20 con 0x7f1640004810 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16480a9070 con 0x7f1640004810 2024-04-22T01:15:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f1655ffb640 1 -- 172.21.15.70:0/1789397958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16480a5640 con 0x7f1640004810 2024-04-22T01:15:12.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f165da03640 1 -- 172.21.15.70:0/1789397958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1640004810 msgr2=0x7f1640004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f165da03640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1640004810 0x7f1640004c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1648007f50 tx=0x7f164801ca80 comp rx=0 tx=0).stop 2024-04-22T01:15:12.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3698632700 >> v2:172.21.15.98:3300/0 conn(0x7fd2740cf780 0x7fd2740cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 -- 172.21.15.70:0/4243677535 shutdown_connections 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 -- start start 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64380b0310 0x7f64380b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64380c3a40 0x7f64380c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f64380b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643de57640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64380b0310 0x7f64380b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643de57640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64380b0310 0x7f64380b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54638/0 (socket says 172.21.15.70:54638) 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27415abb0 con 0x7fd27406e320 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd27a352640 1 --2- 172.21.15.70:0/3698632700 >> v2:172.21.15.98:3300/0 conn(0x7fd2740cf780 0x7fd2740cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd27415a960 con 0x7fd2740cf780 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 -- 172.21.15.70:0/492644413 wait complete. 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 Processor -- start 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f643806bb80 con 0x7f64380b0310 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f643806ba00 con 0x7f64380c3a40 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64380c3a40 0x7f64380c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f64400bb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f643806bd00 con 0x7f6438162b50 2024-04-22T01:15:12.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64380c3a40 0x7f64380c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42874/0 (socket says 172.21.15.70:42874) 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 -- 172.21.15.70:0/3442941250 learned_addr learned my addr 172.21.15.70:0/3442941250 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643e658640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f64380b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd27415ae90 con 0x7fd2740b4110 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 -- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 msgr2=0x7f64380b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f64380b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 -- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64380b0310 msgr2=0x7f64380b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64380b0310 0x7f64380b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 -- 172.21.15.70:0/3442941250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64380b49d0 con 0x7f64380c3a40 2024-04-22T01:15:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643e658640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6438162b50 0x7f64380b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:12.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7ff5472e1640 1 -- 172.21.15.70:0/4243677535 wait complete. 2024-04-22T01:15:12.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f165da03640 1 -- 172.21.15.70:0/1789397958 shutdown_connections 2024-04-22T01:15:12.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f165da03640 1 --2- 172.21.15.70:0/1789397958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1640004810 0x7f1640004c90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f165da03640 1 -- 172.21.15.70:0/1789397958 >> 172.21.15.70:0/1789397958 conn(0x7f16580b3690 msgr2=0x7f16580b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f01cbfff640 1 -- 172.21.15.70:0/3896157037 reap_dead start 2024-04-22T01:15:12.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f643d656640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64380c3a40 0x7f64380c6ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f642800d830 tx=0x7f642800dcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 Processor -- start 2024-04-22T01:15:12.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd279b51640 1 -- 172.21.15.70:0/3698632700 >> v2:172.21.15.98:3300/0 conn(0x7fd2740cf780 msgr2=0x7fd2740cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd279b51640 1 --2- 172.21.15.70:0/3698632700 >> v2:172.21.15.98:3300/0 conn(0x7fd2740cf780 0x7fd2740cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd279b51640 1 -- 172.21.15.70:0/3698632700 >> v1:172.21.15.70:6789/0 conn(0x7fd27406e320 legacy=0x7fd27415a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 -- 172.21.15.70:0/1789397958 shutdown_connections 2024-04-22T01:15:12.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd279b51640 1 -- 172.21.15.70:0/3698632700 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27415d8f0 con 0x7fd2740b4110 2024-04-22T01:15:12.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd27a352640 1 --2- 172.21.15.70:0/3698632700 >> v2:172.21.15.98:3300/0 conn(0x7fd2740cf780 0x7fd2740cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 -- 172.21.15.70:0/1789397958 wait complete. 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 Processor -- start 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01cb7fe640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc003a10 0x7f01bc003e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01b8001780 tx=0x7f01b801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699444ee6423acfc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b80013a0 con 0x7f01bc003a10 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b80052f0 con 0x7f01bc003a10 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01ca7fc640 1 -- 172.21.15.70:0/3896157037 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8003840 con 0x7f01bc003a10 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 -- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc003a10 msgr2=0x7f01bc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc003a10 0x7f01bc003e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01b8001780 tx=0x7f01b801c440 comp rx=0 tx=0).stop 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.438+0000 7f803359e640 1 -- start start 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80340b3e10 0x7f8034157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b41f0 0x7f80341fb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f803415ae80 0x7f803414c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80340b33b0 con 0x7f803415ae80 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80340b3230 con 0x7f80340b41f0 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8034166b30 con 0x7f80340b3e10 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8031d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b41f0 0x7f80341fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8031d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b41f0 0x7f80341fb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42890/0 (socket says 172.21.15.70:42890) 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8031d9b640 1 -- 172.21.15.70:0/3509893251 learned_addr learned my addr 172.21.15.70:0/3509893251 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f803415ae80 0x7f803414c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803259c640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80340b3e10 0x7f8034157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 -- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80340b3e10 msgr2=0x7f8034157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80340b3e10 0x7f8034157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 -- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b41f0 msgr2=0x7f80341fb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b41f0 0x7f80341fb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 -- 172.21.15.70:0/3509893251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8034158370 con 0x7f803415ae80 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8032d9d640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f803415ae80 0x7f803414c100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f802800c3c0 tx=0x7f802800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642800b6c0 con 0x7f64380c3a40 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6428019070 con 0x7f64380c3a40 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642800ba60 con 0x7f64380c3a40 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f64400bb640 1 -- 172.21.15.70:0/3442941250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64380b4cb0 con 0x7f64380c3a40 2024-04-22T01:15:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f64400bb640 1 -- 172.21.15.70:0/3442941250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64380b4f90 con 0x7f64380c3a40 2024-04-22T01:15:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d76ffd640 1 -- 172.21.15.70:0/2390725876 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d700b4070 con 0x7f4d780b0780 2024-04-22T01:15:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 -- start start 2024-04-22T01:15:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff538009630 0x7ff5380e19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff538009a10 0x7ff53804d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff5381898a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff538052060 con 0x7ff538009630 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff538051ee0 con 0x7ff538009a10 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff5380089b0 con 0x7ff538055940 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff538009630 0x7ff5380e19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff545ade640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff538009a10 0x7ff53804d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff538009630 0x7ff5380e19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54652/0 (socket says 172.21.15.70:54652) 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 -- 172.21.15.70:0/1699560582 learned_addr learned my addr 172.21.15.70:0/1699560582 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 -- 172.21.15.70:0/1699560582 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 msgr2=0x7ff5381898a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff538055940 0x7ff5381898a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 -- 172.21.15.70:0/1699560582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff538009a10 msgr2=0x7ff53804d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff538009a10 0x7ff53804d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 -- 172.21.15.70:0/1699560582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff538189de0 con 0x7ff538009630 2024-04-22T01:15:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5462df640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff538009630 0x7ff5380e19c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00e7f0 tx=0x7ff53c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f64400bb640 1 -- 172.21.15.70:0/3442941250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f64380c49f0 con 0x7f64380c3a40 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd279b51640 1 --2- 172.21.15.70:0/3698632700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 0x7fd27415e0f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2700091e0 tx=0x7fd27001e710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38909260d4b2b05c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd278b4f640 1 -- 172.21.15.70:0/3698632700 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700b4070 con 0x7fd2740b4110 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd278b4f640 1 -- 172.21.15.70:0/3698632700 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2700a7740 con 0x7fd2740b4110 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7fd278b4f640 1 -- 172.21.15.70:0/3698632700 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700a7b10 con 0x7fd2740b4110 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 msgr2=0x7fd27415e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3698632700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 0x7fd27415e0f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2700091e0 tx=0x7fd27001e710 comp rx=0 tx=0).stop 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 shutdown_connections 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3698632700 >> v2:172.21.15.98:3300/0 conn(0x7fd2740cf780 0x7fd2740cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3698632700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2740b4110 0x7fd27415e0f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 >> 172.21.15.70:0/3698632700 conn(0x7fd27406b1f0 msgr2=0x7fd27406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c039070 con 0x7ff538009630 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53c00ee40 con 0x7ff538009630 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00cb80 con 0x7ff538009630 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 -- 172.21.15.70:0/1699560582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5380e0920 con 0x7ff538009630 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7ff5472e1640 1 -- 172.21.15.70:0/1699560582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5380e0ee0 con 0x7ff538009630 2024-04-22T01:15:12.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 -- start start 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640004810 0x7f1658201160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16580b6b90 0x7f16580b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1658207170 0x7f16582075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f165809aa60 con 0x7f16580b6b90 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f165809a8e0 con 0x7f1640004810 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f165da03640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16580c4b70 con 0x7f1658207170 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640004810 0x7f1658201160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640004810 0x7f1658201160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42908/0 (socket says 172.21.15.70:42908) 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 -- 172.21.15.70:0/2744625825 learned_addr learned my addr 172.21.15.70:0/2744625825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 -- 172.21.15.70:0/2744625825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1658207170 msgr2=0x7f16582075f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1658207170 0x7f16582075f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 -- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16580b6b90 msgr2=0x7f16580b3d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16580b6b90 0x7f16580b3d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 -- 172.21.15.70:0/2744625825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16580b5870 con 0x7f1640004810 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 -- 172.21.15.70:0/3896157037 shutdown_connections 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 --2- 172.21.15.70:0/3896157037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc003a10 0x7f01bc003e30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 -- 172.21.15.70:0/3896157037 >> 172.21.15.70:0/3896157037 conn(0x7f01c4009fc0 msgr2=0x7f01c400a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 -- 172.21.15.70:0/3896157037 shutdown_connections 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f01d1fae640 1 -- 172.21.15.70:0/3896157037 wait complete. 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 Processor -- start 2024-04-22T01:15:12.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80280052d0 con 0x7f803415ae80 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8028005470 con 0x7f803415ae80 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028007260 con 0x7f803415ae80 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8034158650 con 0x7f803415ae80 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8034158c10 con 0x7f803415ae80 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d76ffd640 1 -- 172.21.15.70:0/2390725876 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d700af400 con 0x7f4d780b0780 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 -- 172.21.15.70:0/2390725876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 msgr2=0x7f4d780adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 --2- 172.21.15.70:0/2390725876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 0x7f4d780adad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009e50 tx=0x7f4d7001f250 comp rx=0 tx=0).stop 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 -- 172.21.15.70:0/2390725876 shutdown_connections 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 --2- 172.21.15.70:0/2390725876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d780b0780 0x7f4d780adad0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 --2- 172.21.15.70:0/2390725876 >> v2:172.21.15.98:3300/0 conn(0x7f4d780b16a0 0x7f4d780ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 -- 172.21.15.70:0/2390725876 >> 172.21.15.70:0/2390725876 conn(0x7f4d780d0860 msgr2=0x7f4d780d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 -- 172.21.15.70:0/2390725876 shutdown_connections 2024-04-22T01:15:12.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 shutdown_connections 2024-04-22T01:15:12.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f1656ffd640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640004810 0x7f1658201160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f164c00fa20 tx=0x7f164c00fee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f989c8d9640 1 Processor -- start 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c00dcd0 con 0x7f1640004810 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f164c00de70 con 0x7f1640004810 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c013640 con 0x7f1640004810 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16580b5af0 con 0x7f1640004810 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16580b60b0 con 0x7f1640004810 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f642801e460 con 0x7f64380c3a40 2024-04-22T01:15:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f642e7fc640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f641c0678c0 0x7f641c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.442+0000 7f4d7cffa640 1 -- 172.21.15.70:0/2390725876 wait complete. 2024-04-22T01:15:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 Processor -- start 2024-04-22T01:15:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f989c8d9640 1 -- start start 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 -- start start 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc003a10 0x7f01c40ed5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01c40ca010 0x7f01c4047d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c40d2520 0x7f01c40d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01c40df770 con 0x7f01c40d2520 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01c4052610 con 0x7f01bc003a10 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f01c40df8f0 con 0x7f01c40ca010 2024-04-22T01:15:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff53c03c070 con 0x7ff538009630 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01cb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc003a10 0x7f01c40ed5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01c40ca010 0x7f01c4047d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6428086fe0 con 0x7f64380c3a40 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f642e7fc640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7f641c071040 0x7f641c073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 --> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f642801fe50 con 0x7f641c071040 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f64280873d0 con 0x7f64380c3a40 2024-04-22T01:15:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01c40ca010 0x7f01c4047d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41124/0 (socket says 172.21.15.70:41124) 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 -- 172.21.15.70:0/4194405909 learned_addr learned my addr 172.21.15.70:0/4194405909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8034159010 con 0x7f803415ae80 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8028007400 con 0x7f803415ae80 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8022ffd640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8014067990 0x7f8014069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8028085380 con 0x7f803415ae80 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8022ffd640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f8014071090 0x7f80140734d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 --> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8028009e50 con 0x7f8014071090 2024-04-22T01:15:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7ff536ffd640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff524067990 0x7ff524069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff53c0a8180 con 0x7ff538009630 2024-04-22T01:15:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f803259c640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8014067990 0x7f8014069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8031d9b640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f8014071090 0x7f80140734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f643de57640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f641c0678c0 0x7f641c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f643e658640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7f641c071040 0x7f641c073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7ff545ade640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff524067990 0x7ff524069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 -- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc003a10 msgr2=0x7f01c40ed5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc003a10 0x7f01c40ed5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 -- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c40d2520 msgr2=0x7f01c40d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c40d2520 0x7f01c40d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 -- 172.21.15.70:0/4194405909 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c40edc60 con 0x7f01c40ca010 2024-04-22T01:15:12.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f989c8d9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98980cff00 0x7f98981630a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f989c8d9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f98980cef00 0x7f98980c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f989c8d9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98981635e0 con 0x7f98980c3d30 2024-04-22T01:15:12.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f643de57640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f641c0678c0 0x7f641c069d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6434009b60 tx=0x7f6434009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 -- start start 2024-04-22T01:15:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f989c8d9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f98980c4140 con 0x7f98980cef00 2024-04-22T01:15:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f989c8d9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f98980d02e0 con 0x7f98980cff00 2024-04-22T01:15:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01caffd640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01c40ca010 0x7f01c4047d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f01cc06afd0 tx=0x7f01cc068cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d780b16a0 0x7f4d781eb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d780c3830 0x7f4d781ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d78200120 0x7f4d780c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d780ae010 con 0x7f4d780c3830 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d780cf090 con 0x7f4d780b16a0 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d7cffa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4d780ae190 con 0x7f4d78200120 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 -- 172.21.15.70:0/3698632700 wait complete. 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 Processor -- start 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d780b16a0 0x7f4d781eb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d780b16a0 0x7f4d781eb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42920/0 (socket says 172.21.15.70:42920) 2024-04-22T01:15:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 -- 172.21.15.70:0/1177179860 learned_addr learned my addr 172.21.15.70:0/1177179860 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc073870 con 0x7f01c40ca010 2024-04-22T01:15:12.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d77fff640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d78200120 0x7f4d780c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f8031d9b640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f8014071090 0x7f80140734d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f643e658640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7f641c071040 0x7f641c073480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 -- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d78200120 msgr2=0x7f4d780c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d78200120 0x7f4d780c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 -- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d780c3830 msgr2=0x7f4d781ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d780c3830 0x7f4d781ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f4d777fe640 1 -- 172.21.15.70:0/1177179860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d78200670 con 0x7f4d780b16a0 2024-04-22T01:15:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7ff5472e1640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7ff5380dffc0 0x7ff5380e1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7ff5472e1640 1 -- 172.21.15.70:0/1699560582 --> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5380396e0 con 0x7ff5380dffc0 2024-04-22T01:15:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7ff545ade640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff524067990 0x7ff524069e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff540069920 tx=0x7ff540072040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9895d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f98980cef00 0x7f98980c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9896d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f98980c3d30 0x7f98980cd5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40794/0 (socket says 172.21.15.70:40794) 2024-04-22T01:15:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9896d9d640 1 -- 172.21.15.70:0/2051159921 learned_addr learned my addr 172.21.15.70:0/2051159921 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1638001040 con 0x7f1640004810 2024-04-22T01:15:12.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f803259c640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8014067990 0x7f8014069e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f80341f9150 tx=0x7f8024008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7ff546ae0640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7ff5380dffc0 0x7ff5380e1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c40edf40 con 0x7f01c40ca010 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c40ee500 con 0x7f01c40ca010 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc0744a0 con 0x7f01c40ca010 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f01c40052d0 con 0x7f01c40ca010 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d777fe640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d780b16a0 0x7f4d781eb6d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c420 tx=0x7f4d6800c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.446+0000 7fd27ab53640 1 -- start start 2024-04-22T01:15:12.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd27406e320 0x7fd2741f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f802808c050 con 0x7f803415ae80 2024-04-22T01:15:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2740b4110 0x7fd274153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d680052d0 con 0x7f4d780b16a0 2024-04-22T01:15:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d68005470 con 0x7f4d780b16a0 2024-04-22T01:15:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9895d9b640 1 -- 172.21.15.70:0/2051159921 >> v2:172.21.15.98:3300/0 conn(0x7f98980cff00 msgr2=0x7f98981630a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9895d9b640 1 --2- 172.21.15.70:0/2051159921 >> v2:172.21.15.98:3300/0 conn(0x7f98980cff00 0x7f98981630a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7ff546ae0640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7ff5380dffc0 0x7ff5380e1120 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd27406e320 0x7fd2741f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd27406e320 0x7fd2741f89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41138/0 (socket says 172.21.15.70:41138) 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d78200950 con 0x7f4d780b16a0 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d78200f10 con 0x7f4d780b16a0 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2740cf780 0x7fd2741fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd27415e7b0 con 0x7fd2740b4110 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd27415e630 con 0x7fd2740cf780 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd27406f980 con 0x7fd27406e320 2024-04-22T01:15:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 -- 172.21.15.70:0/3549633991 learned_addr learned my addr 172.21.15.70:0/3549633991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc074850 con 0x7f01c40ca010 2024-04-22T01:15:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 -- 172.21.15.70:0/3549633991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2740cf780 msgr2=0x7fd2741fbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2740cf780 0x7fd2741fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 -- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2740b4110 msgr2=0x7fd274153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279350640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2740b4110 0x7fd274153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== osd.1 v2:172.21.15.70:6800/2050830481 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5380396e0 con 0x7ff5380dffc0 2024-04-22T01:15:12.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68007260 con 0x7f4d780b16a0 2024-04-22T01:15:12.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2740b4110 0x7fd274153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 -- 172.21.15.70:0/3549633991 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2741553c0 con 0x7fd27406e320 2024-04-22T01:15:12.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279350640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2740b4110 0x7fd274153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:12.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d5c001040 con 0x7f4d780b16a0 2024-04-22T01:15:12.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01cc0749f0 con 0x7f01c40ca010 2024-04-22T01:15:12.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== osd.5 v2:172.21.15.98:6816/3651416447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8028009e50 con 0x7f8014071090 2024-04-22T01:15:12.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd279b51640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd27406e320 0x7fd2741f89d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd27001cdc0 tx=0x7fd27001b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700bd070 con 0x7fd27406e320 2024-04-22T01:15:12.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01b0067a90 0x7f01b0069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2700a7540 con 0x7fd27406e320 2024-04-22T01:15:12.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2740b2c30 con 0x7fd27406e320 2024-04-22T01:15:12.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2740b31f0 con 0x7fd27406e320 2024-04-22T01:15:12.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700b4030 con 0x7fd27406e320 2024-04-22T01:15:12.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9895d9b640 1 -- 172.21.15.70:0/2051159921 >> v1:172.21.15.70:6789/0 conn(0x7f98980c3d30 legacy=0x7f98980cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f989559a640 1 -- 172.21.15.70:0/2051159921 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617227102 0 0) 0x7f98900014a0 con 0x7f98980c3d30 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f9895d9b640 1 -- 172.21.15.70:0/2051159921 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9898166b50 con 0x7f98980cef00 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f164c0137e0 con 0x7f1640004810 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f163ffff640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16440679c0 0x7f1644069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f164c0873a0 con 0x7f1640004810 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f163ffff640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7f1644071130 0x7f1644073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 --> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f164c001d40 con 0x7f1644071130 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f164c087790 con 0x7f1640004810 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f16577fe640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7f1644071130 0x7f1644073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f16567fc640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16440679c0 0x7f1644069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f01cc100d10 con 0x7f01c40ca010 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01cb7fe640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01b0067a90 0x7f01b0069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7f01b0071180 0x7f01b00735c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 --> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01cc0744a0 con 0x7f01b0071180 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f01cc1011a0 con 0x7f01c40ca010 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f01cbfff640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7f01b0071180 0x7f01b00735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== osd.6 v2:172.21.15.98:6800/3535628973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f642801fe50 con 0x7f641c071040 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4d68007400 con 0x7f4d780b16a0 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.450+0000 7f4d74ff9640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d580679c0 0x7f4d58069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4d68085460 con 0x7f4d780b16a0 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d6ffff640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d580679c0 0x7f4d58069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d74ff9640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f4d580710c0 0x7f4d58073500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 --> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d68009e50 con 0x7f4d580710c0 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4d68051890 con 0x7f4d780b16a0 2024-04-22T01:15:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f01cbfff640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7f01b0071180 0x7f01b00735c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f9895d9b640 1 --2- 172.21.15.70:0/2051159921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980cef00 0x7f98980c37f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9898072e40 tx=0x7f988801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1005070f36d71e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d77fff640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f4d580710c0 0x7f4d58073500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd25c001040 con 0x7fd27406e320 2024-04-22T01:15:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989559a640 1 -- 172.21.15.70:0/2051159921 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98880b1070 con 0x7f98980cef00 2024-04-22T01:15:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f16577fe640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7f1644071130 0x7f1644073570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f16567fc640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16440679c0 0x7f1644069e80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f16480ab000 tx=0x7f16480b0040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d6ffff640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d580679c0 0x7f4d58069e80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d780b5830 tx=0x7f4d60008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f4d77fff640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f4d580710c0 0x7f4d58073500 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd2700c0070 con 0x7fd27406e320 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989559a640 1 -- 172.21.15.70:0/2051159921 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98880ac620 con 0x7f98980cef00 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989c8d9640 1 -- 172.21.15.70:0/2051159921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980cef00 msgr2=0x7f98980c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989559a640 1 -- 172.21.15.70:0/2051159921 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98880b1070 con 0x7f98980cef00 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989c8d9640 1 --2- 172.21.15.70:0/2051159921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980cef00 0x7f98980c37f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9898072e40 tx=0x7f988801c400 comp rx=0 tx=0).stop 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989c8d9640 1 -- 172.21.15.70:0/2051159921 shutdown_connections 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989c8d9640 1 --2- 172.21.15.70:0/2051159921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980cef00 0x7f98980c37f0 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9898072e40 tx=0x7f988801c400 comp rx=0 tx=0).stop 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989c8d9640 1 --2- 172.21.15.70:0/2051159921 >> v2:172.21.15.98:3300/0 conn(0x7f98980cff00 0x7f98981630a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f989c8d9640 1 -- 172.21.15.70:0/2051159921 >> 172.21.15.70:0/2051159921 conn(0x7f98980b2350 msgr2=0x7f98980b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== osd.3 v2:172.21.15.70:6808/1842545244 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f164c001d40 con 0x7f1644071130 2024-04-22T01:15:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f01cb7fe640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01b0067a90 0x7f01b0069f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f01b8001750 tx=0x7f01b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== osd.4 v2:172.21.15.98:6804/3056066255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01cc0744a0 con 0x7f01b0071180 2024-04-22T01:15:12.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd26a7fc640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2580678c0 0x7fd258069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd270120140 con 0x7fd27406e320 2024-04-22T01:15:12.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd279350640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2580678c0 0x7fd258069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd26a7fc640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7fd258071040 0x7fd258073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 --> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2700b1ec0 con 0x7fd258071040 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd27a352640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7fd258071040 0x7fd258073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.454+0000 7fd279350640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2580678c0 0x7fd258069d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd26400afd0 tx=0x7fd26400c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.458+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd270120530 con 0x7fd27406e320 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.458+0000 7fd27a352640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7fd258071040 0x7fd258073480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.458+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== osd.0 v2:172.21.15.70:6809/3468450268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2700b1ec0 con 0x7fd258071040 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 -- 172.21.15.70:0/2051159921 shutdown_connections 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 -- 172.21.15.70:0/2051159921 wait complete. 2024-04-22T01:15:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 Processor -- start 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 -- start start 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980c3d30 0x7f9898161090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98980cff00 0x7f9898158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98981659d0 0x7f9898159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f98980afd80 con 0x7f98980c3d30 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9898166870 con 0x7f98980cff00 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f989c8d9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f98980cf370 con 0x7f98981659d0 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f9896d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98981659d0 0x7f9898159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f9896d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98981659d0 0x7f9898159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41140/0 (socket says 172.21.15.70:41140) 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.462+0000 7f9896d9d640 1 -- 172.21.15.70:0/3680632218 learned_addr learned my addr 172.21.15.70:0/3680632218 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9895d9b640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98980cff00 0x7f9898158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== osd.7 v2:172.21.15.98:6824/2897852499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d68009e50 con 0x7f4d580710c0 2024-04-22T01:15:12.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9896d9d640 1 -- 172.21.15.70:0/3680632218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98980cff00 msgr2=0x7f9898158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9896d9d640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98980cff00 0x7f9898158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9896d9d640 1 -- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980c3d30 msgr2=0x7f9898161090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9896d9d640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98980c3d30 0x7f9898161090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9896d9d640 1 -- 172.21.15.70:0/3680632218 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98980aeed0 con 0x7f98981659d0 2024-04-22T01:15:12.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 --> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f803404f510 con 0x7f8014071090 2024-04-22T01:15:12.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9896d9d640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98981659d0 0x7f9898159340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f989000c7f0 tx=0x7f989000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.466+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890018070 con 0x7f98981659d0 2024-04-22T01:15:12.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f989000a640 con 0x7f98981659d0 2024-04-22T01:15:12.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989000aa30 con 0x7f98981659d0 2024-04-22T01:15:12.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f8022ffd640 1 -- 172.21.15.70:0/3509893251 <== osd.5 v2:172.21.15.98:6816/3651416447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f803404f510 con 0x7f8014071090 2024-04-22T01:15:12.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98980af1b0 con 0x7f98981659d0 2024-04-22T01:15:12.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f989815f950 con 0x7f98981659d0 2024-04-22T01:15:12.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f8014071090 msgr2=0x7f80140734d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f8014071090 0x7f80140734d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8014067990 msgr2=0x7f8014069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8014067990 0x7f8014069e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f80341f9150 tx=0x7f8024008040 comp rx=0 tx=0).stop 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f803415ae80 msgr2=0x7f803414c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 --2- 172.21.15.70:0/3509893251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f803415ae80 0x7f803414c100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f802800c3c0 tx=0x7f802800c880 comp rx=0 tx=0).stop 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f989000abd0 con 0x7f98981659d0 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f9886ffd640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f98780678c0 0x7f9878069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.474+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9890085590 con 0x7f98981659d0 2024-04-22T01:15:12.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f8032d9d640 1 -- 172.21.15.70:0/3509893251 reap_dead start 2024-04-22T01:15:12.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 shutdown_connections 2024-04-22T01:15:12.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 >> 172.21.15.70:0/3509893251 conn(0x7f803406b1f0 msgr2=0x7f8034156a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 shutdown_connections 2024-04-22T01:15:12.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.470+0000 7f803359e640 1 -- 172.21.15.70:0/3509893251 wait complete. 2024-04-22T01:15:12.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.474+0000 7f989659c640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f98780678c0 0x7f9878069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.474+0000 7f989659c640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f98780678c0 0x7f9878069d80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f988c009690 tx=0x7f988c006c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7f989c8d9640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f989815d0a0 0x7f989815d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 --> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f989815fb90 con 0x7f989815d0a0 2024-04-22T01:15:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7f9895d9b640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f989815d0a0 0x7f989815d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7f64400bb640 1 -- 172.21.15.70:0/3442941250 --> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f643804f510 con 0x7f641c071040 2024-04-22T01:15:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7ff5472e1640 1 -- 172.21.15.70:0/1699560582 --> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff53803cc60 con 0x7ff5380dffc0 2024-04-22T01:15:12.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7f9895d9b640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f989815d0a0 0x7f989815d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7f642e7fc640 1 -- 172.21.15.70:0/3442941250 <== osd.6 v2:172.21.15.98:6800/3535628973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f643804f510 con 0x7f641c071040 2024-04-22T01:15:12.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.478+0000 7ff536ffd640 1 -- 172.21.15.70:0/1699560582 <== osd.1 v2:172.21.15.70:6800/2050830481 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff53803cc60 con 0x7ff5380dffc0 2024-04-22T01:15:12.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== osd.2 v2:172.21.15.70:6824/3866658519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9888004240 con 0x7f989815d0a0 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7f641c071040 msgr2=0x7f641c073480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7f641c071040 0x7f641c073480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f641c0678c0 msgr2=0x7f641c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f641c0678c0 0x7f641c069d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6434009b60 tx=0x7f6434009190 comp rx=0 tx=0).stop 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64380c3a40 msgr2=0x7f64380c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 --2- 172.21.15.70:0/3442941250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64380c3a40 0x7f64380c6ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f642800d830 tx=0x7f642800dcf0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f643e658640 1 -- 172.21.15.70:0/3442941250 reap_dead start 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 shutdown_connections 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 >> 172.21.15.70:0/3442941250 conn(0x7f6438099a60 msgr2=0x7f643806ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 shutdown_connections 2024-04-22T01:15:12.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f641bfff640 1 -- 172.21.15.70:0/3442941250 wait complete. 2024-04-22T01:15:12.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 --> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1638002d60 con 0x7f1644071130 2024-04-22T01:15:12.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7ff5380dffc0 msgr2=0x7ff5380e1120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7ff534ff9640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7ff5380dffc0 0x7ff5380e1120 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff524067990 msgr2=0x7ff524069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7ff534ff9640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff524067990 0x7ff524069e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff540069920 tx=0x7ff540072040 comp rx=0 tx=0).stop 2024-04-22T01:15:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff538009630 msgr2=0x7ff5380e19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f163ffff640 1 -- 172.21.15.70:0/2744625825 <== osd.3 v2:172.21.15.70:6808/1842545244 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1638002d60 con 0x7f1644071130 2024-04-22T01:15:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7ff534ff9640 1 --2- 172.21.15.70:0/1699560582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff538009630 0x7ff5380e19c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00e7f0 tx=0x7ff53c00ebf0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7f1644071130 msgr2=0x7f1644073570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.482+0000 7f165da03640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7f1644071130 0x7f1644073570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7ff546ae0640 1 -- 172.21.15.70:0/1699560582 reap_dead start 2024-04-22T01:15:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 shutdown_connections 2024-04-22T01:15:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 >> 172.21.15.70:0/1699560582 conn(0x7ff538053760 msgr2=0x7ff5380df8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 --> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f01c40052d0 con 0x7f01b0071180 2024-04-22T01:15:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 --> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd25c002de0 con 0x7fd258071040 2024-04-22T01:15:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01b7fff640 1 -- 172.21.15.70:0/4194405909 <== osd.4 v2:172.21.15.98:6804/3056066255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f01c40052d0 con 0x7f01b0071180 2024-04-22T01:15:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16440679c0 msgr2=0x7f1644069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16440679c0 0x7f1644069e80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f16480ab000 tx=0x7f16480b0040 comp rx=0 tx=0).stop 2024-04-22T01:15:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640004810 msgr2=0x7f1658201160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 --2- 172.21.15.70:0/2744625825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1640004810 0x7f1658201160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f164c00fa20 tx=0x7f164c00fee0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f16577fe640 1 -- 172.21.15.70:0/2744625825 reap_dead start 2024-04-22T01:15:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 shutdown_connections 2024-04-22T01:15:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 >> 172.21.15.70:0/2744625825 conn(0x7f16580b3690 msgr2=0x7f165806fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 shutdown_connections 2024-04-22T01:15:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f165da03640 1 -- 172.21.15.70:0/2744625825 wait complete. 2024-04-22T01:15:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 shutdown_connections 2024-04-22T01:15:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7ff534ff9640 1 -- 172.21.15.70:0/1699560582 wait complete. 2024-04-22T01:15:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7fd26a7fc640 1 -- 172.21.15.70:0/3549633991 <== osd.0 v2:172.21.15.70:6809/3468450268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd25c002de0 con 0x7fd258071040 2024-04-22T01:15:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7f01b0071180 msgr2=0x7f01b00735c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7f01b0071180 0x7f01b00735c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 --> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d5c002d60 con 0x7f4d580710c0 2024-04-22T01:15:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01b0067a90 msgr2=0x7f01b0069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01b0067a90 0x7f01b0069f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f01b8001750 tx=0x7f01b8009040 comp rx=0 tx=0).stop 2024-04-22T01:15:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01c40ca010 msgr2=0x7f01c4047d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 --2- 172.21.15.70:0/4194405909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01c40ca010 0x7f01c4047d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f01cc06afd0 tx=0x7f01cc068cf0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01cbfff640 1 -- 172.21.15.70:0/4194405909 reap_dead start 2024-04-22T01:15:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 shutdown_connections 2024-04-22T01:15:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 >> 172.21.15.70:0/4194405909 conn(0x7f01c4009fc0 msgr2=0x7f01c40d0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 shutdown_connections 2024-04-22T01:15:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.486+0000 7f01d1fae640 1 -- 172.21.15.70:0/4194405909 wait complete. 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7fd258071040 msgr2=0x7fd258073480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7fd258071040 0x7fd258073480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2580678c0 msgr2=0x7fd258069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2580678c0 0x7fd258069d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd26400afd0 tx=0x7fd26400c040 comp rx=0 tx=0).stop 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd27406e320 msgr2=0x7fd2741f89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 --2- 172.21.15.70:0/3549633991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd27406e320 0x7fd2741f89d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd27001cdc0 tx=0x7fd27001b4b0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27a352640 1 -- 172.21.15.70:0/3549633991 reap_dead start 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 shutdown_connections 2024-04-22T01:15:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 >> 172.21.15.70:0/3549633991 conn(0x7fd27406b1f0 msgr2=0x7fd2740b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 shutdown_connections 2024-04-22T01:15:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7fd27ab53640 1 -- 172.21.15.70:0/3549633991 wait complete. 2024-04-22T01:15:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d74ff9640 1 -- 172.21.15.70:0/1177179860 <== osd.7 v2:172.21.15.98:6824/2897852499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4d5c002d60 con 0x7f4d580710c0 2024-04-22T01:15:12.530 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:12.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T01:15:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f4d580710c0 msgr2=0x7f4d58073500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f4d580710c0 0x7f4d58073500 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d580679c0 msgr2=0x7f4d58069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d580679c0 0x7f4d58069e80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d780b5830 tx=0x7f4d60008040 comp rx=0 tx=0).stop 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d780b16a0 msgr2=0x7f4d781eb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 --2- 172.21.15.70:0/1177179860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d780b16a0 0x7f4d781eb6d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c420 tx=0x7f4d6800c8e0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d77fff640 1 -- 172.21.15.70:0/1177179860 reap_dead start 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 shutdown_connections 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 >> 172.21.15.70:0/1177179860 conn(0x7f4d780d0860 msgr2=0x7f4d7806f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 shutdown_connections 2024-04-22T01:15:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.498+0000 7f4d7cffa640 1 -- 172.21.15.70:0/1177179860 wait complete. 2024-04-22T01:15:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.506+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 --> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f989815d9c0 con 0x7f989815d0a0 2024-04-22T01:15:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f9886ffd640 1 -- 172.21.15.70:0/3680632218 <== osd.2 v2:172.21.15.70:6824/3866658519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f989815d9c0 con 0x7f989815d0a0 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f989815d0a0 msgr2=0x7f989815d480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f989815d0a0 0x7f989815d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f98780678c0 msgr2=0x7f9878069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f98780678c0 0x7f9878069d80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f988c009690 tx=0x7f988c006c00 comp rx=0 tx=0).stop 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98981659d0 msgr2=0x7f9898159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 --2- 172.21.15.70:0/3680632218 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98981659d0 0x7f9898159340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f989000c7f0 tx=0x7f989000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f9896d9d640 1 -- 172.21.15.70:0/3680632218 reap_dead start 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 shutdown_connections 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 >> 172.21.15.70:0/3680632218 conn(0x7f98980b2350 msgr2=0x7f98980b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 shutdown_connections 2024-04-22T01:15:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.510+0000 7f989c8d9640 1 -- 172.21.15.70:0/3680632218 wait complete. 2024-04-22T01:15:12.535 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:12.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-22T01:15:12.536 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:12.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:15:12.537 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-22T01:15:12.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T01:15:12.538 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:12.538 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T01:15:12.539 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-22T01:15:12.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:15:12.540 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-22T01:15:12.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-22T01:15:12.541 INFO:teuthology.orchestra.run.smithi070.stdout:55834574859 2024-04-22T01:15:12.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:15:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354359e640 1 Processor -- start 2024-04-22T01:15:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354359e640 1 -- start start 2024-04-22T01:15:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354359e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f354415b400 0x7f3544153d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35440726d0 con 0x7f3544071ed0 2024-04-22T01:15:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354359e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f354406f320 con 0x7f3544071af0 2024-04-22T01:15:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354359e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f35440b2a90 con 0x7f354415b400 2024-04-22T01:15:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3544071af0 0x7f354406e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:40786/0 (socket says 172.21.15.70:40786) 2024-04-22T01:15:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f354259c640 1 -- 172.21.15.70:0/1462218242 learned_addr learned my addr 172.21.15.70:0/1462218242 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f3541d9b640 1 --2- 172.21.15.70:0/1462218242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f354415b400 0x7f3544153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f3541d9b640 1 -- 172.21.15.70:0/1462218242 >> v1:172.21.15.98:6789/0 conn(0x7f3544071af0 legacy=0x7f354406e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f3541d9b640 1 -- 172.21.15.70:0/1462218242 >> v1:172.21.15.70:6789/0 conn(0x7f3544071ed0 legacy=0x7f35440b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.750+0000 7f3541d9b640 1 -- 172.21.15.70:0/1462218242 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354406f600 con 0x7f354415b400 2024-04-22T01:15:12.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 Processor -- start 2024-04-22T01:15:12.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 -- start start 2024-04-22T01:15:12.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 Processor -- start 2024-04-22T01:15:12.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 -- start start 2024-04-22T01:15:12.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b25d3640 1 Processor -- start 2024-04-22T01:15:12.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b25d3640 1 -- start start 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fee24071d60 0x7fee24072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee24165ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee240b4e20 con 0x7fee2415b960 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fee2415e550 con 0x7fee24071d60 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2359e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fee24072680 con 0x7fee24163ad0 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fa16d3dd640 1 Processor -- start 2024-04-22T01:15:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fa16d3dd640 1 -- start start 2024-04-22T01:15:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee21d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fee2415b960 0x7fee2406ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40800/0 (socket says 172.21.15.70:40800) 2024-04-22T01:15:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee21d9b640 1 -- 172.21.15.70:0/1117618787 learned_addr learned my addr 172.21.15.70:0/1117618787 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f3541d9b640 1 --2- 172.21.15.70:0/1462218242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f354415b400 0x7f3544153d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f35440b2c10 tx=0x7f353c01e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d76bb18b60ff2248 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2259c640 1 --2- 172.21.15.70:0/1117618787 >> v2:172.21.15.98:3300/0 conn(0x7fee24071d60 0x7fee24072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee22d9d640 1 --2- 172.21.15.70:0/1117618787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee24165ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f354159a640 1 -- 172.21.15.70:0/1462218242 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0ad070 con 0x7f354415b400 2024-04-22T01:15:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f354159a640 1 -- 172.21.15.70:0/1462218242 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353c0a7960 con 0x7f354415b400 2024-04-22T01:15:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f354159a640 1 -- 172.21.15.70:0/1462218242 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0a7d30 con 0x7f354415b400 2024-04-22T01:15:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c640e30d0 0x7f7c6403a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c6404ff20 0x7f7c640063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c64053e70 con 0x7f7c64053970 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7c64037bc0 con 0x7f7c6404ff20 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c70877640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7c64054150 con 0x7f7c640e30d0 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c6affd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7c64053970 0x7f7c6404f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40812/0 (socket says 172.21.15.70:40812) 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c6affd640 1 -- 172.21.15.70:0/3167340391 learned_addr learned my addr 172.21.15.70:0/3167340391 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c6bfff640 1 --2- 172.21.15.70:0/3167340391 >> v2:172.21.15.98:3300/0 conn(0x7f7c6404ff20 0x7f7c640063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f7c6b7fe640 1 --2- 172.21.15.70:0/3167340391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c640e30d0 0x7f7c6403a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b25d3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f78ac0cf200 0x7f78ac06e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b25d3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ac1456b0 con 0x7f78ac15c030 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b25d3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ac06e6d0 con 0x7f78ac1452d0 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b25d3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f78ac0cbfe0 con 0x7f78ac0cf200 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b15d1640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f78ac1452d0 0x7f78ac0ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:40796/0 (socket says 172.21.15.70:40796) 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b15d1640 1 -- 172.21.15.70:0/3764337862 learned_addr learned my addr 172.21.15.70:0/3764337862 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7f78b1dd2640 1 --2- 172.21.15.70:0/3764337862 >> v2:172.21.15.70:3300/0 conn(0x7f78ac0cf200 0x7f78ac06e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2159a640 1 -- 172.21.15.70:0/1117618787 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2892487009 0 0) 0x7fee240b4e20 con 0x7fee2415b960 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee2159a640 1 -- 172.21.15.70:0/1117618787 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee0c003530 con 0x7fee2415b960 2024-04-22T01:15:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fa80ce94640 1 Processor -- start 2024-04-22T01:15:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f78a3fff640 1 -- 172.21.15.70:0/3764337862 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064031043 0 0) 0x7f78ac06e6d0 con 0x7f78ac1452d0 2024-04-22T01:15:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f78a3fff640 1 -- 172.21.15.70:0/3764337862 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7898003530 con 0x7f78ac1452d0 2024-04-22T01:15:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- 172.21.15.70:0/1462218242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f354415b400 msgr2=0x7f3544153d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 --2- 172.21.15.70:0/1462218242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f354415b400 0x7f3544153d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f35440b2c10 tx=0x7f353c01e910 comp rx=0 tx=0).stop 2024-04-22T01:15:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- 172.21.15.70:0/1462218242 shutdown_connections 2024-04-22T01:15:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 --2- 172.21.15.70:0/1462218242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f354415b400 0x7f3544153d30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- 172.21.15.70:0/1462218242 >> 172.21.15.70:0/1462218242 conn(0x7f354406b1f0 msgr2=0x7f354406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- 172.21.15.70:0/1462218242 shutdown_connections 2024-04-22T01:15:12.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fa80ce94640 1 -- start start 2024-04-22T01:15:12.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80ce94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 0x7fa8081630a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80ce94640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8080c3f00 0x7fa8080cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80ce94640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8081635e0 con 0x7fa8080cef00 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80ce94640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa8080d02e0 con 0x7fa8080c3f00 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80ce94640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa808166b50 con 0x7fa8080cff00 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 0x7fa8081630a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 0x7fa8081630a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41168/0 (socket says 172.21.15.70:41168) 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 -- 172.21.15.70:0/2743903100 learned_addr learned my addr 172.21.15.70:0/2743903100 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa16d3dd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa16815f260 0x7fa1680cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa16d3dd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa16809a870 0x7fa1680c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 Processor -- start 2024-04-22T01:15:12.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 -- start start 2024-04-22T01:15:12.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60680aedd0 0x7f606806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6068071fb0 0x7f606816bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606806f0c0 con 0x7f606808a750 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f606816b240 con 0x7f6068071fb0 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f60700c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f606816b520 con 0x7f60680aedd0 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606d664640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f606808a750 0x7f60680718a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:40802/0 (socket says 172.21.15.70:40802) 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606d664640 1 -- 172.21.15.70:0/4126915432 learned_addr learned my addr 172.21.15.70:0/4126915432 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa16d3dd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1680cd4e0 con 0x7fa16809ac90 2024-04-22T01:15:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606de65640 1 --2- 172.21.15.70:0/4126915432 >> v2:172.21.15.98:3300/0 conn(0x7f60680aedd0 0x7f606806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa16d3dd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa16815f640 con 0x7fa16809a870 2024-04-22T01:15:12.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa16d3dd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa168099b40 con 0x7fa16815f260 2024-04-22T01:15:12.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee22d9d640 1 -- 172.21.15.70:0/1117618787 >> v2:172.21.15.98:3300/0 conn(0x7fee24071d60 msgr2=0x7fee24072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee22d9d640 1 --2- 172.21.15.70:0/1117618787 >> v2:172.21.15.98:3300/0 conn(0x7fee24071d60 0x7fee24072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee22d9d640 1 -- 172.21.15.70:0/1117618787 >> v1:172.21.15.70:6789/0 conn(0x7fee2415b960 legacy=0x7fee2406ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.754+0000 7fee22d9d640 1 -- 172.21.15.70:0/1117618787 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2415e7a0 con 0x7fee24163ad0 2024-04-22T01:15:12.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa1677fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa16809ac90 0x7fa1680c5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:40808/0 (socket says 172.21.15.70:40808) 2024-04-22T01:15:12.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa1677fe640 1 -- 172.21.15.70:0/1034119838 learned_addr learned my addr 172.21.15.70:0/1034119838 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:12.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- 172.21.15.70:0/1462218242 wait complete. 2024-04-22T01:15:12.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 Processor -- start 2024-04-22T01:15:12.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606de65640 1 -- 172.21.15.70:0/4126915432 >> v2:172.21.15.70:3300/0 conn(0x7f6068071fb0 msgr2=0x7f606816bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa166ffd640 1 --2- 172.21.15.70:0/1034119838 >> v2:172.21.15.98:3300/0 conn(0x7fa16815f260 0x7fa1680cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fee22d9d640 1 --2- 172.21.15.70:0/1117618787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee24165ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fee2415e6d0 tx=0x7fee1c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994d7598f393b205 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606de65640 1 --2- 172.21.15.70:0/4126915432 >> v2:172.21.15.70:3300/0 conn(0x7f6068071fb0 0x7f606816bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606de65640 1 -- 172.21.15.70:0/4126915432 >> v1:172.21.15.98:6789/0 conn(0x7f606808a750 legacy=0x7f60680718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fee2159a640 1 -- 172.21.15.70:0/1117618787 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0b2070 con 0x7fee24163ad0 2024-04-22T01:15:12.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- start start 2024-04-22T01:15:12.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f606de65640 1 -- 172.21.15.70:0/4126915432 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606816b800 con 0x7f60680aedd0 2024-04-22T01:15:12.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/3167340391 >> v2:172.21.15.98:3300/0 conn(0x7f7c6404ff20 msgr2=0x7f7c640063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f7c6b7fe640 1 --2- 172.21.15.70:0/3167340391 >> v2:172.21.15.98:3300/0 conn(0x7f7c6404ff20 0x7f7c640063d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/3167340391 >> v1:172.21.15.70:6789/0 conn(0x7f7c64053970 legacy=0x7f7c6404f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/3167340391 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c64054430 con 0x7f7c640e30d0 2024-04-22T01:15:12.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f78b1dd2640 1 -- 172.21.15.70:0/3764337862 >> v1:172.21.15.98:6789/0 conn(0x7f78ac1452d0 legacy=0x7f78ac0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f78b1dd2640 1 -- 172.21.15.70:0/3764337862 >> v1:172.21.15.70:6789/0 conn(0x7f78ac15c030 legacy=0x7f78ac06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f78b1dd2640 1 -- 172.21.15.70:0/3764337862 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac0b48a0 con 0x7f78ac0cf200 2024-04-22T01:15:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fee2159a640 1 -- 172.21.15.70:0/1117618787 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c0a5650 con 0x7fee24163ad0 2024-04-22T01:15:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fee2159a640 1 -- 172.21.15.70:0/1117618787 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0a5a40 con 0x7fee24163ad0 2024-04-22T01:15:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544071af0 0x7f354414b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 0x7f354414bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f354415b400 0x7f354414cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f354406eec0 con 0x7f354415b400 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35440b4400 con 0x7f3544071af0 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7f354359e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f354406f040 con 0x7f3544071ed0 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa166ffd640 1 -- 172.21.15.70:0/1034119838 >> v2:172.21.15.70:3300/0 conn(0x7fa16809a870 msgr2=0x7fa1680c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa166ffd640 1 --2- 172.21.15.70:0/1034119838 >> v2:172.21.15.70:3300/0 conn(0x7fa16809a870 0x7fa1680c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa166ffd640 1 -- 172.21.15.70:0/1034119838 >> v1:172.21.15.98:6789/0 conn(0x7fa16809ac90 legacy=0x7fa1680c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa166ffd640 1 -- 172.21.15.70:0/1034119838 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1680cd760 con 0x7fa16815f260 2024-04-22T01:15:12.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b1dd2640 1 --2- 172.21.15.70:0/3764337862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac0cf200 0x7f78ac06e070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0b2850 tx=0x7f78a401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12acf2ffe65792cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 0x7f354414bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 0x7f354414bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41178/0 (socket says 172.21.15.70:41178) 2024-04-22T01:15:12.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 -- 172.21.15.70:0/617166529 learned_addr learned my addr 172.21.15.70:0/617166529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f354259c640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544071af0 0x7f354414b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c6b7fe640 1 --2- 172.21.15.70:0/3167340391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c640e30d0 0x7f7c6403a440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7c600097c0 tx=0x7f7c6001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17caf6fabd0e2f9d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- 172.21.15.70:0/1117618787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 msgr2=0x7fee24165ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 --2- 172.21.15.70:0/1117618787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee24165ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fee2415e6d0 tx=0x7fee1c01c990 comp rx=0 tx=0).stop 2024-04-22T01:15:12.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 -- 172.21.15.70:0/2743903100 >> v2:172.21.15.70:3300/0 conn(0x7fa8080c3f00 msgr2=0x7fa8080cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 --2- 172.21.15.70:0/2743903100 >> v2:172.21.15.70:3300/0 conn(0x7fa8080c3f00 0x7fa8080cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 -- 172.21.15.70:0/2743903100 >> v1:172.21.15.70:6789/0 conn(0x7fa8080cef00 legacy=0x7fa8080c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3542d9d640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f354415b400 0x7f354414cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.758+0000 7fa80659c640 1 -- 172.21.15.70:0/2743903100 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa808165fd0 con 0x7fa8080cff00 2024-04-22T01:15:12.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- 172.21.15.70:0/1117618787 shutdown_connections 2024-04-22T01:15:12.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 --2- 172.21.15.70:0/1117618787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee24165ec0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 --2- 172.21.15.70:0/1117618787 >> v2:172.21.15.98:3300/0 conn(0x7fee24071d60 0x7fee24072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- 172.21.15.70:0/1117618787 >> 172.21.15.70:0/1117618787 conn(0x7fee240cfc60 msgr2=0x7fee240d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 -- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544071af0 msgr2=0x7f354414b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544071af0 0x7f354414b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 -- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f354415b400 msgr2=0x7f354414cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f354415b400 0x7f354414cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 -- 172.21.15.70:0/617166529 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35441fc440 con 0x7f3544071ed0 2024-04-22T01:15:12.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3541d9b640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 0x7f354414bf40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f353c01c540 tx=0x7f353c01b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0bf020 con 0x7f3544071ed0 2024-04-22T01:15:12.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353c0b1790 con 0x7f3544071ed0 2024-04-22T01:15:12.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0ad040 con 0x7f3544071ed0 2024-04-22T01:15:12.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c6a7fc640 1 -- 172.21.15.70:0/3167340391 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600b0070 con 0x7f7c640e30d0 2024-04-22T01:15:12.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c6a7fc640 1 -- 172.21.15.70:0/3167340391 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c600ab600 con 0x7f7c640e30d0 2024-04-22T01:15:12.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c6a7fc640 1 -- 172.21.15.70:0/3167340391 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600aba10 con 0x7f7c640e30d0 2024-04-22T01:15:12.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 -- 172.21.15.70:0/3167340391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c640e30d0 msgr2=0x7f7c6403a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 --2- 172.21.15.70:0/3167340391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c640e30d0 0x7f7c6403a440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7c600097c0 tx=0x7f7c6001c420 comp rx=0 tx=0).stop 2024-04-22T01:15:12.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 -- 172.21.15.70:0/3167340391 shutdown_connections 2024-04-22T01:15:12.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 --2- 172.21.15.70:0/3167340391 >> v2:172.21.15.98:3300/0 conn(0x7f7c6404ff20 0x7f7c640063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 --2- 172.21.15.70:0/3167340391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c640e30d0 0x7f7c6403a440 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 -- 172.21.15.70:0/3167340391 >> 172.21.15.70:0/3167340391 conn(0x7f7c64009fc0 msgr2=0x7f7c6400a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78a3fff640 1 -- 172.21.15.70:0/3764337862 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a40b2070 con 0x7f78ac0cf200 2024-04-22T01:15:12.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78a3fff640 1 -- 172.21.15.70:0/3764337862 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a40a58e0 con 0x7f78ac0cf200 2024-04-22T01:15:12.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78a3fff640 1 -- 172.21.15.70:0/3764337862 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a40a5cf0 con 0x7f78ac0cf200 2024-04-22T01:15:12.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 -- 172.21.15.70:0/3764337862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac0cf200 msgr2=0x7f78ac06e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 --2- 172.21.15.70:0/3764337862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac0cf200 0x7f78ac06e070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0b2850 tx=0x7f78a401c930 comp rx=0 tx=0).stop 2024-04-22T01:15:12.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- 172.21.15.70:0/1117618787 shutdown_connections 2024-04-22T01:15:12.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- 172.21.15.70:0/1117618787 wait complete. 2024-04-22T01:15:12.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 Processor -- start 2024-04-22T01:15:12.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80659c640 1 --2- 172.21.15.70:0/2743903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 0x7fa8081630a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa800009e50 tx=0x7fa80001c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2eab939f4a59131 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80559a640 1 -- 172.21.15.70:0/2743903100 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8000a55d0 con 0x7fa8080cff00 2024-04-22T01:15:12.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80559a640 1 -- 172.21.15.70:0/2743903100 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8000a5770 con 0x7fa8080cff00 2024-04-22T01:15:12.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80559a640 1 -- 172.21.15.70:0/2743903100 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8000a5b40 con 0x7fa8080cff00 2024-04-22T01:15:12.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 -- 172.21.15.70:0/3167340391 shutdown_connections 2024-04-22T01:15:12.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 -- 172.21.15.70:0/3764337862 shutdown_connections 2024-04-22T01:15:12.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 --2- 172.21.15.70:0/3764337862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac0cf200 0x7f78ac06e070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 -- 172.21.15.70:0/3764337862 >> 172.21.15.70:0/3764337862 conn(0x7f78ac06b1f0 msgr2=0x7f78ac06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 -- 172.21.15.70:0/3167340391 wait complete. 2024-04-22T01:15:12.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354414b850 con 0x7f3544071ed0 2024-04-22T01:15:12.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441fbba0 con 0x7f3544071ed0 2024-04-22T01:15:12.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 -- 172.21.15.70:0/3764337862 shutdown_connections 2024-04-22T01:15:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa166ffd640 1 --2- 172.21.15.70:0/1034119838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16815f260 0x7fa1680cc560 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1680731e0 tx=0x7fa15001c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa78d0f7b9df22bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 -- 172.21.15.70:0/2743903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 msgr2=0x7fa8081630a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 --2- 172.21.15.70:0/2743903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 0x7fa8081630a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa800009e50 tx=0x7fa80001c750 comp rx=0 tx=0).stop 2024-04-22T01:15:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 -- 172.21.15.70:0/2743903100 shutdown_connections 2024-04-22T01:15:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 --2- 172.21.15.70:0/2743903100 >> v2:172.21.15.70:3300/0 conn(0x7fa8080c3f00 0x7fa8080cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 --2- 172.21.15.70:0/2743903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cff00 0x7fa8081630a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 -- 172.21.15.70:0/2743903100 >> 172.21.15.70:0/2743903100 conn(0x7fa8080b2350 msgr2=0x7fa8080b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35280052a0 con 0x7f3544071ed0 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- start start 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee24071d60 0x7fee24205460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 0x7fee24154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee242013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee240ce0c0 con 0x7fee24071d60 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee240cdf40 con 0x7fee2415b960 2024-04-22T01:15:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee2359e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fee2416b430 con 0x7fee24163ad0 2024-04-22T01:15:12.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f78b25d3640 1 -- 172.21.15.70:0/3764337862 wait complete. 2024-04-22T01:15:12.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 -- 172.21.15.70:0/2743903100 shutdown_connections 2024-04-22T01:15:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa80ce94640 1 -- 172.21.15.70:0/2743903100 wait complete. 2024-04-22T01:15:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa1667fc640 1 -- 172.21.15.70:0/1034119838 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1500b0070 con 0x7fa16815f260 2024-04-22T01:15:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fcd7f59e640 1 Processor -- start 2024-04-22T01:15:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee21d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 0x7fee24154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee21d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 0x7fee24154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:42998/0 (socket says 172.21.15.70:42998) 2024-04-22T01:15:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee21d9b640 1 -- 172.21.15.70:0/4003619487 learned_addr learned my addr 172.21.15.70:0/4003619487 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fee22d9d640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee242013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fcd7f59e640 1 -- start start 2024-04-22T01:15:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7f7c70877640 1 Processor -- start 2024-04-22T01:15:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa1667fc640 1 -- 172.21.15.70:0/1034119838 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1500ab600 con 0x7fa16815f260 2024-04-22T01:15:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.762+0000 7fa1667fc640 1 -- 172.21.15.70:0/1034119838 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1500ab9d0 con 0x7fa16815f260 2024-04-22T01:15:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 Processor -- start 2024-04-22T01:15:12.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee21d9b640 1 -- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 msgr2=0x7fee242013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee21d9b640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee242013d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee21d9b640 1 -- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee24071d60 msgr2=0x7fee24205460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee21d9b640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee24071d60 0x7fee24205460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee21d9b640 1 -- 172.21.15.70:0/4003619487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee242059a0 con 0x7fee2415b960 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1034119838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16815f260 msgr2=0x7fa1680cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 --2- 172.21.15.70:0/1034119838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16815f260 0x7fa1680cc560 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1680731e0 tx=0x7fa15001c310 comp rx=0 tx=0).stop 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1034119838 shutdown_connections 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 --2- 172.21.15.70:0/1034119838 >> v2:172.21.15.70:3300/0 conn(0x7fa16809a870 0x7fa1680c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 --2- 172.21.15.70:0/1034119838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16815f260 0x7fa1680cc560 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1034119838 >> 172.21.15.70:0/1034119838 conn(0x7fa1680b34d0 msgr2=0x7fa1680b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1034119838 shutdown_connections 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1034119838 wait complete. 2024-04-22T01:15:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee21d9b640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 0x7fee24154350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee1800c500 tx=0x7fee1800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7f59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcd780e30d0 0x7fcd78005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7f59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd78054500 0x7fcd7804f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7f59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd7804faa0 0x7fcd78052a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7f59e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcd78037bc0 con 0x7fcd780e30d0 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7f59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780542c0 con 0x7fcd78054500 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7f59e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd780e7850 con 0x7fcd7804faa0 2024-04-22T01:15:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee18017070 con 0x7fee2415b960 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 -- start start 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 0x7f7c64035d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c64053970 0x7f7c64036240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c640e30d0 0x7f7c6418c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c640072a0 con 0x7f7c640e30d0 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c64007120 con 0x7f7c6404ff20 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7c64007420 con 0x7f7c64053970 2024-04-22T01:15:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 Processor -- start 2024-04-22T01:15:12.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 -- start start 2024-04-22T01:15:12.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd78054500 0x7fcd7804f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd78054500 0x7fcd7804f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43014/0 (socket says 172.21.15.70:43014) 2024-04-22T01:15:12.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7dd9b640 1 -- 172.21.15.70:0/869210735 learned_addr learned my addr 172.21.15.70:0/869210735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7ed9d640 1 --2- 172.21.15.70:0/869210735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd7804faa0 0x7fcd78052a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 -- start start 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 0x7f78ac15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac1452d0 0x7f78ac2065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac15c030 0x7f78ac15a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac16b820 con 0x7f78ac1452d0 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac16b6a0 con 0x7f78ac15c030 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b25d3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f78ac16b9a0 con 0x7f78ac0cf200 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 0x7f78ac15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 0x7f7c64035d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 0x7f7c64035d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:43018/0 (socket says 172.21.15.70:43018) 2024-04-22T01:15:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/759479640 learned_addr learned my addr 172.21.15.70:0/759479640 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 0x7f78ac15b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41200/0 (socket says 172.21.15.70:41200) 2024-04-22T01:15:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 -- 172.21.15.70:0/2099782397 learned_addr learned my addr 172.21.15.70:0/2099782397 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b1dd2640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac15c030 0x7f78ac15a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6affd640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c64053970 0x7f7c64036240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f353c0c2070 con 0x7f3544071ed0 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f3532ffd640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3524067910 0x7f3524069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f353c120b50 con 0x7f3544071ed0 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f354259c640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3524067910 0x7f3524069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 Processor -- start 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 -- start start 2024-04-22T01:15:12.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6bfff640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c640e30d0 0x7f7c6418c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b0dd0640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac1452d0 0x7f78ac2065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 -- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac15c030 msgr2=0x7f78ac15a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac15c030 0x7f78ac15a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 -- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac1452d0 msgr2=0x7f78ac2065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac1452d0 0x7f78ac2065a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 -- 172.21.15.70:0/2099782397 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac15f220 con 0x7f78ac0cf200 2024-04-22T01:15:12.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee18005cf0 con 0x7fee2415b960 2024-04-22T01:15:12.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee180072f0 con 0x7fee2415b960 2024-04-22T01:15:12.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee2359e640 1 -- 172.21.15.70:0/4003619487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee24154d40 con 0x7fee2415b960 2024-04-22T01:15:12.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee2359e640 1 -- 172.21.15.70:0/4003619487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee240b09c0 con 0x7fee2415b960 2024-04-22T01:15:12.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fee2359e640 1 -- 172.21.15.70:0/4003619487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee240b10d0 con 0x7fee2415b960 2024-04-22T01:15:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 0x7fa8081544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cef00 0x7fa8081549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8080cff00 0x7fa8081637c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa808165c10 con 0x7fa8080c3f00 2024-04-22T01:15:12.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8080afd80 con 0x7fa8080cff00 2024-04-22T01:15:12.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80ce94640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa808165d90 con 0x7fa8080cef00 2024-04-22T01:15:12.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7e59c640 1 --2- 172.21.15.70:0/869210735 >> v2:172.21.15.70:3300/0 conn(0x7fcd780e30d0 0x7fcd78005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7ed9d640 1 -- 172.21.15.70:0/869210735 >> v2:172.21.15.98:3300/0 conn(0x7fcd78054500 msgr2=0x7fcd7804f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7ed9d640 1 --2- 172.21.15.70:0/869210735 >> v2:172.21.15.98:3300/0 conn(0x7fcd78054500 0x7fcd7804f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7ed9d640 1 -- 172.21.15.70:0/869210735 >> v2:172.21.15.70:3300/0 conn(0x7fcd780e30d0 msgr2=0x7fcd78005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:12.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7ed9d640 1 --2- 172.21.15.70:0/869210735 >> v2:172.21.15.70:3300/0 conn(0x7fcd780e30d0 0x7fcd78005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7ed9d640 1 -- 172.21.15.70:0/869210735 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd78053c90 con 0x7fcd7804faa0 2024-04-22T01:15:12.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fcd7e59c640 1 --2- 172.21.15.70:0/869210735 >> v2:172.21.15.70:3300/0 conn(0x7fcd780e30d0 0x7fcd78005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:12.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f606de65640 1 --2- 172.21.15.70:0/4126915432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60680aedd0 0x7f606806eae0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6068072910 tx=0x7f606401eb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95acc9ecaf490a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c64053970 msgr2=0x7f7c64036240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c64053970 0x7f7c64036240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c640e30d0 msgr2=0x7f7c6418c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c640e30d0 0x7f7c6418c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 -- 172.21.15.70:0/759479640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c640d7f90 con 0x7f7c6404ff20 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa806d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8080cff00 0x7fa8081637c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa805d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cef00 0x7fa8081549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 0x7fa8081544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 0x7fa8081544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54766/0 (socket says 172.21.15.70:54766) 2024-04-22T01:15:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa805d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cef00 0x7fa8081549f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:41208/0 (socket says 172.21.15.70:41208) 2024-04-22T01:15:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 -- 172.21.15.70:0/1623602915 learned_addr learned my addr 172.21.15.70:0/1623602915 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f354259c640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3524067910 0x7f3524069dd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3534000c10 tx=0x7f35340047f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f78b15d1640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 0x7f78ac15b7a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c480 tx=0x7f78a800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c6b7fe640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 0x7f7c64035d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c600ae040 tx=0x7f7c600191c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 -- 172.21.15.70:0/1623602915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cef00 msgr2=0x7fa8081549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cef00 0x7fa8081549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 -- 172.21.15.70:0/1623602915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8080cff00 msgr2=0x7fa8081637c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8080cff00 0x7fa8081637c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa80659c640 1 -- 172.21.15.70:0/1623602915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa808153fd0 con 0x7fa8080c3f00 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16809a870 0x7fa16814bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7fa16d3dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 0x7fa1680cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa16815f260 0x7fa1680c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa16809a350 con 0x7fa16809ac90 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa16809a070 con 0x7fa16809a870 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa1680cf8d0 con 0x7fa16815f260 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600b0040 con 0x7f7c6404ff20 2024-04-22T01:15:12.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.766+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c640d8210 con 0x7f7c6404ff20 2024-04-22T01:15:12.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c640d87d0 con 0x7f7c6404ff20 2024-04-22T01:15:12.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa80659c640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 0x7fa8081544b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa80001ae00 tx=0x7fa8000194f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c600b9070 con 0x7f7c6404ff20 2024-04-22T01:15:12.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600ab600 con 0x7f7c6404ff20 2024-04-22T01:15:12.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8000bc020 con 0x7fa8080c3f00 2024-04-22T01:15:12.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8000b8690 con 0x7fa8080c3f00 2024-04-22T01:15:12.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800009510 con 0x7fa8080c3f00 2024-04-22T01:15:12.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa80ce94640 1 -- 172.21.15.70:0/1623602915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa808152cc0 con 0x7fa8080c3f00 2024-04-22T01:15:12.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa80ce94640 1 -- 172.21.15.70:0/1623602915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa808153280 con 0x7fa8080c3f00 2024-04-22T01:15:12.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa166ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16809a870 0x7fa16814bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 0x7fa1680cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 0x7fa1680cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54778/0 (socket says 172.21.15.70:54778) 2024-04-22T01:15:12.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 -- 172.21.15.70:0/1866281233 learned_addr learned my addr 172.21.15.70:0/1866281233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c0052a0 con 0x7f7c6404ff20 2024-04-22T01:15:12.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fcd7ed9d640 1 --2- 172.21.15.70:0/869210735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd7804faa0 0x7fcd78052a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800dcb0 tx=0x7fcd7401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb575e31bc9dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa1677fe640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa16815f260 0x7fa1680c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fee1801a070 con 0x7fee2415b960 2024-04-22T01:15:12.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fee12ffd640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fee04067990 0x7fee04069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fee18085800 con 0x7fee2415b960 2024-04-22T01:15:12.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fee2259c640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fee04067990 0x7fee04069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f606ce63640 1 -- 172.21.15.70:0/4126915432 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640ad070 con 0x7f60680aedd0 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f606ce63640 1 -- 172.21.15.70:0/4126915432 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60640a7ae0 con 0x7f60680aedd0 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 -- 172.21.15.70:0/4126915432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60680aedd0 msgr2=0x7f606806eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f606ce63640 1 -- 172.21.15.70:0/4126915432 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640ad070 con 0x7f60680aedd0 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 --2- 172.21.15.70:0/4126915432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60680aedd0 0x7f606806eae0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6068072910 tx=0x7f606401eb40 comp rx=0 tx=0).stop 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 -- 172.21.15.70:0/4126915432 shutdown_connections 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 --2- 172.21.15.70:0/4126915432 >> v2:172.21.15.70:3300/0 conn(0x7f6068071fb0 0x7f606816bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 --2- 172.21.15.70:0/4126915432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60680aedd0 0x7f606806eae0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 -- 172.21.15.70:0/4126915432 >> 172.21.15.70:0/4126915432 conn(0x7f606806b1f0 msgr2=0x7f606806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 -- 172.21.15.70:0/4126915432 shutdown_connections 2024-04-22T01:15:12.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 -- 172.21.15.70:0/4126915432 wait complete. 2024-04-22T01:15:12.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa80ce94640 1 -- 172.21.15.70:0/1623602915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ec0052a0 con 0x7fa8080c3f00 2024-04-22T01:15:12.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 -- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa16815f260 msgr2=0x7fa1680c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa16815f260 0x7fa1680c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 -- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16809a870 msgr2=0x7fa16814bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16809a870 0x7fa16814bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 -- 172.21.15.70:0/1866281233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16815e6a0 con 0x7fa16809ac90 2024-04-22T01:15:12.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fee2259c640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fee04067990 0x7fee04069e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fee14006aa0 tx=0x7fee14006a30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15ffff640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 0x7fa1680cb360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa15800b480 tx=0x7fa15800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15801d070 con 0x7fa16809ac90 2024-04-22T01:15:12.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1866281233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16815b960 con 0x7fa16809ac90 2024-04-22T01:15:12.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1866281233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1680cab60 con 0x7fa16809ac90 2024-04-22T01:15:12.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 Processor -- start 2024-04-22T01:15:12.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f60700c9640 1 -- start start 2024-04-22T01:15:12.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1866281233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1680ce1d0 con 0x7fa16809ac90 2024-04-22T01:15:12.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.770+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c600c8430 con 0x7f7c6404ff20 2024-04-22T01:15:12.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f7c57fff640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7c50067a60 0x7f7c50069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8000bf020 con 0x7fa8080c3f00 2024-04-22T01:15:12.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f60700c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 0x7f60680cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f60700c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606808a750 0x7f60680cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f60700c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60680aedd0 0x7f60680cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f60700c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f606816ce40 con 0x7f6068071fb0 2024-04-22T01:15:12.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f60700c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f606816ccc0 con 0x7f606808a750 2024-04-22T01:15:12.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f60700c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f606816cfc0 con 0x7f60680aedd0 2024-04-22T01:15:12.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f7c6affd640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7c50067a60 0x7f7c50069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 0x7f60680cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 0x7f60680cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54784/0 (socket says 172.21.15.70:54784) 2024-04-22T01:15:12.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 -- 172.21.15.70:0/1338252037 learned_addr learned my addr 172.21.15.70:0/1338252037 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7fa7f6ffd640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa7e8067910 0x7fa7e8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606d664640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606808a750 0x7f60680cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f7c6affd640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7c50067a60 0x7f7c50069f20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7c6418bfb0 tx=0x7f7c58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606e666640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60680aedd0 0x7f60680cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa800121760 con 0x7fa8080c3f00 2024-04-22T01:15:12.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7fa805d9b640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa7e8067910 0x7fa7e8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 -- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60680aedd0 msgr2=0x7f60680cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60680aedd0 0x7f60680cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 -- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606808a750 msgr2=0x7f60680cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606808a750 0x7f60680cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 -- 172.21.15.70:0/1338252037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6068162780 con 0x7f6068071fb0 2024-04-22T01:15:12.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7fa805d9b640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa7e8067910 0x7fa7e8069dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa8081652d0 tx=0x7fa7f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.774+0000 7f606de65640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 0x7f60680cdb40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f60640b3040 tx=0x7f606401b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa158010b70 con 0x7fa16809ac90 2024-04-22T01:15:12.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158027060 con 0x7fa16809ac90 2024-04-22T01:15:12.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640bf070 con 0x7f6068071fb0 2024-04-22T01:15:12.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6068162a60 con 0x7f6068071fb0 2024-04-22T01:15:12.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6068163300 con 0x7f6068071fb0 2024-04-22T01:15:12.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60640a7ae0 con 0x7f6068071fb0 2024-04-22T01:15:12.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640c9060 con 0x7f6068071fb0 2024-04-22T01:15:12.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa8081635e0 con 0x7fa8080c3f00 2024-04-22T01:15:12.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa158020070 con 0x7fa16809ac90 2024-04-22T01:15:12.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa15f7fe640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1480678f0 0x7fa148069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa158089d80 con 0x7fa16809ac90 2024-04-22T01:15:12.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa166ffd640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1480678f0 0x7fa148069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f60640c2070 con 0x7f6068071fb0 2024-04-22T01:15:12.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60500052a0 con 0x7f6068071fb0 2024-04-22T01:15:12.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7fa166ffd640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1480678f0 0x7fa148069db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1500ad000 tx=0x7fa1500a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f605e7fc640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f604c0678c0 0x7f604c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.778+0000 7f606d664640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f604c0678c0 0x7f604c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f606d664640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f604c0678c0 0x7f604c069d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f606816aa10 tx=0x7f6060009500 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8017070 con 0x7f78ac0cf200 2024-04-22T01:15:12.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a8005cf0 con 0x7f78ac0cf200 2024-04-22T01:15:12.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8007290 con 0x7f78ac0cf200 2024-04-22T01:15:12.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f353c0c9070 con 0x7f3544071ed0 2024-04-22T01:15:12.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7d59a640 1 -- 172.21.15.70:0/869210735 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd740b5070 con 0x7fcd7804faa0 2024-04-22T01:15:12.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7d59a640 1 -- 172.21.15.70:0/869210735 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd740a8830 con 0x7fcd7804faa0 2024-04-22T01:15:12.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7d59a640 1 -- 172.21.15.70:0/869210735 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd740a8c00 con 0x7fcd7804faa0 2024-04-22T01:15:12.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac13a490 con 0x7f78ac0cf200 2024-04-22T01:15:12.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac15e980 con 0x7f78ac0cf200 2024-04-22T01:15:12.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac06c180 con 0x7f78ac0cf200 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 -- 172.21.15.70:0/869210735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd7804faa0 msgr2=0x7fcd78052a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/869210735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd7804faa0 0x7fcd78052a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800dcb0 tx=0x7fcd7401ec70 comp rx=0 tx=0).stop 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 -- 172.21.15.70:0/869210735 shutdown_connections 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/869210735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd7804faa0 0x7fcd78052a50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/869210735 >> v2:172.21.15.98:3300/0 conn(0x7fcd78054500 0x7fcd7804f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/869210735 >> v2:172.21.15.70:3300/0 conn(0x7fcd780e30d0 0x7fcd78005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.782+0000 7fcd7f59e640 1 -- 172.21.15.70:0/869210735 >> 172.21.15.70:0/869210735 conn(0x7fcd78009fc0 msgr2=0x7fcd7800a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:12.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fee18055a20 con 0x7fee2415b960 2024-04-22T01:15:12.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7c600ee6d0 con 0x7f7c6404ff20 2024-04-22T01:15:12.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78a801a070 con 0x7f78ac0cf200 2024-04-22T01:15:12.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f78a1ffb640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7890067990 0x7f7890069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa158059f20 con 0x7fa16809ac90 2024-04-22T01:15:12.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f78a8085790 con 0x7f78ac0cf200 2024-04-22T01:15:12.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f606411e910 con 0x7f6068071fb0 2024-04-22T01:15:12.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f78b0dd0640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7890067990 0x7f7890069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7fcd7f59e640 1 -- 172.21.15.70:0/869210735 shutdown_connections 2024-04-22T01:15:12.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7fcd7f59e640 1 -- 172.21.15.70:0/869210735 wait complete. 2024-04-22T01:15:12.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.786+0000 7f78b0dd0640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7890067990 0x7f7890069e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78ac15f510 tx=0x7f789c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 Processor -- start 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 -- start start 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7804faa0 0x7fcd78049180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 0x7fcd78037d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd780e30d0 0x7fcd78038290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd78053f50 con 0x7fcd78054500 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780e7af0 con 0x7fcd7804faa0 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7f59e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7804ff50 con 0x7fcd780e30d0 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 0x7fcd78037d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 0x7fcd78037d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54786/0 (socket says 172.21.15.70:54786) 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 -- 172.21.15.70:0/3904871917 learned_addr learned my addr 172.21.15.70:0/3904871917 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7e59c640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7804faa0 0x7fcd78049180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7ed9d640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd780e30d0 0x7fcd78038290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 -- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd780e30d0 msgr2=0x7fcd78038290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd780e30d0 0x7fcd78038290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 -- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7804faa0 msgr2=0x7fcd78049180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7804faa0 0x7fcd78049180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 -- 172.21.15.70:0/3904871917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd780de1f0 con 0x7fcd78054500 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.790+0000 7fcd7dd9b640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 0x7fcd78037d50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcd800750f0 tx=0x7fcd80068cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6064018730 con 0x7f6068071fb0 2024-04-22T01:15:12.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f78a8021020 con 0x7f78ac0cf200 2024-04-22T01:15:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd800744a0 con 0x7fcd78054500 2024-04-22T01:15:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd780388f0 con 0x7fcd78054500 2024-04-22T01:15:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd78048980 con 0x7fcd78054500 2024-04-22T01:15:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd80074640 con 0x7fcd78054500 2024-04-22T01:15:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd800749f0 con 0x7fcd78054500 2024-04-22T01:15:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7c640d7f90 con 0x7f7c6404ff20 2024-04-22T01:15:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.798+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd80093430 con 0x7fcd78054500 2024-04-22T01:15:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.802+0000 7fcd6effd640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcd600678c0 0x7fcd60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.802+0000 7fcd7e59c640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcd600678c0 0x7fcd60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.802+0000 7fcd7e59c640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcd600678c0 0x7fcd60069d80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcd70006fd0 tx=0x7fcd70008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.802+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcd80100730 con 0x7fcd78054500 2024-04-22T01:15:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.810+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd640052a0 con 0x7fcd78054500 2024-04-22T01:15:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:12.818+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcd800d0eb0 con 0x7fcd78054500 2024-04-22T01:15:13.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.086+0000 7fa80ce94640 1 -- 172.21.15.70:0/1623602915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa7ec005860 con 0x7fa8080c3f00 2024-04-22T01:15:13.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.086+0000 7fa7f6ffd640 1 -- 172.21.15.70:0/1623602915 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa8000f2160 con 0x7fa8080c3f00 2024-04-22T01:15:13.090 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-22T01:15:13.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.090+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3528005860 con 0x7f3544071ed0 2024-04-22T01:15:13.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.090+0000 7f3532ffd640 1 -- 172.21.15.70:0/617166529 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f353c0185c0 con 0x7f3544071ed0 2024-04-22T01:15:13.091 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:13.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 -- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa7e8067910 msgr2=0x7fa7e8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa7e8067910 0x7fa7e8069dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa8081652d0 tx=0x7fa7f8009040 comp rx=0 tx=0).stop 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 -- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 msgr2=0x7fa8081544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 0x7fa8081544b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa80001ae00 tx=0x7fa8000194f0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 -- 172.21.15.70:0/1623602915 shutdown_connections 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa7e8067910 0x7fa7e8069dd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8080cff00 0x7fa8081637c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8080cef00 0x7fa8081549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 --2- 172.21.15.70:0/1623602915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8080c3f00 0x7fa8081544b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 -- 172.21.15.70:0/1623602915 >> 172.21.15.70:0/1623602915 conn(0x7fa8080b2350 msgr2=0x7fa8080d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 -- 172.21.15.70:0/1623602915 shutdown_connections 2024-04-22T01:15:13.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7fa7f4ff9640 1 -- 172.21.15.70:0/1623602915 wait complete. 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3524067910 msgr2=0x7f3524069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3524067910 0x7f3524069dd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3534000c10 tx=0x7f35340047f0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 msgr2=0x7f354414bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 0x7f354414bf40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f353c01c540 tx=0x7f353c01b6b0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 shutdown_connections 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3524067910 0x7f3524069dd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f354415b400 0x7f354414cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3544071ed0 0x7f354414bf40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 --2- 172.21.15.70:0/617166529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544071af0 0x7f354414b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 >> 172.21.15.70:0/617166529 conn(0x7f354406b1f0 msgr2=0x7f35440af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 shutdown_connections 2024-04-22T01:15:13.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.094+0000 7f354359e640 1 -- 172.21.15.70:0/617166529 wait complete. 2024-04-22T01:15:13.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.098+0000 7fee2359e640 1 -- 172.21.15.70:0/4003619487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fee2404f510 con 0x7fee2415b960 2024-04-22T01:15:13.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.098+0000 7fee12ffd640 1 -- 172.21.15.70:0/4003619487 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee18059d20 con 0x7fee2415b960 2024-04-22T01:15:13.100 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 -- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fee04067990 msgr2=0x7fee04069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fee04067990 0x7fee04069e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fee14006aa0 tx=0x7fee14006a30 comp rx=0 tx=0).stop 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 -- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 msgr2=0x7fee24154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 0x7fee24154350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee1800c500 tx=0x7fee1800c9c0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 -- 172.21.15.70:0/4003619487 shutdown_connections 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fee04067990 0x7fee04069e50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee24163ad0 0x7fee242013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee2415b960 0x7fee24154350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 --2- 172.21.15.70:0/4003619487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee24071d60 0x7fee24205460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 -- 172.21.15.70:0/4003619487 >> 172.21.15.70:0/4003619487 conn(0x7fee240cfc60 msgr2=0x7fee240d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 -- 172.21.15.70:0/4003619487 shutdown_connections 2024-04-22T01:15:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.102+0000 7fee10ff9640 1 -- 172.21.15.70:0/4003619487 wait complete. 2024-04-22T01:15:13.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.110+0000 7fa16d3dd640 1 -- 172.21.15.70:0/1866281233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa1680c50b0 con 0x7fa16809ac90 2024-04-22T01:15:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.110+0000 7fa15f7fe640 1 -- 172.21.15.70:0/1866281233 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa15805e220 con 0x7fa16809ac90 2024-04-22T01:15:13.113 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:13.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 -- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1480678f0 msgr2=0x7fa148069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1480678f0 0x7fa148069db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1500ad000 tx=0x7fa1500a6000 comp rx=0 tx=0).stop 2024-04-22T01:15:13.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 -- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 msgr2=0x7fa1680cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 0x7fa1680cb360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa15800b480 tx=0x7fa15800b940 comp rx=0 tx=0).stop 2024-04-22T01:15:13.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 -- 172.21.15.70:0/1866281233 shutdown_connections 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1480678f0 0x7fa148069db0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa16815f260 0x7fa1680c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa16809ac90 0x7fa1680cb360 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 --2- 172.21.15.70:0/1866281233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa16809a870 0x7fa16814bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 -- 172.21.15.70:0/1866281233 >> 172.21.15.70:0/1866281233 conn(0x7fa1680b34d0 msgr2=0x7fa1680c5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 -- 172.21.15.70:0/1866281233 shutdown_connections 2024-04-22T01:15:13.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.114+0000 7fa15d7fa640 1 -- 172.21.15.70:0/1866281233 wait complete. 2024-04-22T01:15:13.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.118+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f78ac04f510 con 0x7f78ac0cf200 2024-04-22T01:15:13.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.122+0000 7f78a1ffb640 1 -- 172.21.15.70:0/2099782397 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f78ac13a490 con 0x7f78ac0cf200 2024-04-22T01:15:13.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-04-22T01:15:13.122 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-22T01:15:13.123 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-04-22T01:15:13.124 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7890067990 msgr2=0x7f7890069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7890067990 0x7f7890069e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78ac15f510 tx=0x7f789c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 msgr2=0x7f78ac15b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 0x7f78ac15b7a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c480 tx=0x7f78a800c940 comp rx=0 tx=0).stop 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 shutdown_connections 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7890067990 0x7f7890069e50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac15c030 0x7f78ac15a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac1452d0 0x7f78ac2065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 --2- 172.21.15.70:0/2099782397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0cf200 0x7f78ac15b7a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 >> 172.21.15.70:0/2099782397 conn(0x7f78ac06b1f0 msgr2=0x7f78ac157660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 shutdown_connections 2024-04-22T01:15:13.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f78b25d3640 1 -- 172.21.15.70:0/2099782397 wait complete. 2024-04-22T01:15:13.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6050005590 con 0x7f6068071fb0 2024-04-22T01:15:13.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.126+0000 7f605e7fc640 1 -- 172.21.15.70:0/1338252037 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f60640f07d0 con 0x7f6068071fb0 2024-04-22T01:15:13.128 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-22T01:15:13.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f604c0678c0 msgr2=0x7f604c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f604c0678c0 0x7f604c069d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f606816aa10 tx=0x7f6060009500 comp rx=0 tx=0).stop 2024-04-22T01:15:13.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 msgr2=0x7f60680cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 0x7f60680cdb40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f60640b3040 tx=0x7f606401b8e0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 shutdown_connections 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f604c0678c0 0x7f604c069d80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60680aedd0 0x7f60680cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f606808a750 0x7f60680cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 --2- 172.21.15.70:0/1338252037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6068071fb0 0x7f60680cdb40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 >> 172.21.15.70:0/1338252037 conn(0x7f606806b1f0 msgr2=0x7f60680b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 shutdown_connections 2024-04-22T01:15:13.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f60700c9640 1 -- 172.21.15.70:0/1338252037 wait complete. 2024-04-22T01:15:13.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fcd64005860 con 0x7fcd78054500 2024-04-22T01:15:13.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7fcd6effd640 1 -- 172.21.15.70:0/3904871917 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcd800d07b0 con 0x7fcd78054500 2024-04-22T01:15:13.132 INFO:teuthology.orchestra.run.smithi070.stdout:55834574859 2024-04-22T01:15:13.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-22T01:15:13.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.130+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7c4c005860 con 0x7f7c6404ff20 2024-04-22T01:15:13.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7f7c57fff640 1 -- 172.21.15.70:0/759479640 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7c60016010 con 0x7f7c6404ff20 2024-04-22T01:15:13.134 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.134 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-22T01:15:13.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7c50067a60 msgr2=0x7f7c50069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7f7c70877640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7c50067a60 0x7f7c50069f20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7c6418bfb0 tx=0x7f7c58009040 comp rx=0 tx=0).stop 2024-04-22T01:15:13.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 msgr2=0x7f7c64035d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7f7c70877640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 0x7f7c64035d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c600ae040 tx=0x7f7c600191c0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcd600678c0 msgr2=0x7fcd60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcd600678c0 0x7fcd60069d80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcd70006fd0 tx=0x7fcd70008040 comp rx=0 tx=0).stop 2024-04-22T01:15:13.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 msgr2=0x7fcd78037d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 0x7fcd78037d50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcd800750f0 tx=0x7fcd80068cf0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 shutdown_connections 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcd600678c0 0x7fcd60069d80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd780e30d0 0x7fcd78038290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd78054500 0x7fcd78037d50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 --2- 172.21.15.70:0/3904871917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7804faa0 0x7fcd78049180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 >> 172.21.15.70:0/3904871917 conn(0x7fcd78009fc0 msgr2=0x7fcd780e1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 shutdown_connections 2024-04-22T01:15:13.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.134+0000 7fcd7f59e640 1 -- 172.21.15.70:0/3904871917 wait complete. 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 shutdown_connections 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7c50067a60 0x7f7c50069f20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c640e30d0 0x7f7c6418c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c64053970 0x7f7c64036240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 --2- 172.21.15.70:0/759479640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6404ff20 0x7f7c64035d00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 >> 172.21.15.70:0/759479640 conn(0x7f7c64009fc0 msgr2=0x7f7c640dbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 shutdown_connections 2024-04-22T01:15:13.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.138+0000 7f7c70877640 1 -- 172.21.15.70:0/759479640 wait complete. 2024-04-22T01:15:13.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-04-22T01:15:13.139 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.151 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.4 2024-04-22T01:15:13.151 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-22T01:15:13.152 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-04-22T01:15:13.163 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.164 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-04-22T01:15:13.164 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:13.164 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T01:15:13.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:15:13.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.310+0000 7f2ee1872640 1 Processor -- start 2024-04-22T01:15:13.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.310+0000 7f2ee1872640 1 -- start start 2024-04-22T01:15:13.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2edc09a860 0x7f2edc0cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc15f7b0 con 0x7f2edc15f3d0 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc0cd3e0 con 0x7f2edc09a3d0 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2edc0d0ba0 con 0x7f2edc09a860 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2edc09a860 0x7f2edc0cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2edc09a860 0x7f2edc0cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54800/0 (socket says 172.21.15.70:54800) 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 -- 172.21.15.70:0/1066171122 learned_addr learned my addr 172.21.15.70:0/1066171122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edaffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2edc15f3d0 0x7f2edc099cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40852/0 (socket says 172.21.15.70:40852) 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 -- 172.21.15.70:0/1066171122 >> v1:172.21.15.98:6789/0 conn(0x7f2edc09a3d0 legacy=0x7f2edc0cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:13.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 -- 172.21.15.70:0/1066171122 >> v1:172.21.15.70:6789/0 conn(0x7f2edc15f3d0 legacy=0x7f2edc099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 -- 172.21.15.70:0/1066171122 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc0cd6c0 con 0x7f2edc09a860 2024-04-22T01:15:13.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ed9ffb640 1 -- 172.21.15.70:0/1066171122 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3204185622 0 0) 0x7f2edc15f7b0 con 0x7f2edc15f3d0 2024-04-22T01:15:13.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2edb7fe640 1 --2- 172.21.15.70:0/1066171122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a860 0x7f2edc0cfdc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4009870 tx=0x7f2ed401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ca85060ea04394 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ed9ffb640 1 -- 172.21.15.70:0/1066171122 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed40b0070 con 0x7f2edc09a860 2024-04-22T01:15:13.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ed9ffb640 1 -- 172.21.15.70:0/1066171122 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed40ab600 con 0x7f2edc09a860 2024-04-22T01:15:13.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ed9ffb640 1 -- 172.21.15.70:0/1066171122 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed40ab9d0 con 0x7f2edc09a860 2024-04-22T01:15:13.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- 172.21.15.70:0/1066171122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a860 msgr2=0x7f2edc0cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 --2- 172.21.15.70:0/1066171122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a860 0x7f2edc0cfdc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4009870 tx=0x7f2ed401c520 comp rx=0 tx=0).stop 2024-04-22T01:15:13.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- 172.21.15.70:0/1066171122 shutdown_connections 2024-04-22T01:15:13.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 --2- 172.21.15.70:0/1066171122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a860 0x7f2edc0cfdc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- 172.21.15.70:0/1066171122 >> 172.21.15.70:0/1066171122 conn(0x7f2edc0b35d0 msgr2=0x7f2edc0b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- 172.21.15.70:0/1066171122 shutdown_connections 2024-04-22T01:15:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.314+0000 7f2ee1872640 1 -- 172.21.15.70:0/1066171122 wait complete. 2024-04-22T01:15:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 Processor -- start 2024-04-22T01:15:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- start start 2024-04-22T01:15:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 0x7f2edc14df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2edc09a860 0x7f2edc14f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2edc15f3d0 0x7f2edc14b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc0c67c0 con 0x7f2edc09a3d0 2024-04-22T01:15:13.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc0c6640 con 0x7f2edc09a860 2024-04-22T01:15:13.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2edc0d0710 con 0x7f2edc15f3d0 2024-04-22T01:15:13.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 0x7f2edc14df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 0x7f2edc14df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54812/0 (socket says 172.21.15.70:54812) 2024-04-22T01:15:13.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 -- 172.21.15.70:0/2319216661 learned_addr learned my addr 172.21.15.70:0/2319216661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:13.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2eda7fc640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2edc09a860 0x7f2edc14f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edb7fe640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2edc15f3d0 0x7f2edc14b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 -- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2edc15f3d0 msgr2=0x7f2edc14b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2edc15f3d0 0x7f2edc14b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 -- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2edc09a860 msgr2=0x7f2edc14f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2edc09a860 0x7f2edc14f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 -- 172.21.15.70:0/2319216661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc14fe80 con 0x7f2edc09a3d0 2024-04-22T01:15:13.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2edaffd640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 0x7f2edc14df80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00c7c0 tx=0x7f2ecc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc018070 con 0x7f2edc09a3d0 2024-04-22T01:15:13.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ecc00a640 con 0x7f2edc09a3d0 2024-04-22T01:15:13.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2edc150100 con 0x7f2edc09a3d0 2024-04-22T01:15:13.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc00aa30 con 0x7f2edc09a3d0 2024-04-22T01:15:13.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2edc0b4af0 con 0x7f2edc09a3d0 2024-04-22T01:15:13.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2edc068920 con 0x7f2edc09a3d0 2024-04-22T01:15:13.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.318+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2ecc01b070 con 0x7f2edc09a3d0 2024-04-22T01:15:13.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.322+0000 7f2ec3fff640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ec80678f0 0x7f2ec8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.322+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2ecc085560 con 0x7f2edc09a3d0 2024-04-22T01:15:13.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.322+0000 7f2eda7fc640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ec80678f0 0x7f2ec8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.322+0000 7f2eda7fc640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ec80678f0 0x7f2ec8069db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0d1920 tx=0x7f2ed0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.326+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2edc14fe80 con 0x7f2edc09a3d0 2024-04-22T01:15:13.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.470+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2edc0c7900 con 0x7f2ec80678f0 2024-04-22T01:15:13.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.478+0000 7f2ec3fff640 1 -- 172.21.15.70:0/2319216661 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f2edc0c7900 con 0x7f2ec80678f0 2024-04-22T01:15:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:13.482 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ec80678f0 msgr2=0x7f2ec8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ec80678f0 0x7f2ec8069db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0d1920 tx=0x7f2ed0009040 comp rx=0 tx=0).stop 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 msgr2=0x7f2edc14df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 0x7f2edc14df80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00c7c0 tx=0x7f2ecc00cc80 comp rx=0 tx=0).stop 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 shutdown_connections 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ec80678f0 0x7f2ec8069db0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2edc15f3d0 0x7f2edc14b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2edc09a860 0x7f2edc14f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 --2- 172.21.15.70:0/2319216661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc09a3d0 0x7f2edc14df80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 >> 172.21.15.70:0/2319216661 conn(0x7f2edc0b35d0 msgr2=0x7f2edc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 shutdown_connections 2024-04-22T01:15:13.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.482+0000 7f2ee1872640 1 -- 172.21.15.70:0/2319216661 wait complete. 2024-04-22T01:15:13.497 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-22T01:15:12.594624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236020,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738700,"statfs":{"total":773094113280,"available":772852428800,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005039"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805910+0000","last_change":"2024-04-22T01:14:45.690356+0000","last_active":"2024-04-22T01:14:52.805910+0000","last_peered":"2024-04-22T01:14:52.805910+0000","last_clean":"2024-04-22T01:14:52.805910+0000","last_became_active":"2024-04-22T01:14:45.690080+0000","last_became_peered":"2024-04-22T01:14:45.690080+0000","last_unstale":"2024-04-22T01:14:52.805910+0000","last_undegraded":"2024-04-22T01:14:52.805910+0000","last_fullsized":"2024-04-22T01:14:52.805910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917875+0000","last_change":"2024-04-22T01:14:46.720257+0000","last_active":"2024-04-22T01:14:53.917875+0000","last_peered":"2024-04-22T01:14:53.917875+0000","last_clean":"2024-04-22T01:14:53.917875+0000","last_became_active":"2024-04-22T01:14:46.720047+0000","last_became_peered":"2024-04-22T01:14:46.720047+0000","last_unstale":"2024-04-22T01:14:53.917875+0000","last_undegraded":"2024-04-22T01:14:53.917875+0000","last_fullsized":"2024-04-22T01:14:53.917875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916284+0000","last_change":"2024-04-22T01:14:50.933593+0000","last_active":"2024-04-22T01:14:53.916284+0000","last_peered":"2024-04-22T01:14:53.916284+0000","last_clean":"2024-04-22T01:14:53.916284+0000","last_became_active":"2024-04-22T01:14:50.933273+0000","last_became_peered":"2024-04-22T01:14:50.933273+0000","last_unstale":"2024-04-22T01:14:53.916284+0000","last_undegraded":"2024-04-22T01:14:53.916284+0000","last_fullsized":"2024-04-22T01:14:53.916284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:46.804215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915513+0000","last_change":"2024-04-22T01:14:45.689486+0000","last_active":"2024-04-22T01:14:53.915513+0000","last_peered":"2024-04-22T01:14:53.915513+0000","last_clean":"2024-04-22T01:14:53.915513+0000","last_became_active":"2024-04-22T01:14:45.689150+0000","last_became_peered":"2024-04-22T01:14:45.689150+0000","last_unstale":"2024-04-22T01:14:53.915513+0000","last_undegraded":"2024-04-22T01:14:53.915513+0000","last_fullsized":"2024-04-22T01:14:53.915513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805716+0000","last_change":"2024-04-22T01:14:46.688092+0000","last_active":"2024-04-22T01:14:52.805716+0000","last_peered":"2024-04-22T01:14:52.805716+0000","last_clean":"2024-04-22T01:14:52.805716+0000","last_became_active":"2024-04-22T01:14:46.687812+0000","last_became_peered":"2024-04-22T01:14:46.687812+0000","last_unstale":"2024-04-22T01:14:52.805716+0000","last_undegraded":"2024-04-22T01:14:52.805716+0000","last_fullsized":"2024-04-22T01:14:52.805716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745650+0000","last_change":"2024-04-22T01:14:50.778199+0000","last_active":"2024-04-22T01:14:51.745650+0000","last_peered":"2024-04-22T01:14:51.745650+0000","last_clean":"2024-04-22T01:14:51.745650+0000","last_became_active":"2024-04-22T01:14:50.773891+0000","last_became_peered":"2024-04-22T01:14:50.773891+0000","last_unstale":"2024-04-22T01:14:51.745650+0000","last_undegraded":"2024-04-22T01:14:51.745650+0000","last_fullsized":"2024-04-22T01:14:51.745650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:50.558971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445520+0000","last_change":"2024-04-22T01:14:45.702628+0000","last_active":"2024-04-22T01:14:53.445520+0000","last_peered":"2024-04-22T01:14:53.445520+0000","last_clean":"2024-04-22T01:14:53.445520+0000","last_became_active":"2024-04-22T01:14:45.701650+0000","last_became_peered":"2024-04-22T01:14:45.701650+0000","last_unstale":"2024-04-22T01:14:53.445520+0000","last_undegraded":"2024-04-22T01:14:53.445520+0000","last_fullsized":"2024-04-22T01:14:53.445520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749682+0000","last_change":"2024-04-22T01:14:46.694700+0000","last_active":"2024-04-22T01:14:51.749682+0000","last_peered":"2024-04-22T01:14:51.749682+0000","last_clean":"2024-04-22T01:14:51.749682+0000","last_became_active":"2024-04-22T01:14:46.694050+0000","last_became_peered":"2024-04-22T01:14:46.694050+0000","last_unstale":"2024-04-22T01:14:51.749682+0000","last_undegraded":"2024-04-22T01:14:51.749682+0000","last_fullsized":"2024-04-22T01:14:51.749682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748089+0000","last_change":"2024-04-22T01:14:50.782409+0000","last_active":"2024-04-22T01:14:51.748089+0000","last_peered":"2024-04-22T01:14:51.748089+0000","last_clean":"2024-04-22T01:14:51.748089+0000","last_became_active":"2024-04-22T01:14:50.780084+0000","last_became_peered":"2024-04-22T01:14:50.780084+0000","last_unstale":"2024-04-22T01:14:51.748089+0000","last_undegraded":"2024-04-22T01:14:51.748089+0000","last_fullsized":"2024-04-22T01:14:51.748089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444880+0000","last_change":"2024-04-22T01:14:45.708166+0000","last_active":"2024-04-22T01:14:53.444880+0000","last_peered":"2024-04-22T01:14:53.444880+0000","last_clean":"2024-04-22T01:14:53.444880+0000","last_became_active":"2024-04-22T01:14:45.707823+0000","last_became_peered":"2024-04-22T01:14:45.707823+0000","last_unstale":"2024-04-22T01:14:53.444880+0000","last_undegraded":"2024-04-22T01:14:53.444880+0000","last_fullsized":"2024-04-22T01:14:53.444880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750004+0000","last_change":"2024-04-22T01:14:46.692519+0000","last_active":"2024-04-22T01:14:51.750004+0000","last_peered":"2024-04-22T01:14:51.750004+0000","last_clean":"2024-04-22T01:14:51.750004+0000","last_became_active":"2024-04-22T01:14:46.691673+0000","last_became_peered":"2024-04-22T01:14:46.691673+0000","last_unstale":"2024-04-22T01:14:51.750004+0000","last_undegraded":"2024-04-22T01:14:51.750004+0000","last_fullsized":"2024-04-22T01:14:51.750004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745176+0000","last_change":"2024-04-22T01:14:50.778402+0000","last_active":"2024-04-22T01:14:51.745176+0000","last_peered":"2024-04-22T01:14:51.745176+0000","last_clean":"2024-04-22T01:14:51.745176+0000","last_became_active":"2024-04-22T01:14:50.775401+0000","last_became_peered":"2024-04-22T01:14:50.775401+0000","last_unstale":"2024-04-22T01:14:51.745176+0000","last_undegraded":"2024-04-22T01:14:51.745176+0000","last_fullsized":"2024-04-22T01:14:51.745176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:05.161322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940679+0000","last_change":"2024-04-22T01:14:46.717563+0000","last_active":"2024-04-22T01:14:53.940679+0000","last_peered":"2024-04-22T01:14:53.940679+0000","last_clean":"2024-04-22T01:14:53.940679+0000","last_became_active":"2024-04-22T01:14:46.715882+0000","last_became_peered":"2024-04-22T01:14:46.715882+0000","last_unstale":"2024-04-22T01:14:53.940679+0000","last_undegraded":"2024-04-22T01:14:53.940679+0000","last_fullsized":"2024-04-22T01:14:53.940679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445124+0000","last_change":"2024-04-22T01:14:45.704690+0000","last_active":"2024-04-22T01:14:53.445124+0000","last_peered":"2024-04-22T01:14:53.445124+0000","last_clean":"2024-04-22T01:14:53.445124+0000","last_became_active":"2024-04-22T01:14:45.704066+0000","last_became_peered":"2024-04-22T01:14:45.704066+0000","last_unstale":"2024-04-22T01:14:53.445124+0000","last_undegraded":"2024-04-22T01:14:53.445124+0000","last_fullsized":"2024-04-22T01:14:53.445124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:15.442187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747783+0000","last_change":"2024-04-22T01:14:50.785203+0000","last_active":"2024-04-22T01:14:51.747783+0000","last_peered":"2024-04-22T01:14:51.747783+0000","last_clean":"2024-04-22T01:14:51.747783+0000","last_became_active":"2024-04-22T01:14:50.783868+0000","last_became_peered":"2024-04-22T01:14:50.783868+0000","last_unstale":"2024-04-22T01:14:51.747783+0000","last_undegraded":"2024-04-22T01:14:51.747783+0000","last_fullsized":"2024-04-22T01:14:51.747783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917056+0000","last_change":"2024-04-22T01:14:46.695499+0000","last_active":"2024-04-22T01:14:53.917056+0000","last_peered":"2024-04-22T01:14:53.917056+0000","last_clean":"2024-04-22T01:14:53.917056+0000","last_became_active":"2024-04-22T01:14:46.695144+0000","last_became_peered":"2024-04-22T01:14:46.695144+0000","last_unstale":"2024-04-22T01:14:53.917056+0000","last_undegraded":"2024-04-22T01:14:53.917056+0000","last_fullsized":"2024-04-22T01:14:53.917056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865258+0000","last_change":"2024-04-22T01:14:45.689968+0000","last_active":"2024-04-22T01:14:53.865258+0000","last_peered":"2024-04-22T01:14:53.865258+0000","last_clean":"2024-04-22T01:14:53.865258+0000","last_became_active":"2024-04-22T01:14:45.689627+0000","last_became_peered":"2024-04-22T01:14:45.689627+0000","last_unstale":"2024-04-22T01:14:53.865258+0000","last_undegraded":"2024-04-22T01:14:53.865258+0000","last_fullsized":"2024-04-22T01:14:53.865258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746331+0000","last_change":"2024-04-22T01:14:50.777923+0000","last_active":"2024-04-22T01:14:51.746331+0000","last_peered":"2024-04-22T01:14:51.746331+0000","last_clean":"2024-04-22T01:14:51.746331+0000","last_became_active":"2024-04-22T01:14:50.774329+0000","last_became_peered":"2024-04-22T01:14:50.774329+0000","last_unstale":"2024-04-22T01:14:51.746331+0000","last_undegraded":"2024-04-22T01:14:51.746331+0000","last_fullsized":"2024-04-22T01:14:51.746331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:10.629355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916652+0000","last_change":"2024-04-22T01:14:46.698998+0000","last_active":"2024-04-22T01:14:53.916652+0000","last_peered":"2024-04-22T01:14:53.916652+0000","last_clean":"2024-04-22T01:14:53.916652+0000","last_became_active":"2024-04-22T01:14:46.695438+0000","last_became_peered":"2024-04-22T01:14:46.695438+0000","last_unstale":"2024-04-22T01:14:53.916652+0000","last_undegraded":"2024-04-22T01:14:53.916652+0000","last_fullsized":"2024-04-22T01:14:53.916652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863231+0000","last_change":"2024-04-22T01:14:45.699027+0000","last_active":"2024-04-22T01:14:53.863231+0000","last_peered":"2024-04-22T01:14:53.863231+0000","last_clean":"2024-04-22T01:14:53.863231+0000","last_became_active":"2024-04-22T01:14:45.698655+0000","last_became_peered":"2024-04-22T01:14:45.698655+0000","last_unstale":"2024-04-22T01:14:53.863231+0000","last_undegraded":"2024-04-22T01:14:53.863231+0000","last_fullsized":"2024-04-22T01:14:53.863231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865839+0000","last_change":"2024-04-22T01:14:50.930489+0000","last_active":"2024-04-22T01:14:53.865839+0000","last_peered":"2024-04-22T01:14:53.865839+0000","last_clean":"2024-04-22T01:14:53.865839+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.865839+0000","last_undegraded":"2024-04-22T01:14:53.865839+0000","last_fullsized":"2024-04-22T01:14:53.865839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746136+0000","last_change":"2024-04-22T01:14:46.686147+0000","last_active":"2024-04-22T01:14:51.746136+0000","last_peered":"2024-04-22T01:14:51.746136+0000","last_clean":"2024-04-22T01:14:51.746136+0000","last_became_active":"2024-04-22T01:14:46.680316+0000","last_became_peered":"2024-04-22T01:14:46.680316+0000","last_unstale":"2024-04-22T01:14:51.746136+0000","last_undegraded":"2024-04-22T01:14:51.746136+0000","last_fullsized":"2024-04-22T01:14:51.746136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:48.053524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444098+0000","last_change":"2024-04-22T01:14:45.696794+0000","last_active":"2024-04-22T01:14:53.444098+0000","last_peered":"2024-04-22T01:14:53.444098+0000","last_clean":"2024-04-22T01:14:53.444098+0000","last_became_active":"2024-04-22T01:14:45.688246+0000","last_became_peered":"2024-04-22T01:14:45.688246+0000","last_unstale":"2024-04-22T01:14:53.444098+0000","last_undegraded":"2024-04-22T01:14:53.444098+0000","last_fullsized":"2024-04-22T01:14:53.444098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746641+0000","last_change":"2024-04-22T01:14:50.778437+0000","last_active":"2024-04-22T01:14:51.746641+0000","last_peered":"2024-04-22T01:14:51.746641+0000","last_clean":"2024-04-22T01:14:51.746641+0000","last_became_active":"2024-04-22T01:14:50.774011+0000","last_became_peered":"2024-04-22T01:14:50.774011+0000","last_unstale":"2024-04-22T01:14:51.746641+0000","last_undegraded":"2024-04-22T01:14:51.746641+0000","last_fullsized":"2024-04-22T01:14:51.746641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:35.973709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806149+0000","last_change":"2024-04-22T01:14:46.683735+0000","last_active":"2024-04-22T01:14:52.806149+0000","last_peered":"2024-04-22T01:14:52.806149+0000","last_clean":"2024-04-22T01:14:52.806149+0000","last_became_active":"2024-04-22T01:14:46.683395+0000","last_became_peered":"2024-04-22T01:14:46.683395+0000","last_unstale":"2024-04-22T01:14:52.806149+0000","last_undegraded":"2024-04-22T01:14:52.806149+0000","last_fullsized":"2024-04-22T01:14:52.806149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:32:35.242494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939744+0000","last_change":"2024-04-22T01:14:45.690873+0000","last_active":"2024-04-22T01:14:53.939744+0000","last_peered":"2024-04-22T01:14:53.939744+0000","last_clean":"2024-04-22T01:14:53.939744+0000","last_became_active":"2024-04-22T01:14:45.689980+0000","last_became_peered":"2024-04-22T01:14:45.689980+0000","last_unstale":"2024-04-22T01:14:53.939744+0000","last_undegraded":"2024-04-22T01:14:53.939744+0000","last_fullsized":"2024-04-22T01:14:53.939744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745320+0000","last_change":"2024-04-22T01:14:50.934553+0000","last_active":"2024-04-22T01:14:51.745320+0000","last_peered":"2024-04-22T01:14:51.745320+0000","last_clean":"2024-04-22T01:14:51.745320+0000","last_became_active":"2024-04-22T01:14:50.933460+0000","last_became_peered":"2024-04-22T01:14:50.933460+0000","last_unstale":"2024-04-22T01:14:51.745320+0000","last_undegraded":"2024-04-22T01:14:51.745320+0000","last_fullsized":"2024-04-22T01:14:51.745320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:14:00.624653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918403+0000","last_change":"2024-04-22T01:14:46.736500+0000","last_active":"2024-04-22T01:14:53.918403+0000","last_peered":"2024-04-22T01:14:53.918403+0000","last_clean":"2024-04-22T01:14:53.918403+0000","last_became_active":"2024-04-22T01:14:46.736199+0000","last_became_peered":"2024-04-22T01:14:46.736199+0000","last_unstale":"2024-04-22T01:14:53.918403+0000","last_undegraded":"2024-04-22T01:14:53.918403+0000","last_fullsized":"2024-04-22T01:14:53.918403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865748+0000","last_change":"2024-04-22T01:14:45.691876+0000","last_active":"2024-04-22T01:14:53.865748+0000","last_peered":"2024-04-22T01:14:53.865748+0000","last_clean":"2024-04-22T01:14:53.865748+0000","last_became_active":"2024-04-22T01:14:45.691301+0000","last_became_peered":"2024-04-22T01:14:45.691301+0000","last_unstale":"2024-04-22T01:14:53.865748+0000","last_undegraded":"2024-04-22T01:14:53.865748+0000","last_fullsized":"2024-04-22T01:14:53.865748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915922+0000","last_change":"2024-04-22T01:14:50.935981+0000","last_active":"2024-04-22T01:14:53.915922+0000","last_peered":"2024-04-22T01:14:53.915922+0000","last_clean":"2024-04-22T01:14:53.915922+0000","last_became_active":"2024-04-22T01:14:50.935496+0000","last_became_peered":"2024-04-22T01:14:50.935496+0000","last_unstale":"2024-04-22T01:14:53.915922+0000","last_undegraded":"2024-04-22T01:14:53.915922+0000","last_fullsized":"2024-04-22T01:14:53.915922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:24:46.014214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865369+0000","last_change":"2024-04-22T01:14:46.690549+0000","last_active":"2024-04-22T01:14:53.865369+0000","last_peered":"2024-04-22T01:14:53.865369+0000","last_clean":"2024-04-22T01:14:53.865369+0000","last_became_active":"2024-04-22T01:14:46.689835+0000","last_became_peered":"2024-04-22T01:14:46.689835+0000","last_unstale":"2024-04-22T01:14:53.865369+0000","last_undegraded":"2024-04-22T01:14:53.865369+0000","last_fullsized":"2024-04-22T01:14:53.865369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445817+0000","last_change":"2024-04-22T01:14:45.702632+0000","last_active":"2024-04-22T01:14:53.445817+0000","last_peered":"2024-04-22T01:14:53.445817+0000","last_clean":"2024-04-22T01:14:53.445817+0000","last_became_active":"2024-04-22T01:14:45.698621+0000","last_became_peered":"2024-04-22T01:14:45.698621+0000","last_unstale":"2024-04-22T01:14:53.445817+0000","last_undegraded":"2024-04-22T01:14:53.445817+0000","last_fullsized":"2024-04-22T01:14:53.445817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805742+0000","last_change":"2024-04-22T01:14:50.932335+0000","last_active":"2024-04-22T01:14:52.805742+0000","last_peered":"2024-04-22T01:14:52.805742+0000","last_clean":"2024-04-22T01:14:52.805742+0000","last_became_active":"2024-04-22T01:14:50.931895+0000","last_became_peered":"2024-04-22T01:14:50.931895+0000","last_unstale":"2024-04-22T01:14:52.805742+0000","last_undegraded":"2024-04-22T01:14:52.805742+0000","last_fullsized":"2024-04-22T01:14:52.805742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750227+0000","last_change":"2024-04-22T01:14:46.704411+0000","last_active":"2024-04-22T01:14:51.750227+0000","last_peered":"2024-04-22T01:14:51.750227+0000","last_clean":"2024-04-22T01:14:51.750227+0000","last_became_active":"2024-04-22T01:14:46.703696+0000","last_became_peered":"2024-04-22T01:14:46.703696+0000","last_unstale":"2024-04-22T01:14:51.750227+0000","last_undegraded":"2024-04-22T01:14:51.750227+0000","last_fullsized":"2024-04-22T01:14:51.750227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863380+0000","last_change":"2024-04-22T01:14:45.685209+0000","last_active":"2024-04-22T01:14:53.863380+0000","last_peered":"2024-04-22T01:14:53.863380+0000","last_clean":"2024-04-22T01:14:53.863380+0000","last_became_active":"2024-04-22T01:14:45.684801+0000","last_became_peered":"2024-04-22T01:14:45.684801+0000","last_unstale":"2024-04-22T01:14:53.863380+0000","last_undegraded":"2024-04-22T01:14:53.863380+0000","last_fullsized":"2024-04-22T01:14:53.863380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864672+0000","last_change":"2024-04-22T01:14:50.788944+0000","last_active":"2024-04-22T01:14:53.864672+0000","last_peered":"2024-04-22T01:14:53.864672+0000","last_clean":"2024-04-22T01:14:53.864672+0000","last_became_active":"2024-04-22T01:14:50.788405+0000","last_became_peered":"2024-04-22T01:14:50.788405+0000","last_unstale":"2024-04-22T01:14:53.864672+0000","last_undegraded":"2024-04-22T01:14:53.864672+0000","last_fullsized":"2024-04-22T01:14:53.864672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747528+0000","last_change":"2024-04-22T01:14:46.680069+0000","last_active":"2024-04-22T01:14:51.747528+0000","last_peered":"2024-04-22T01:14:51.747528+0000","last_clean":"2024-04-22T01:14:51.747528+0000","last_became_active":"2024-04-22T01:14:46.679311+0000","last_became_peered":"2024-04-22T01:14:46.679311+0000","last_unstale":"2024-04-22T01:14:51.747528+0000","last_undegraded":"2024-04-22T01:14:51.747528+0000","last_fullsized":"2024-04-22T01:14:51.747528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748679+0000","last_change":"2024-04-22T01:14:45.701963+0000","last_active":"2024-04-22T01:14:51.748679+0000","last_peered":"2024-04-22T01:14:51.748679+0000","last_clean":"2024-04-22T01:14:51.748679+0000","last_became_active":"2024-04-22T01:14:45.700246+0000","last_became_peered":"2024-04-22T01:14:45.700246+0000","last_unstale":"2024-04-22T01:14:51.748679+0000","last_undegraded":"2024-04-22T01:14:51.748679+0000","last_fullsized":"2024-04-22T01:14:51.748679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865444+0000","last_change":"2024-04-22T01:14:50.783908+0000","last_active":"2024-04-22T01:14:53.865444+0000","last_peered":"2024-04-22T01:14:53.865444+0000","last_clean":"2024-04-22T01:14:53.865444+0000","last_became_active":"2024-04-22T01:14:50.782537+0000","last_became_peered":"2024-04-22T01:14:50.782537+0000","last_unstale":"2024-04-22T01:14:53.865444+0000","last_undegraded":"2024-04-22T01:14:53.865444+0000","last_fullsized":"2024-04-22T01:14:53.865444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864361+0000","last_change":"2024-04-22T01:14:46.681168+0000","last_active":"2024-04-22T01:14:53.864361+0000","last_peered":"2024-04-22T01:14:53.864361+0000","last_clean":"2024-04-22T01:14:53.864361+0000","last_became_active":"2024-04-22T01:14:46.680888+0000","last_became_peered":"2024-04-22T01:14:46.680888+0000","last_unstale":"2024-04-22T01:14:53.864361+0000","last_undegraded":"2024-04-22T01:14:53.864361+0000","last_fullsized":"2024-04-22T01:14:53.864361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744510+0000","last_change":"2024-04-22T01:14:45.677217+0000","last_active":"2024-04-22T01:14:51.744510+0000","last_peered":"2024-04-22T01:14:51.744510+0000","last_clean":"2024-04-22T01:14:51.744510+0000","last_became_active":"2024-04-22T01:14:45.676630+0000","last_became_peered":"2024-04-22T01:14:45.676630+0000","last_unstale":"2024-04-22T01:14:51.744510+0000","last_undegraded":"2024-04-22T01:14:51.744510+0000","last_fullsized":"2024-04-22T01:14:51.744510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865873+0000","last_change":"2024-04-22T01:14:50.778827+0000","last_active":"2024-04-22T01:14:53.865873+0000","last_peered":"2024-04-22T01:14:53.865873+0000","last_clean":"2024-04-22T01:14:53.865873+0000","last_became_active":"2024-04-22T01:14:50.773642+0000","last_became_peered":"2024-04-22T01:14:50.773642+0000","last_unstale":"2024-04-22T01:14:53.865873+0000","last_undegraded":"2024-04-22T01:14:53.865873+0000","last_fullsized":"2024-04-22T01:14:53.865873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.804659+0000","last_change":"2024-04-22T01:14:46.688078+0000","last_active":"2024-04-22T01:14:52.804659+0000","last_peered":"2024-04-22T01:14:52.804659+0000","last_clean":"2024-04-22T01:14:52.804659+0000","last_became_active":"2024-04-22T01:14:46.687685+0000","last_became_peered":"2024-04-22T01:14:46.687685+0000","last_unstale":"2024-04-22T01:14:52.804659+0000","last_undegraded":"2024-04-22T01:14:52.804659+0000","last_fullsized":"2024-04-22T01:14:52.804659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444194+0000","last_change":"2024-04-22T01:14:45.687324+0000","last_active":"2024-04-22T01:14:53.444194+0000","last_peered":"2024-04-22T01:14:53.444194+0000","last_clean":"2024-04-22T01:14:53.444194+0000","last_became_active":"2024-04-22T01:14:45.686553+0000","last_became_peered":"2024-04-22T01:14:45.686553+0000","last_unstale":"2024-04-22T01:14:53.444194+0000","last_undegraded":"2024-04-22T01:14:53.444194+0000","last_fullsized":"2024-04-22T01:14:53.444194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806076+0000","last_change":"2024-04-22T01:14:50.931597+0000","last_active":"2024-04-22T01:14:52.806076+0000","last_peered":"2024-04-22T01:14:52.806076+0000","last_clean":"2024-04-22T01:14:52.806076+0000","last_became_active":"2024-04-22T01:14:50.931200+0000","last_became_peered":"2024-04-22T01:14:50.931200+0000","last_unstale":"2024-04-22T01:14:52.806076+0000","last_undegraded":"2024-04-22T01:14:52.806076+0000","last_fullsized":"2024-04-22T01:14:52.806076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916581+0000","last_change":"2024-04-22T01:14:46.694352+0000","last_active":"2024-04-22T01:14:53.916581+0000","last_peered":"2024-04-22T01:14:53.916581+0000","last_clean":"2024-04-22T01:14:53.916581+0000","last_became_active":"2024-04-22T01:14:46.693993+0000","last_became_peered":"2024-04-22T01:14:46.693993+0000","last_unstale":"2024-04-22T01:14:53.916581+0000","last_undegraded":"2024-04-22T01:14:53.916581+0000","last_fullsized":"2024-04-22T01:14:53.916581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749278+0000","last_change":"2024-04-22T01:14:45.692182+0000","last_active":"2024-04-22T01:14:51.749278+0000","last_peered":"2024-04-22T01:14:51.749278+0000","last_clean":"2024-04-22T01:14:51.749278+0000","last_became_active":"2024-04-22T01:14:45.670363+0000","last_became_peered":"2024-04-22T01:14:45.670363+0000","last_unstale":"2024-04-22T01:14:51.749278+0000","last_undegraded":"2024-04-22T01:14:51.749278+0000","last_fullsized":"2024-04-22T01:14:51.749278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444406+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:14:53.444406+0000","last_peered":"2024-04-22T01:14:53.444406+0000","last_clean":"2024-04-22T01:14:53.444406+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:14:53.444406+0000","last_undegraded":"2024-04-22T01:14:53.444406+0000","last_fullsized":"2024-04-22T01:14:53.444406+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941257+0000","last_change":"2024-04-22T01:14:50.933466+0000","last_active":"2024-04-22T01:14:53.941257+0000","last_peered":"2024-04-22T01:14:53.941257+0000","last_clean":"2024-04-22T01:14:53.941257+0000","last_became_active":"2024-04-22T01:14:50.932162+0000","last_became_peered":"2024-04-22T01:14:50.932162+0000","last_unstale":"2024-04-22T01:14:53.941257+0000","last_undegraded":"2024-04-22T01:14:53.941257+0000","last_fullsized":"2024-04-22T01:14:53.941257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445273+0000","last_change":"2024-04-22T01:14:46.726237+0000","last_active":"2024-04-22T01:14:53.445273+0000","last_peered":"2024-04-22T01:14:53.445273+0000","last_clean":"2024-04-22T01:14:53.445273+0000","last_became_active":"2024-04-22T01:14:46.725538+0000","last_became_peered":"2024-04-22T01:14:46.725538+0000","last_unstale":"2024-04-22T01:14:53.445273+0000","last_undegraded":"2024-04-22T01:14:53.445273+0000","last_fullsized":"2024-04-22T01:14:53.445273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744593+0000","last_change":"2024-04-22T01:14:45.695208+0000","last_active":"2024-04-22T01:14:51.744593+0000","last_peered":"2024-04-22T01:14:51.744593+0000","last_clean":"2024-04-22T01:14:51.744593+0000","last_became_active":"2024-04-22T01:14:45.694829+0000","last_became_peered":"2024-04-22T01:14:45.694829+0000","last_unstale":"2024-04-22T01:14:51.744593+0000","last_undegraded":"2024-04-22T01:14:51.744593+0000","last_fullsized":"2024-04-22T01:14:51.744593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940759+0000","last_change":"2024-04-22T01:14:50.925049+0000","last_active":"2024-04-22T01:14:53.940759+0000","last_peered":"2024-04-22T01:14:53.940759+0000","last_clean":"2024-04-22T01:14:53.940759+0000","last_became_active":"2024-04-22T01:14:50.923916+0000","last_became_peered":"2024-04-22T01:14:50.923916+0000","last_unstale":"2024-04-22T01:14:53.940759+0000","last_undegraded":"2024-04-22T01:14:53.940759+0000","last_fullsized":"2024-04-22T01:14:53.940759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749772+0000","last_change":"2024-04-22T01:14:46.705928+0000","last_active":"2024-04-22T01:14:51.749772+0000","last_peered":"2024-04-22T01:14:51.749772+0000","last_clean":"2024-04-22T01:14:51.749772+0000","last_became_active":"2024-04-22T01:14:46.703511+0000","last_became_peered":"2024-04-22T01:14:46.703511+0000","last_unstale":"2024-04-22T01:14:51.749772+0000","last_undegraded":"2024-04-22T01:14:51.749772+0000","last_fullsized":"2024-04-22T01:14:51.749772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863430+0000","last_change":"2024-04-22T01:14:45.690308+0000","last_active":"2024-04-22T01:14:53.863430+0000","last_peered":"2024-04-22T01:14:53.863430+0000","last_clean":"2024-04-22T01:14:53.863430+0000","last_became_active":"2024-04-22T01:14:45.689948+0000","last_became_peered":"2024-04-22T01:14:45.689948+0000","last_unstale":"2024-04-22T01:14:53.863430+0000","last_undegraded":"2024-04-22T01:14:53.863430+0000","last_fullsized":"2024-04-22T01:14:53.863430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805877+0000","last_change":"2024-04-22T01:14:50.789559+0000","last_active":"2024-04-22T01:14:52.805877+0000","last_peered":"2024-04-22T01:14:52.805877+0000","last_clean":"2024-04-22T01:14:52.805877+0000","last_became_active":"2024-04-22T01:14:50.789026+0000","last_became_peered":"2024-04-22T01:14:50.789026+0000","last_unstale":"2024-04-22T01:14:52.805877+0000","last_undegraded":"2024-04-22T01:14:52.805877+0000","last_fullsized":"2024-04-22T01:14:52.805877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.446055+0000","last_change":"2024-04-22T01:14:46.734875+0000","last_active":"2024-04-22T01:14:53.446055+0000","last_peered":"2024-04-22T01:14:53.446055+0000","last_clean":"2024-04-22T01:14:53.446055+0000","last_became_active":"2024-04-22T01:14:46.733035+0000","last_became_peered":"2024-04-22T01:14:46.733035+0000","last_unstale":"2024-04-22T01:14:53.446055+0000","last_undegraded":"2024-04-22T01:14:53.446055+0000","last_fullsized":"2024-04-22T01:14:53.446055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744490+0000","last_change":"2024-04-22T01:14:45.692886+0000","last_active":"2024-04-22T01:14:51.744490+0000","last_peered":"2024-04-22T01:14:51.744490+0000","last_clean":"2024-04-22T01:14:51.744490+0000","last_became_active":"2024-04-22T01:14:45.692578+0000","last_became_peered":"2024-04-22T01:14:45.692578+0000","last_unstale":"2024-04-22T01:14:51.744490+0000","last_undegraded":"2024-04-22T01:14:51.744490+0000","last_fullsized":"2024-04-22T01:14:51.744490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745112+0000","last_change":"2024-04-22T01:14:50.925456+0000","last_active":"2024-04-22T01:14:51.745112+0000","last_peered":"2024-04-22T01:14:51.745112+0000","last_clean":"2024-04-22T01:14:51.745112+0000","last_became_active":"2024-04-22T01:14:50.924327+0000","last_became_peered":"2024-04-22T01:14:50.924327+0000","last_unstale":"2024-04-22T01:14:51.745112+0000","last_undegraded":"2024-04-22T01:14:51.745112+0000","last_fullsized":"2024-04-22T01:14:51.745112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445557+0000","last_change":"2024-04-22T01:14:46.711269+0000","last_active":"2024-04-22T01:14:53.445557+0000","last_peered":"2024-04-22T01:14:53.445557+0000","last_clean":"2024-04-22T01:14:53.445557+0000","last_became_active":"2024-04-22T01:14:46.710502+0000","last_became_peered":"2024-04-22T01:14:46.710502+0000","last_unstale":"2024-04-22T01:14:53.445557+0000","last_undegraded":"2024-04-22T01:14:53.445557+0000","last_fullsized":"2024-04-22T01:14:53.445557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746774+0000","last_change":"2024-04-22T01:14:45.679532+0000","last_active":"2024-04-22T01:14:51.746774+0000","last_peered":"2024-04-22T01:14:51.746774+0000","last_clean":"2024-04-22T01:14:51.746774+0000","last_became_active":"2024-04-22T01:14:45.678724+0000","last_became_peered":"2024-04-22T01:14:45.678724+0000","last_unstale":"2024-04-22T01:14:51.746774+0000","last_undegraded":"2024-04-22T01:14:51.746774+0000","last_fullsized":"2024-04-22T01:14:51.746774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:09.261855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744892+0000","last_change":"2024-04-22T01:14:50.780337+0000","last_active":"2024-04-22T01:14:51.744892+0000","last_peered":"2024-04-22T01:14:51.744892+0000","last_clean":"2024-04-22T01:14:51.744892+0000","last_became_active":"2024-04-22T01:14:50.779283+0000","last_became_peered":"2024-04-22T01:14:50.779283+0000","last_unstale":"2024-04-22T01:14:51.744892+0000","last_undegraded":"2024-04-22T01:14:51.744892+0000","last_fullsized":"2024-04-22T01:14:51.744892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:34.466322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444639+0000","last_change":"2024-04-22T01:14:46.719676+0000","last_active":"2024-04-22T01:14:53.444639+0000","last_peered":"2024-04-22T01:14:53.444639+0000","last_clean":"2024-04-22T01:14:53.444639+0000","last_became_active":"2024-04-22T01:14:46.712569+0000","last_became_peered":"2024-04-22T01:14:46.712569+0000","last_unstale":"2024-04-22T01:14:53.444639+0000","last_undegraded":"2024-04-22T01:14:53.444639+0000","last_fullsized":"2024-04-22T01:14:53.444639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744016+0000","last_change":"2024-04-22T01:14:45.681046+0000","last_active":"2024-04-22T01:14:51.744016+0000","last_peered":"2024-04-22T01:14:51.744016+0000","last_clean":"2024-04-22T01:14:51.744016+0000","last_became_active":"2024-04-22T01:14:45.680666+0000","last_became_peered":"2024-04-22T01:14:45.680666+0000","last_unstale":"2024-04-22T01:14:51.744016+0000","last_undegraded":"2024-04-22T01:14:51.744016+0000","last_fullsized":"2024-04-22T01:14:51.744016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805846+0000","last_change":"2024-04-22T01:14:50.767393+0000","last_active":"2024-04-22T01:14:52.805846+0000","last_peered":"2024-04-22T01:14:52.805846+0000","last_clean":"2024-04-22T01:14:52.805846+0000","last_became_active":"2024-04-22T01:14:50.766951+0000","last_became_peered":"2024-04-22T01:14:50.766951+0000","last_unstale":"2024-04-22T01:14:52.805846+0000","last_undegraded":"2024-04-22T01:14:52.805846+0000","last_fullsized":"2024-04-22T01:14:52.805846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940415+0000","last_change":"2024-04-22T01:14:46.706644+0000","last_active":"2024-04-22T01:14:53.940415+0000","last_peered":"2024-04-22T01:14:53.940415+0000","last_clean":"2024-04-22T01:14:53.940415+0000","last_became_active":"2024-04-22T01:14:46.705447+0000","last_became_peered":"2024-04-22T01:14:46.705447+0000","last_unstale":"2024-04-22T01:14:53.940415+0000","last_undegraded":"2024-04-22T01:14:53.940415+0000","last_fullsized":"2024-04-22T01:14:53.940415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939587+0000","last_change":"2024-04-22T01:14:45.692258+0000","last_active":"2024-04-22T01:14:53.939587+0000","last_peered":"2024-04-22T01:14:53.939587+0000","last_clean":"2024-04-22T01:14:53.939587+0000","last_became_active":"2024-04-22T01:14:45.691375+0000","last_became_peered":"2024-04-22T01:14:45.691375+0000","last_unstale":"2024-04-22T01:14:53.939587+0000","last_undegraded":"2024-04-22T01:14:53.939587+0000","last_fullsized":"2024-04-22T01:14:53.939587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918239+0000","last_change":"2024-04-22T01:14:50.932861+0000","last_active":"2024-04-22T01:14:53.918239+0000","last_peered":"2024-04-22T01:14:53.918239+0000","last_clean":"2024-04-22T01:14:53.918239+0000","last_became_active":"2024-04-22T01:14:50.932500+0000","last_became_peered":"2024-04-22T01:14:50.932500+0000","last_unstale":"2024-04-22T01:14:53.918239+0000","last_undegraded":"2024-04-22T01:14:53.918239+0000","last_fullsized":"2024-04-22T01:14:53.918239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:13.452468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445087+0000","last_change":"2024-04-22T01:14:46.733784+0000","last_active":"2024-04-22T01:14:53.445087+0000","last_peered":"2024-04-22T01:14:53.445087+0000","last_clean":"2024-04-22T01:14:53.445087+0000","last_became_active":"2024-04-22T01:14:46.732753+0000","last_became_peered":"2024-04-22T01:14:46.732753+0000","last_unstale":"2024-04-22T01:14:53.445087+0000","last_undegraded":"2024-04-22T01:14:53.445087+0000","last_fullsized":"2024-04-22T01:14:53.445087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750181+0000","last_change":"2024-04-22T01:14:45.706819+0000","last_active":"2024-04-22T01:14:51.750181+0000","last_peered":"2024-04-22T01:14:51.750181+0000","last_clean":"2024-04-22T01:14:51.750181+0000","last_became_active":"2024-04-22T01:14:45.705704+0000","last_became_peered":"2024-04-22T01:14:45.705704+0000","last_unstale":"2024-04-22T01:14:51.750181+0000","last_undegraded":"2024-04-22T01:14:51.750181+0000","last_fullsized":"2024-04-22T01:14:51.750181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917603+0000","last_change":"2024-04-22T01:14:50.925247+0000","last_active":"2024-04-22T01:14:53.917603+0000","last_peered":"2024-04-22T01:14:53.917603+0000","last_clean":"2024-04-22T01:14:53.917603+0000","last_became_active":"2024-04-22T01:14:50.924739+0000","last_became_peered":"2024-04-22T01:14:50.924739+0000","last_unstale":"2024-04-22T01:14:53.917603+0000","last_undegraded":"2024-04-22T01:14:53.917603+0000","last_fullsized":"2024-04-22T01:14:53.917603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:36.206609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805145+0000","last_change":"2024-04-22T01:14:46.686331+0000","last_active":"2024-04-22T01:14:52.805145+0000","last_peered":"2024-04-22T01:14:52.805145+0000","last_clean":"2024-04-22T01:14:52.805145+0000","last_became_active":"2024-04-22T01:14:46.686010+0000","last_became_peered":"2024-04-22T01:14:46.686010+0000","last_unstale":"2024-04-22T01:14:52.805145+0000","last_undegraded":"2024-04-22T01:14:52.805145+0000","last_fullsized":"2024-04-22T01:14:52.805145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806145+0000","last_change":"2024-04-22T01:14:45.686652+0000","last_active":"2024-04-22T01:14:52.806145+0000","last_peered":"2024-04-22T01:14:52.806145+0000","last_clean":"2024-04-22T01:14:52.806145+0000","last_became_active":"2024-04-22T01:14:45.686365+0000","last_became_peered":"2024-04-22T01:14:45.686365+0000","last_unstale":"2024-04-22T01:14:52.806145+0000","last_undegraded":"2024-04-22T01:14:52.806145+0000","last_fullsized":"2024-04-22T01:14:52.806145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749848+0000","last_change":"2024-04-22T01:14:50.925756+0000","last_active":"2024-04-22T01:14:51.749848+0000","last_peered":"2024-04-22T01:14:51.749848+0000","last_clean":"2024-04-22T01:14:51.749848+0000","last_became_active":"2024-04-22T01:14:50.924521+0000","last_became_peered":"2024-04-22T01:14:50.924521+0000","last_unstale":"2024-04-22T01:14:51.749848+0000","last_undegraded":"2024-04-22T01:14:51.749848+0000","last_fullsized":"2024-04-22T01:14:51.749848+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444945+0000","last_change":"2024-04-22T01:14:46.717218+0000","last_active":"2024-04-22T01:14:53.444945+0000","last_peered":"2024-04-22T01:14:53.444945+0000","last_clean":"2024-04-22T01:14:53.444945+0000","last_became_active":"2024-04-22T01:14:46.716342+0000","last_became_peered":"2024-04-22T01:14:46.716342+0000","last_unstale":"2024-04-22T01:14:53.444945+0000","last_undegraded":"2024-04-22T01:14:53.444945+0000","last_fullsized":"2024-04-22T01:14:53.444945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:57.350819+0000","last_change":"2024-04-22T01:14:45.688781+0000","last_active":"2024-04-22T01:14:57.350819+0000","last_peered":"2024-04-22T01:14:57.350819+0000","last_clean":"2024-04-22T01:14:57.350819+0000","last_became_active":"2024-04-22T01:14:45.688153+0000","last_became_peered":"2024-04-22T01:14:45.688153+0000","last_unstale":"2024-04-22T01:14:57.350819+0000","last_undegraded":"2024-04-22T01:14:57.350819+0000","last_fullsized":"2024-04-22T01:14:57.350819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864909+0000","last_change":"2024-04-22T01:14:50.778877+0000","last_active":"2024-04-22T01:14:53.864909+0000","last_peered":"2024-04-22T01:14:53.864909+0000","last_clean":"2024-04-22T01:14:53.864909+0000","last_became_active":"2024-04-22T01:14:50.773623+0000","last_became_peered":"2024-04-22T01:14:50.773623+0000","last_unstale":"2024-04-22T01:14:53.864909+0000","last_undegraded":"2024-04-22T01:14:53.864909+0000","last_fullsized":"2024-04-22T01:14:53.864909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746481+0000","last_change":"2024-04-22T01:14:46.727618+0000","last_active":"2024-04-22T01:14:51.746481+0000","last_peered":"2024-04-22T01:14:51.746481+0000","last_clean":"2024-04-22T01:14:51.746481+0000","last_became_active":"2024-04-22T01:14:46.727055+0000","last_became_peered":"2024-04-22T01:14:46.727055+0000","last_unstale":"2024-04-22T01:14:51.746481+0000","last_undegraded":"2024-04-22T01:14:51.746481+0000","last_fullsized":"2024-04-22T01:14:51.746481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:15.551380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863451+0000","last_change":"2024-04-22T01:14:45.695088+0000","last_active":"2024-04-22T01:14:53.863451+0000","last_peered":"2024-04-22T01:14:53.863451+0000","last_clean":"2024-04-22T01:14:53.863451+0000","last_became_active":"2024-04-22T01:14:45.694765+0000","last_became_peered":"2024-04-22T01:14:45.694765+0000","last_unstale":"2024-04-22T01:14:53.863451+0000","last_undegraded":"2024-04-22T01:14:53.863451+0000","last_fullsized":"2024-04-22T01:14:53.863451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747096+0000","last_change":"2024-04-22T01:14:50.782280+0000","last_active":"2024-04-22T01:14:51.747096+0000","last_peered":"2024-04-22T01:14:51.747096+0000","last_clean":"2024-04-22T01:14:51.747096+0000","last_became_active":"2024-04-22T01:14:50.779944+0000","last_became_peered":"2024-04-22T01:14:50.779944+0000","last_unstale":"2024-04-22T01:14:51.747096+0000","last_undegraded":"2024-04-22T01:14:51.747096+0000","last_fullsized":"2024-04-22T01:14:51.747096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805792+0000","last_change":"2024-04-22T01:14:46.688527+0000","last_active":"2024-04-22T01:14:52.805792+0000","last_peered":"2024-04-22T01:14:52.805792+0000","last_clean":"2024-04-22T01:14:52.805792+0000","last_became_active":"2024-04-22T01:14:46.687989+0000","last_became_peered":"2024-04-22T01:14:46.687989+0000","last_unstale":"2024-04-22T01:14:52.805792+0000","last_undegraded":"2024-04-22T01:14:52.805792+0000","last_fullsized":"2024-04-22T01:14:52.805792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749455+0000","last_change":"2024-04-22T01:14:45.699988+0000","last_active":"2024-04-22T01:14:51.749455+0000","last_peered":"2024-04-22T01:14:51.749455+0000","last_clean":"2024-04-22T01:14:51.749455+0000","last_became_active":"2024-04-22T01:14:45.692830+0000","last_became_peered":"2024-04-22T01:14:45.692830+0000","last_unstale":"2024-04-22T01:14:51.749455+0000","last_undegraded":"2024-04-22T01:14:51.749455+0000","last_fullsized":"2024-04-22T01:14:51.749455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916418+0000","last_change":"2024-04-22T01:14:50.781549+0000","last_active":"2024-04-22T01:14:53.916418+0000","last_peered":"2024-04-22T01:14:53.916418+0000","last_clean":"2024-04-22T01:14:53.916418+0000","last_became_active":"2024-04-22T01:14:50.780774+0000","last_became_peered":"2024-04-22T01:14:50.780774+0000","last_unstale":"2024-04-22T01:14:53.916418+0000","last_undegraded":"2024-04-22T01:14:53.916418+0000","last_fullsized":"2024-04-22T01:14:53.916418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749318+0000","last_change":"2024-04-22T01:14:46.686118+0000","last_active":"2024-04-22T01:14:51.749318+0000","last_peered":"2024-04-22T01:14:51.749318+0000","last_clean":"2024-04-22T01:14:51.749318+0000","last_became_active":"2024-04-22T01:14:46.685329+0000","last_became_peered":"2024-04-22T01:14:46.685329+0000","last_unstale":"2024-04-22T01:14:51.749318+0000","last_undegraded":"2024-04-22T01:14:51.749318+0000","last_fullsized":"2024-04-22T01:14:51.749318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939780+0000","last_change":"2024-04-22T01:14:45.690257+0000","last_active":"2024-04-22T01:14:53.939780+0000","last_peered":"2024-04-22T01:14:53.939780+0000","last_clean":"2024-04-22T01:14:53.939780+0000","last_became_active":"2024-04-22T01:14:45.688794+0000","last_became_peered":"2024-04-22T01:14:45.688794+0000","last_unstale":"2024-04-22T01:14:53.939780+0000","last_undegraded":"2024-04-22T01:14:53.939780+0000","last_fullsized":"2024-04-22T01:14:53.939780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916549+0000","last_change":"2024-04-22T01:14:50.930535+0000","last_active":"2024-04-22T01:14:53.916549+0000","last_peered":"2024-04-22T01:14:53.916549+0000","last_clean":"2024-04-22T01:14:53.916549+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.916549+0000","last_undegraded":"2024-04-22T01:14:53.916549+0000","last_fullsized":"2024-04-22T01:14:53.916549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805736+0000","last_change":"2024-04-22T01:14:50.932710+0000","last_active":"2024-04-22T01:14:52.805736+0000","last_peered":"2024-04-22T01:14:52.805736+0000","last_clean":"2024-04-22T01:14:52.805736+0000","last_became_active":"2024-04-22T01:14:50.931957+0000","last_became_peered":"2024-04-22T01:14:50.931957+0000","last_unstale":"2024-04-22T01:14:52.805736+0000","last_undegraded":"2024-04-22T01:14:52.805736+0000","last_fullsized":"2024-04-22T01:14:52.805736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864231+0000","last_change":"2024-04-22T01:14:46.697192+0000","last_active":"2024-04-22T01:14:53.864231+0000","last_peered":"2024-04-22T01:14:53.864231+0000","last_clean":"2024-04-22T01:14:53.864231+0000","last_became_active":"2024-04-22T01:14:46.696894+0000","last_became_peered":"2024-04-22T01:14:46.696894+0000","last_unstale":"2024-04-22T01:14:53.864231+0000","last_undegraded":"2024-04-22T01:14:53.864231+0000","last_fullsized":"2024-04-22T01:14:53.864231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745866+0000","last_change":"2024-04-22T01:14:45.701127+0000","last_active":"2024-04-22T01:14:51.745866+0000","last_peered":"2024-04-22T01:14:51.745866+0000","last_clean":"2024-04-22T01:14:51.745866+0000","last_became_active":"2024-04-22T01:14:45.699950+0000","last_became_peered":"2024-04-22T01:14:45.699950+0000","last_unstale":"2024-04-22T01:14:51.745866+0000","last_undegraded":"2024-04-22T01:14:51.745866+0000","last_fullsized":"2024-04-22T01:14:51.745866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941786+0000","last_change":"2024-04-22T01:14:50.781112+0000","last_active":"2024-04-22T01:14:53.941786+0000","last_peered":"2024-04-22T01:14:53.941786+0000","last_clean":"2024-04-22T01:14:53.941786+0000","last_became_active":"2024-04-22T01:14:50.780333+0000","last_became_peered":"2024-04-22T01:14:50.780333+0000","last_unstale":"2024-04-22T01:14:53.941786+0000","last_undegraded":"2024-04-22T01:14:53.941786+0000","last_fullsized":"2024-04-22T01:14:53.941786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916408+0000","last_change":"2024-04-22T01:14:46.699240+0000","last_active":"2024-04-22T01:14:53.916408+0000","last_peered":"2024-04-22T01:14:53.916408+0000","last_clean":"2024-04-22T01:14:53.916408+0000","last_became_active":"2024-04-22T01:14:46.698319+0000","last_became_peered":"2024-04-22T01:14:46.698319+0000","last_unstale":"2024-04-22T01:14:53.916408+0000","last_undegraded":"2024-04-22T01:14:53.916408+0000","last_fullsized":"2024-04-22T01:14:53.916408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916909+0000","last_change":"2024-04-22T01:14:45.692605+0000","last_active":"2024-04-22T01:14:53.916909+0000","last_peered":"2024-04-22T01:14:53.916909+0000","last_clean":"2024-04-22T01:14:53.916909+0000","last_became_active":"2024-04-22T01:14:45.692201+0000","last_became_peered":"2024-04-22T01:14:45.692201+0000","last_unstale":"2024-04-22T01:14:53.916909+0000","last_undegraded":"2024-04-22T01:14:53.916909+0000","last_fullsized":"2024-04-22T01:14:53.916909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939598+0000","last_change":"2024-04-22T01:14:45.690896+0000","last_active":"2024-04-22T01:14:53.939598+0000","last_peered":"2024-04-22T01:14:53.939598+0000","last_clean":"2024-04-22T01:14:53.939598+0000","last_became_active":"2024-04-22T01:14:45.689871+0000","last_became_peered":"2024-04-22T01:14:45.689871+0000","last_unstale":"2024-04-22T01:14:53.939598+0000","last_undegraded":"2024-04-22T01:14:53.939598+0000","last_fullsized":"2024-04-22T01:14:53.939598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806029+0000","last_change":"2024-04-22T01:14:46.685575+0000","last_active":"2024-04-22T01:14:52.806029+0000","last_peered":"2024-04-22T01:14:52.806029+0000","last_clean":"2024-04-22T01:14:52.806029+0000","last_became_active":"2024-04-22T01:14:46.685301+0000","last_became_peered":"2024-04-22T01:14:46.685301+0000","last_unstale":"2024-04-22T01:14:52.806029+0000","last_undegraded":"2024-04-22T01:14:52.806029+0000","last_fullsized":"2024-04-22T01:14:52.806029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744699+0000","last_change":"2024-04-22T01:14:50.775646+0000","last_active":"2024-04-22T01:14:51.744699+0000","last_peered":"2024-04-22T01:14:51.744699+0000","last_clean":"2024-04-22T01:14:51.744699+0000","last_became_active":"2024-04-22T01:14:50.775153+0000","last_became_peered":"2024-04-22T01:14:50.775153+0000","last_unstale":"2024-04-22T01:14:51.744699+0000","last_undegraded":"2024-04-22T01:14:51.744699+0000","last_fullsized":"2024-04-22T01:14:51.744699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:46.711296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915399+0000","last_change":"2024-04-22T01:14:45.700287+0000","last_active":"2024-04-22T01:14:53.915399+0000","last_peered":"2024-04-22T01:14:53.915399+0000","last_clean":"2024-04-22T01:14:53.915399+0000","last_became_active":"2024-04-22T01:14:45.700063+0000","last_became_peered":"2024-04-22T01:14:45.700063+0000","last_unstale":"2024-04-22T01:14:53.915399+0000","last_undegraded":"2024-04-22T01:14:53.915399+0000","last_fullsized":"2024-04-22T01:14:53.915399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745633+0000","last_change":"2024-04-22T01:14:46.694633+0000","last_active":"2024-04-22T01:14:51.745633+0000","last_peered":"2024-04-22T01:14:51.745633+0000","last_clean":"2024-04-22T01:14:51.745633+0000","last_became_active":"2024-04-22T01:14:46.690904+0000","last_became_peered":"2024-04-22T01:14:46.690904+0000","last_unstale":"2024-04-22T01:14:51.745633+0000","last_undegraded":"2024-04-22T01:14:51.745633+0000","last_fullsized":"2024-04-22T01:14:51.745633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:35.052516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917858+0000","last_change":"2024-04-22T01:14:50.775487+0000","last_active":"2024-04-22T01:14:53.917858+0000","last_peered":"2024-04-22T01:14:53.917858+0000","last_clean":"2024-04-22T01:14:53.917858+0000","last_became_active":"2024-04-22T01:14:50.774770+0000","last_became_peered":"2024-04-22T01:14:50.774770+0000","last_unstale":"2024-04-22T01:14:53.917858+0000","last_undegraded":"2024-04-22T01:14:53.917858+0000","last_fullsized":"2024-04-22T01:14:53.917858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:59:18.378937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940165+0000","last_change":"2024-04-22T01:14:46.715466+0000","last_active":"2024-04-22T01:14:53.940165+0000","last_peered":"2024-04-22T01:14:53.940165+0000","last_clean":"2024-04-22T01:14:53.940165+0000","last_became_active":"2024-04-22T01:14:46.706194+0000","last_became_peered":"2024-04-22T01:14:46.706194+0000","last_unstale":"2024-04-22T01:14:53.940165+0000","last_undegraded":"2024-04-22T01:14:53.940165+0000","last_fullsized":"2024-04-22T01:14:53.940165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744256+0000","last_change":"2024-04-22T01:14:46.695116+0000","last_active":"2024-04-22T01:14:51.744256+0000","last_peered":"2024-04-22T01:14:51.744256+0000","last_clean":"2024-04-22T01:14:51.744256+0000","last_became_active":"2024-04-22T01:14:46.694817+0000","last_became_peered":"2024-04-22T01:14:46.694817+0000","last_unstale":"2024-04-22T01:14:51.744256+0000","last_undegraded":"2024-04-22T01:14:51.744256+0000","last_fullsized":"2024-04-22T01:14:51.744256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748407+0000","last_change":"2024-04-22T01:14:46.697812+0000","last_active":"2024-04-22T01:14:51.748407+0000","last_peered":"2024-04-22T01:14:51.748407+0000","last_clean":"2024-04-22T01:14:51.748407+0000","last_became_active":"2024-04-22T01:14:46.697056+0000","last_became_peered":"2024-04-22T01:14:46.697056+0000","last_unstale":"2024-04-22T01:14:51.748407+0000","last_undegraded":"2024-04-22T01:14:51.748407+0000","last_fullsized":"2024-04-22T01:14:51.748407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805883+0000","last_change":"2024-04-22T01:14:46.696293+0000","last_active":"2024-04-22T01:14:52.805883+0000","last_peered":"2024-04-22T01:14:52.805883+0000","last_clean":"2024-04-22T01:14:52.805883+0000","last_became_active":"2024-04-22T01:14:46.695950+0000","last_became_peered":"2024-04-22T01:14:46.695950+0000","last_unstale":"2024-04-22T01:14:52.805883+0000","last_undegraded":"2024-04-22T01:14:52.805883+0000","last_fullsized":"2024-04-22T01:14:52.805883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917964+0000","last_change":"2024-04-22T01:14:46.705052+0000","last_active":"2024-04-22T01:14:53.917964+0000","last_peered":"2024-04-22T01:14:53.917964+0000","last_clean":"2024-04-22T01:14:53.917964+0000","last_became_active":"2024-04-22T01:14:46.704239+0000","last_became_peered":"2024-04-22T01:14:46.704239+0000","last_unstale":"2024-04-22T01:14:53.917964+0000","last_undegraded":"2024-04-22T01:14:53.917964+0000","last_fullsized":"2024-04-22T01:14:53.917964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865529+0000","last_change":"2024-04-22T01:14:46.702587+0000","last_active":"2024-04-22T01:14:53.865529+0000","last_peered":"2024-04-22T01:14:53.865529+0000","last_clean":"2024-04-22T01:14:53.865529+0000","last_became_active":"2024-04-22T01:14:46.702257+0000","last_became_peered":"2024-04-22T01:14:46.702257+0000","last_unstale":"2024-04-22T01:14:53.865529+0000","last_undegraded":"2024-04-22T01:14:53.865529+0000","last_fullsized":"2024-04-22T01:14:53.865529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941652+0000","last_change":"2024-04-22T01:14:46.729745+0000","last_active":"2024-04-22T01:14:53.941652+0000","last_peered":"2024-04-22T01:14:53.941652+0000","last_clean":"2024-04-22T01:14:53.941652+0000","last_became_active":"2024-04-22T01:14:46.729165+0000","last_became_peered":"2024-04-22T01:14:46.729165+0000","last_unstale":"2024-04-22T01:14:53.941652+0000","last_undegraded":"2024-04-22T01:14:53.941652+0000","last_fullsized":"2024-04-22T01:14:53.941652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918240+0000","last_change":"2024-04-22T01:14:46.707058+0000","last_active":"2024-04-22T01:14:53.918240+0000","last_peered":"2024-04-22T01:14:53.918240+0000","last_clean":"2024-04-22T01:14:53.918240+0000","last_became_active":"2024-04-22T01:14:46.706772+0000","last_became_peered":"2024-04-22T01:14:46.706772+0000","last_unstale":"2024-04-22T01:14:53.918240+0000","last_undegraded":"2024-04-22T01:14:53.918240+0000","last_fullsized":"2024-04-22T01:14:53.918240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745635+0000","last_change":"2024-04-22T01:14:46.702309+0000","last_active":"2024-04-22T01:14:51.745635+0000","last_peered":"2024-04-22T01:14:51.745635+0000","last_clean":"2024-04-22T01:14:51.745635+0000","last_became_active":"2024-04-22T01:14:46.701437+0000","last_became_peered":"2024-04-22T01:14:46.701437+0000","last_unstale":"2024-04-22T01:14:51.745635+0000","last_undegraded":"2024-04-22T01:14:51.745635+0000","last_fullsized":"2024-04-22T01:14:51.745635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916675+0000","last_change":"2024-04-22T01:14:46.706939+0000","last_active":"2024-04-22T01:14:53.916675+0000","last_peered":"2024-04-22T01:14:53.916675+0000","last_clean":"2024-04-22T01:14:53.916675+0000","last_became_active":"2024-04-22T01:14:46.706649+0000","last_became_peered":"2024-04-22T01:14:46.706649+0000","last_unstale":"2024-04-22T01:14:53.916675+0000","last_undegraded":"2024-04-22T01:14:53.916675+0000","last_fullsized":"2024-04-22T01:14:53.916675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748360+0000","last_change":"2024-04-22T01:14:46.708136+0000","last_active":"2024-04-22T01:14:51.748360+0000","last_peered":"2024-04-22T01:14:51.748360+0000","last_clean":"2024-04-22T01:14:51.748360+0000","last_became_active":"2024-04-22T01:14:46.704481+0000","last_became_peered":"2024-04-22T01:14:46.704481+0000","last_unstale":"2024-04-22T01:14:51.748360+0000","last_undegraded":"2024-04-22T01:14:51.748360+0000","last_fullsized":"2024-04-22T01:14:51.748360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805288+0000","last_change":"2024-04-22T01:14:46.730861+0000","last_active":"2024-04-22T01:14:52.805288+0000","last_peered":"2024-04-22T01:14:52.805288+0000","last_clean":"2024-04-22T01:14:52.805288+0000","last_became_active":"2024-04-22T01:14:46.730520+0000","last_became_peered":"2024-04-22T01:14:46.730520+0000","last_unstale":"2024-04-22T01:14:52.805288+0000","last_undegraded":"2024-04-22T01:14:52.805288+0000","last_fullsized":"2024-04-22T01:14:52.805288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:42.634594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941258+0000","last_change":"2024-04-22T01:14:46.724974+0000","last_active":"2024-04-22T01:14:53.941258+0000","last_peered":"2024-04-22T01:14:53.941258+0000","last_clean":"2024-04-22T01:14:53.941258+0000","last_became_active":"2024-04-22T01:14:46.717427+0000","last_became_peered":"2024-04-22T01:14:46.717427+0000","last_unstale":"2024-04-22T01:14:53.941258+0000","last_undegraded":"2024-04-22T01:14:53.941258+0000","last_fullsized":"2024-04-22T01:14:53.941258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445528+0000","last_change":"2024-04-22T01:14:46.724990+0000","last_active":"2024-04-22T01:14:53.445528+0000","last_peered":"2024-04-22T01:14:53.445528+0000","last_clean":"2024-04-22T01:14:53.445528+0000","last_became_active":"2024-04-22T01:14:46.724307+0000","last_became_peered":"2024-04-22T01:14:46.724307+0000","last_unstale":"2024-04-22T01:14:53.445528+0000","last_undegraded":"2024-04-22T01:14:53.445528+0000","last_fullsized":"2024-04-22T01:14:53.445528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744480+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.744480+0000","last_peered":"2024-04-22T01:14:51.744480+0000","last_clean":"2024-04-22T01:14:51.744480+0000","last_became_active":"2024-04-22T01:14:46.719087+0000","last_became_peered":"2024-04-22T01:14:46.719087+0000","last_unstale":"2024-04-22T01:14:51.744480+0000","last_undegraded":"2024-04-22T01:14:51.744480+0000","last_fullsized":"2024-04-22T01:14:51.744480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749914+0000","last_change":"2024-04-22T01:14:46.726617+0000","last_active":"2024-04-22T01:14:51.749914+0000","last_peered":"2024-04-22T01:14:51.749914+0000","last_clean":"2024-04-22T01:14:51.749914+0000","last_became_active":"2024-04-22T01:14:46.725959+0000","last_became_peered":"2024-04-22T01:14:46.725959+0000","last_unstale":"2024-04-22T01:14:51.749914+0000","last_undegraded":"2024-04-22T01:14:51.749914+0000","last_fullsized":"2024-04-22T01:14:51.749914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745404+0000","last_change":"2024-04-22T01:14:46.678143+0000","last_active":"2024-04-22T01:14:51.745404+0000","last_peered":"2024-04-22T01:14:51.745404+0000","last_clean":"2024-04-22T01:14:51.745404+0000","last_became_active":"2024-04-22T01:14:46.677398+0000","last_became_peered":"2024-04-22T01:14:46.677398+0000","last_unstale":"2024-04-22T01:14:51.745404+0000","last_undegraded":"2024-04-22T01:14:51.745404+0000","last_fullsized":"2024-04-22T01:14:51.745404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864510+0000","last_change":"2024-04-22T01:14:46.697838+0000","last_active":"2024-04-22T01:14:53.864510+0000","last_peered":"2024-04-22T01:14:53.864510+0000","last_clean":"2024-04-22T01:14:53.864510+0000","last_became_active":"2024-04-22T01:14:46.697564+0000","last_became_peered":"2024-04-22T01:14:46.697564+0000","last_unstale":"2024-04-22T01:14:53.864510+0000","last_undegraded":"2024-04-22T01:14:53.864510+0000","last_fullsized":"2024-04-22T01:14:53.864510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805023+0000","last_change":"2024-04-22T01:14:46.679516+0000","last_active":"2024-04-22T01:14:52.805023+0000","last_peered":"2024-04-22T01:14:52.805023+0000","last_clean":"2024-04-22T01:14:52.805023+0000","last_became_active":"2024-04-22T01:14:46.679223+0000","last_became_peered":"2024-04-22T01:14:46.679223+0000","last_unstale":"2024-04-22T01:14:52.805023+0000","last_undegraded":"2024-04-22T01:14:52.805023+0000","last_fullsized":"2024-04-22T01:14:52.805023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916981+0000","last_change":"2024-04-22T01:14:46.709145+0000","last_active":"2024-04-22T01:14:53.916981+0000","last_peered":"2024-04-22T01:14:53.916981+0000","last_clean":"2024-04-22T01:14:53.916981+0000","last_became_active":"2024-04-22T01:14:46.708806+0000","last_became_peered":"2024-04-22T01:14:46.708806+0000","last_unstale":"2024-04-22T01:14:53.916981+0000","last_undegraded":"2024-04-22T01:14:53.916981+0000","last_fullsized":"2024-04-22T01:14:53.916981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746371+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.746371+0000","last_peered":"2024-04-22T01:14:51.746371+0000","last_clean":"2024-04-22T01:14:51.746371+0000","last_became_active":"2024-04-22T01:14:46.718623+0000","last_became_peered":"2024-04-22T01:14:46.718623+0000","last_unstale":"2024-04-22T01:14:51.746371+0000","last_undegraded":"2024-04-22T01:14:51.746371+0000","last_fullsized":"2024-04-22T01:14:51.746371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941253+0000","last_change":"2024-04-22T01:14:46.705553+0000","last_active":"2024-04-22T01:14:53.941253+0000","last_peered":"2024-04-22T01:14:53.941253+0000","last_clean":"2024-04-22T01:14:53.941253+0000","last_became_active":"2024-04-22T01:14:46.704621+0000","last_became_peered":"2024-04-22T01:14:46.704621+0000","last_unstale":"2024-04-22T01:14:53.941253+0000","last_undegraded":"2024-04-22T01:14:53.941253+0000","last_fullsized":"2024-04-22T01:14:53.941253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941576+0000","last_change":"2024-04-22T01:14:46.705656+0000","last_active":"2024-04-22T01:14:53.941576+0000","last_peered":"2024-04-22T01:14:53.941576+0000","last_clean":"2024-04-22T01:14:53.941576+0000","last_became_active":"2024-04-22T01:14:46.704857+0000","last_became_peered":"2024-04-22T01:14:46.704857+0000","last_unstale":"2024-04-22T01:14:53.941576+0000","last_undegraded":"2024-04-22T01:14:53.941576+0000","last_fullsized":"2024-04-22T01:14:53.941576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445983+0000","last_change":"2024-04-22T01:14:46.735811+0000","last_active":"2024-04-22T01:14:53.445983+0000","last_peered":"2024-04-22T01:14:53.445983+0000","last_clean":"2024-04-22T01:14:53.445983+0000","last_became_active":"2024-04-22T01:14:46.727833+0000","last_became_peered":"2024-04-22T01:14:46.727833+0000","last_unstale":"2024-04-22T01:14:53.445983+0000","last_undegraded":"2024-04-22T01:14:53.445983+0000","last_fullsized":"2024-04-22T01:14:53.445983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916197+0000","last_change":"2024-04-22T01:14:46.699497+0000","last_active":"2024-04-22T01:14:53.916197+0000","last_peered":"2024-04-22T01:14:53.916197+0000","last_clean":"2024-04-22T01:14:53.916197+0000","last_became_active":"2024-04-22T01:14:46.698632+0000","last_became_peered":"2024-04-22T01:14:46.698632+0000","last_unstale":"2024-04-22T01:14:53.916197+0000","last_undegraded":"2024-04-22T01:14:53.916197+0000","last_fullsized":"2024-04-22T01:14:53.916197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444562+0000","last_change":"2024-04-22T01:14:46.699917+0000","last_active":"2024-04-22T01:14:53.444562+0000","last_peered":"2024-04-22T01:14:53.444562+0000","last_clean":"2024-04-22T01:14:53.444562+0000","last_became_active":"2024-04-22T01:14:46.684087+0000","last_became_peered":"2024-04-22T01:14:46.684087+0000","last_unstale":"2024-04-22T01:14:53.444562+0000","last_undegraded":"2024-04-22T01:14:53.444562+0000","last_fullsized":"2024-04-22T01:14:53.444562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748148+0000","last_change":"2024-04-22T01:14:46.683759+0000","last_active":"2024-04-22T01:14:51.748148+0000","last_peered":"2024-04-22T01:14:51.748148+0000","last_clean":"2024-04-22T01:14:51.748148+0000","last_became_active":"2024-04-22T01:14:46.682687+0000","last_became_peered":"2024-04-22T01:14:46.682687+0000","last_unstale":"2024-04-22T01:14:51.748148+0000","last_undegraded":"2024-04-22T01:14:51.748148+0000","last_fullsized":"2024-04-22T01:14:51.748148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444815+0000","last_change":"2024-04-22T01:14:46.732970+0000","last_active":"2024-04-22T01:14:53.444815+0000","last_peered":"2024-04-22T01:14:53.444815+0000","last_clean":"2024-04-22T01:14:53.444815+0000","last_became_active":"2024-04-22T01:14:46.727597+0000","last_became_peered":"2024-04-22T01:14:46.727597+0000","last_unstale":"2024-04-22T01:14:53.444815+0000","last_undegraded":"2024-04-22T01:14:53.444815+0000","last_fullsized":"2024-04-22T01:14:53.444815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749390+0000","last_change":"2024-04-22T01:14:46.705305+0000","last_active":"2024-04-22T01:14:51.749390+0000","last_peered":"2024-04-22T01:14:51.749390+0000","last_clean":"2024-04-22T01:14:51.749390+0000","last_became_active":"2024-04-22T01:14:46.704267+0000","last_became_peered":"2024-04-22T01:14:46.704267+0000","last_unstale":"2024-04-22T01:14:51.749390+0000","last_undegraded":"2024-04-22T01:14:51.749390+0000","last_fullsized":"2024-04-22T01:14:51.749390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805327+0000","last_change":"2024-04-22T01:14:46.678217+0000","last_active":"2024-04-22T01:14:52.805327+0000","last_peered":"2024-04-22T01:14:52.805327+0000","last_clean":"2024-04-22T01:14:52.805327+0000","last_became_active":"2024-04-22T01:14:46.677881+0000","last_became_peered":"2024-04-22T01:14:46.677881+0000","last_unstale":"2024-04-22T01:14:52.805327+0000","last_undegraded":"2024-04-22T01:14:52.805327+0000","last_fullsized":"2024-04-22T01:14:52.805327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865661+0000","last_change":"2024-04-22T01:14:46.682640+0000","last_active":"2024-04-22T01:14:53.865661+0000","last_peered":"2024-04-22T01:14:53.865661+0000","last_clean":"2024-04-22T01:14:53.865661+0000","last_became_active":"2024-04-22T01:14:46.682389+0000","last_became_peered":"2024-04-22T01:14:46.682389+0000","last_unstale":"2024-04-22T01:14:53.865661+0000","last_undegraded":"2024-04-22T01:14:53.865661+0000","last_fullsized":"2024-04-22T01:14:53.865661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445838+0000","last_change":"2024-04-22T01:14:46.727060+0000","last_active":"2024-04-22T01:14:53.445838+0000","last_peered":"2024-04-22T01:14:53.445838+0000","last_clean":"2024-04-22T01:14:53.445838+0000","last_became_active":"2024-04-22T01:14:46.726485+0000","last_became_peered":"2024-04-22T01:14:46.726485+0000","last_unstale":"2024-04-22T01:14:53.445838+0000","last_undegraded":"2024-04-22T01:14:53.445838+0000","last_fullsized":"2024-04-22T01:14:53.445838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:15:13.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:15:13.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.646+0000 7f57bd44b640 1 Processor -- start 2024-04-22T01:15:13.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.646+0000 7f57bd44b640 1 -- start start 2024-04-22T01:15:13.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57bd44b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f57b81445a0 0x7f57b80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57bd44b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57b80b2d70 0x7f57b80b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57bd44b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57b80715f0 con 0x7f57b80705d0 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57bd44b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f57b80b3f10 con 0x7f57b81445a0 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57bd44b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f57b80d05f0 con 0x7f57b80b2d70 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f57b81445a0 0x7f57b80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f57b81445a0 0x7f57b80cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54828/0 (socket says 172.21.15.70:54828) 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 -- 172.21.15.70:0/404353821 learned_addr learned my addr 172.21.15.70:0/404353821 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b77fe640 1 --2- 172.21.15.70:0/404353821 >> v2:172.21.15.98:3300/0 conn(0x7f57b80b2d70 0x7f57b80b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 -- 172.21.15.70:0/404353821 >> v2:172.21.15.98:3300/0 conn(0x7f57b80b2d70 msgr2=0x7f57b80b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 --2- 172.21.15.70:0/404353821 >> v2:172.21.15.98:3300/0 conn(0x7f57b80b2d70 0x7f57b80b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 -- 172.21.15.70:0/404353821 >> v1:172.21.15.98:6789/0 conn(0x7f57b80705d0 legacy=0x7f57b80b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 -- 172.21.15.70:0/404353821 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b8071030 con 0x7f57b81445a0 2024-04-22T01:15:13.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b6ffd640 1 --2- 172.21.15.70:0/404353821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b81445a0 0x7f57b80cdec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57b80ce860 tx=0x7f57a801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77601dd4310e1f9f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b67fc640 1 -- 172.21.15.70:0/404353821 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a80b2070 con 0x7f57b81445a0 2024-04-22T01:15:13.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b67fc640 1 -- 172.21.15.70:0/404353821 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a80a9070 con 0x7f57b81445a0 2024-04-22T01:15:13.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.650+0000 7f57b67fc640 1 -- 172.21.15.70:0/404353821 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a80ad880 con 0x7f57b81445a0 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/404353821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b81445a0 msgr2=0x7f57b80cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 --2- 172.21.15.70:0/404353821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b81445a0 0x7f57b80cdec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57b80ce860 tx=0x7f57a801c450 comp rx=0 tx=0).stop 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/404353821 shutdown_connections 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 --2- 172.21.15.70:0/404353821 >> v2:172.21.15.98:3300/0 conn(0x7f57b80b2d70 0x7f57b80b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 --2- 172.21.15.70:0/404353821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b81445a0 0x7f57b80cdec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/404353821 >> 172.21.15.70:0/404353821 conn(0x7f57b806c300 msgr2=0x7f57b806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/404353821 shutdown_connections 2024-04-22T01:15:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/404353821 wait complete. 2024-04-22T01:15:13.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 Processor -- start 2024-04-22T01:15:13.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- start start 2024-04-22T01:15:13.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 0x7f57b81fb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b80b2d70 0x7f57b8156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b81445a0 0x7f57b81376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57b8162890 con 0x7f57b80705d0 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57b8162710 con 0x7f57b80b2d70 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f57b8162a10 con 0x7f57b81445a0 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 0x7f57b81fb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 0x7f57b81fb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54836/0 (socket says 172.21.15.70:54836) 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 -- 172.21.15.70:0/852658158 learned_addr learned my addr 172.21.15.70:0/852658158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57affff640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b80b2d70 0x7f57b8156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 -- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b81445a0 msgr2=0x7f57b81376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b81445a0 0x7f57b81376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 -- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b80b2d70 msgr2=0x7f57b8156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b80b2d70 0x7f57b8156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 -- 172.21.15.70:0/852658158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b81576f0 con 0x7f57b80705d0 2024-04-22T01:15:13.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57affff640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b80b2d70 0x7f57b8156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:13.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57b6ffd640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 0x7f57b81fb3f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57a80b0040 tx=0x7f57a80191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a80b2040 con 0x7f57b80705d0 2024-04-22T01:15:13.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a80add60 con 0x7f57b80705d0 2024-04-22T01:15:13.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b813b6e0 con 0x7f57b80705d0 2024-04-22T01:15:13.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a80b6650 con 0x7f57b80705d0 2024-04-22T01:15:13.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b8157400 con 0x7f57b80705d0 2024-04-22T01:15:13.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.654+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b8068210 con 0x7f57b80705d0 2024-04-22T01:15:13.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.658+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f57a80ab070 con 0x7f57b80705d0 2024-04-22T01:15:13.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.658+0000 7f57af7fe640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57980678c0 0x7f5798069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.658+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f57a811fd10 con 0x7f57b80705d0 2024-04-22T01:15:13.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.658+0000 7f57affff640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57980678c0 0x7f5798069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.658+0000 7f57affff640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57980678c0 0x7f5798069d80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f57b81fb930 tx=0x7f57a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.662+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f57a80160d0 con 0x7f57b80705d0 2024-04-22T01:15:13.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.806+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f57b80d12a0 con 0x7f57980678c0 2024-04-22T01:15:13.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.818+0000 7f57af7fe640 1 -- 172.21.15.70:0/852658158 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f57a0069310 con 0x7f57980678c0 2024-04-22T01:15:13.821 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:13.823 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57980678c0 msgr2=0x7f5798069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57980678c0 0x7f5798069d80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f57b81fb930 tx=0x7f57a0009040 comp rx=0 tx=0).stop 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 msgr2=0x7f57b81fb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 0x7f57b81fb3f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57a80b0040 tx=0x7f57a80191f0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 shutdown_connections 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57980678c0 0x7f5798069d80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b81445a0 0x7f57b81376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b80b2d70 0x7f57b8156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 --2- 172.21.15.70:0/852658158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57b80705d0 0x7f57b81fb3f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 >> 172.21.15.70:0/852658158 conn(0x7f57b806c300 msgr2=0x7f57b815f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 shutdown_connections 2024-04-22T01:15:13.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.822+0000 7f57bd44b640 1 -- 172.21.15.70:0/852658158 wait complete. 2024-04-22T01:15:13.837 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-22T01:15:12.594624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236020,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738700,"statfs":{"total":773094113280,"available":772852428800,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005039"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805910+0000","last_change":"2024-04-22T01:14:45.690356+0000","last_active":"2024-04-22T01:14:52.805910+0000","last_peered":"2024-04-22T01:14:52.805910+0000","last_clean":"2024-04-22T01:14:52.805910+0000","last_became_active":"2024-04-22T01:14:45.690080+0000","last_became_peered":"2024-04-22T01:14:45.690080+0000","last_unstale":"2024-04-22T01:14:52.805910+0000","last_undegraded":"2024-04-22T01:14:52.805910+0000","last_fullsized":"2024-04-22T01:14:52.805910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917875+0000","last_change":"2024-04-22T01:14:46.720257+0000","last_active":"2024-04-22T01:14:53.917875+0000","last_peered":"2024-04-22T01:14:53.917875+0000","last_clean":"2024-04-22T01:14:53.917875+0000","last_became_active":"2024-04-22T01:14:46.720047+0000","last_became_peered":"2024-04-22T01:14:46.720047+0000","last_unstale":"2024-04-22T01:14:53.917875+0000","last_undegraded":"2024-04-22T01:14:53.917875+0000","last_fullsized":"2024-04-22T01:14:53.917875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916284+0000","last_change":"2024-04-22T01:14:50.933593+0000","last_active":"2024-04-22T01:14:53.916284+0000","last_peered":"2024-04-22T01:14:53.916284+0000","last_clean":"2024-04-22T01:14:53.916284+0000","last_became_active":"2024-04-22T01:14:50.933273+0000","last_became_peered":"2024-04-22T01:14:50.933273+0000","last_unstale":"2024-04-22T01:14:53.916284+0000","last_undegraded":"2024-04-22T01:14:53.916284+0000","last_fullsized":"2024-04-22T01:14:53.916284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:46.804215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915513+0000","last_change":"2024-04-22T01:14:45.689486+0000","last_active":"2024-04-22T01:14:53.915513+0000","last_peered":"2024-04-22T01:14:53.915513+0000","last_clean":"2024-04-22T01:14:53.915513+0000","last_became_active":"2024-04-22T01:14:45.689150+0000","last_became_peered":"2024-04-22T01:14:45.689150+0000","last_unstale":"2024-04-22T01:14:53.915513+0000","last_undegraded":"2024-04-22T01:14:53.915513+0000","last_fullsized":"2024-04-22T01:14:53.915513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:57:49.969388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805716+0000","last_change":"2024-04-22T01:14:46.688092+0000","last_active":"2024-04-22T01:14:52.805716+0000","last_peered":"2024-04-22T01:14:52.805716+0000","last_clean":"2024-04-22T01:14:52.805716+0000","last_became_active":"2024-04-22T01:14:46.687812+0000","last_became_peered":"2024-04-22T01:14:46.687812+0000","last_unstale":"2024-04-22T01:14:52.805716+0000","last_undegraded":"2024-04-22T01:14:52.805716+0000","last_fullsized":"2024-04-22T01:14:52.805716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745650+0000","last_change":"2024-04-22T01:14:50.778199+0000","last_active":"2024-04-22T01:14:51.745650+0000","last_peered":"2024-04-22T01:14:51.745650+0000","last_clean":"2024-04-22T01:14:51.745650+0000","last_became_active":"2024-04-22T01:14:50.773891+0000","last_became_peered":"2024-04-22T01:14:50.773891+0000","last_unstale":"2024-04-22T01:14:51.745650+0000","last_undegraded":"2024-04-22T01:14:51.745650+0000","last_fullsized":"2024-04-22T01:14:51.745650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:50.558971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445520+0000","last_change":"2024-04-22T01:14:45.702628+0000","last_active":"2024-04-22T01:14:53.445520+0000","last_peered":"2024-04-22T01:14:53.445520+0000","last_clean":"2024-04-22T01:14:53.445520+0000","last_became_active":"2024-04-22T01:14:45.701650+0000","last_became_peered":"2024-04-22T01:14:45.701650+0000","last_unstale":"2024-04-22T01:14:53.445520+0000","last_undegraded":"2024-04-22T01:14:53.445520+0000","last_fullsized":"2024-04-22T01:14:53.445520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749682+0000","last_change":"2024-04-22T01:14:46.694700+0000","last_active":"2024-04-22T01:14:51.749682+0000","last_peered":"2024-04-22T01:14:51.749682+0000","last_clean":"2024-04-22T01:14:51.749682+0000","last_became_active":"2024-04-22T01:14:46.694050+0000","last_became_peered":"2024-04-22T01:14:46.694050+0000","last_unstale":"2024-04-22T01:14:51.749682+0000","last_undegraded":"2024-04-22T01:14:51.749682+0000","last_fullsized":"2024-04-22T01:14:51.749682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748089+0000","last_change":"2024-04-22T01:14:50.782409+0000","last_active":"2024-04-22T01:14:51.748089+0000","last_peered":"2024-04-22T01:14:51.748089+0000","last_clean":"2024-04-22T01:14:51.748089+0000","last_became_active":"2024-04-22T01:14:50.780084+0000","last_became_peered":"2024-04-22T01:14:50.780084+0000","last_unstale":"2024-04-22T01:14:51.748089+0000","last_undegraded":"2024-04-22T01:14:51.748089+0000","last_fullsized":"2024-04-22T01:14:51.748089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444880+0000","last_change":"2024-04-22T01:14:45.708166+0000","last_active":"2024-04-22T01:14:53.444880+0000","last_peered":"2024-04-22T01:14:53.444880+0000","last_clean":"2024-04-22T01:14:53.444880+0000","last_became_active":"2024-04-22T01:14:45.707823+0000","last_became_peered":"2024-04-22T01:14:45.707823+0000","last_unstale":"2024-04-22T01:14:53.444880+0000","last_undegraded":"2024-04-22T01:14:53.444880+0000","last_fullsized":"2024-04-22T01:14:53.444880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750004+0000","last_change":"2024-04-22T01:14:46.692519+0000","last_active":"2024-04-22T01:14:51.750004+0000","last_peered":"2024-04-22T01:14:51.750004+0000","last_clean":"2024-04-22T01:14:51.750004+0000","last_became_active":"2024-04-22T01:14:46.691673+0000","last_became_peered":"2024-04-22T01:14:46.691673+0000","last_unstale":"2024-04-22T01:14:51.750004+0000","last_undegraded":"2024-04-22T01:14:51.750004+0000","last_fullsized":"2024-04-22T01:14:51.750004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745176+0000","last_change":"2024-04-22T01:14:50.778402+0000","last_active":"2024-04-22T01:14:51.745176+0000","last_peered":"2024-04-22T01:14:51.745176+0000","last_clean":"2024-04-22T01:14:51.745176+0000","last_became_active":"2024-04-22T01:14:50.775401+0000","last_became_peered":"2024-04-22T01:14:50.775401+0000","last_unstale":"2024-04-22T01:14:51.745176+0000","last_undegraded":"2024-04-22T01:14:51.745176+0000","last_fullsized":"2024-04-22T01:14:51.745176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:05.161322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940679+0000","last_change":"2024-04-22T01:14:46.717563+0000","last_active":"2024-04-22T01:14:53.940679+0000","last_peered":"2024-04-22T01:14:53.940679+0000","last_clean":"2024-04-22T01:14:53.940679+0000","last_became_active":"2024-04-22T01:14:46.715882+0000","last_became_peered":"2024-04-22T01:14:46.715882+0000","last_unstale":"2024-04-22T01:14:53.940679+0000","last_undegraded":"2024-04-22T01:14:53.940679+0000","last_fullsized":"2024-04-22T01:14:53.940679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445124+0000","last_change":"2024-04-22T01:14:45.704690+0000","last_active":"2024-04-22T01:14:53.445124+0000","last_peered":"2024-04-22T01:14:53.445124+0000","last_clean":"2024-04-22T01:14:53.445124+0000","last_became_active":"2024-04-22T01:14:45.704066+0000","last_became_peered":"2024-04-22T01:14:45.704066+0000","last_unstale":"2024-04-22T01:14:53.445124+0000","last_undegraded":"2024-04-22T01:14:53.445124+0000","last_fullsized":"2024-04-22T01:14:53.445124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:15.442187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747783+0000","last_change":"2024-04-22T01:14:50.785203+0000","last_active":"2024-04-22T01:14:51.747783+0000","last_peered":"2024-04-22T01:14:51.747783+0000","last_clean":"2024-04-22T01:14:51.747783+0000","last_became_active":"2024-04-22T01:14:50.783868+0000","last_became_peered":"2024-04-22T01:14:50.783868+0000","last_unstale":"2024-04-22T01:14:51.747783+0000","last_undegraded":"2024-04-22T01:14:51.747783+0000","last_fullsized":"2024-04-22T01:14:51.747783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917056+0000","last_change":"2024-04-22T01:14:46.695499+0000","last_active":"2024-04-22T01:14:53.917056+0000","last_peered":"2024-04-22T01:14:53.917056+0000","last_clean":"2024-04-22T01:14:53.917056+0000","last_became_active":"2024-04-22T01:14:46.695144+0000","last_became_peered":"2024-04-22T01:14:46.695144+0000","last_unstale":"2024-04-22T01:14:53.917056+0000","last_undegraded":"2024-04-22T01:14:53.917056+0000","last_fullsized":"2024-04-22T01:14:53.917056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865258+0000","last_change":"2024-04-22T01:14:45.689968+0000","last_active":"2024-04-22T01:14:53.865258+0000","last_peered":"2024-04-22T01:14:53.865258+0000","last_clean":"2024-04-22T01:14:53.865258+0000","last_became_active":"2024-04-22T01:14:45.689627+0000","last_became_peered":"2024-04-22T01:14:45.689627+0000","last_unstale":"2024-04-22T01:14:53.865258+0000","last_undegraded":"2024-04-22T01:14:53.865258+0000","last_fullsized":"2024-04-22T01:14:53.865258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746331+0000","last_change":"2024-04-22T01:14:50.777923+0000","last_active":"2024-04-22T01:14:51.746331+0000","last_peered":"2024-04-22T01:14:51.746331+0000","last_clean":"2024-04-22T01:14:51.746331+0000","last_became_active":"2024-04-22T01:14:50.774329+0000","last_became_peered":"2024-04-22T01:14:50.774329+0000","last_unstale":"2024-04-22T01:14:51.746331+0000","last_undegraded":"2024-04-22T01:14:51.746331+0000","last_fullsized":"2024-04-22T01:14:51.746331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:10.629355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916652+0000","last_change":"2024-04-22T01:14:46.698998+0000","last_active":"2024-04-22T01:14:53.916652+0000","last_peered":"2024-04-22T01:14:53.916652+0000","last_clean":"2024-04-22T01:14:53.916652+0000","last_became_active":"2024-04-22T01:14:46.695438+0000","last_became_peered":"2024-04-22T01:14:46.695438+0000","last_unstale":"2024-04-22T01:14:53.916652+0000","last_undegraded":"2024-04-22T01:14:53.916652+0000","last_fullsized":"2024-04-22T01:14:53.916652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863231+0000","last_change":"2024-04-22T01:14:45.699027+0000","last_active":"2024-04-22T01:14:53.863231+0000","last_peered":"2024-04-22T01:14:53.863231+0000","last_clean":"2024-04-22T01:14:53.863231+0000","last_became_active":"2024-04-22T01:14:45.698655+0000","last_became_peered":"2024-04-22T01:14:45.698655+0000","last_unstale":"2024-04-22T01:14:53.863231+0000","last_undegraded":"2024-04-22T01:14:53.863231+0000","last_fullsized":"2024-04-22T01:14:53.863231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865839+0000","last_change":"2024-04-22T01:14:50.930489+0000","last_active":"2024-04-22T01:14:53.865839+0000","last_peered":"2024-04-22T01:14:53.865839+0000","last_clean":"2024-04-22T01:14:53.865839+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.865839+0000","last_undegraded":"2024-04-22T01:14:53.865839+0000","last_fullsized":"2024-04-22T01:14:53.865839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746136+0000","last_change":"2024-04-22T01:14:46.686147+0000","last_active":"2024-04-22T01:14:51.746136+0000","last_peered":"2024-04-22T01:14:51.746136+0000","last_clean":"2024-04-22T01:14:51.746136+0000","last_became_active":"2024-04-22T01:14:46.680316+0000","last_became_peered":"2024-04-22T01:14:46.680316+0000","last_unstale":"2024-04-22T01:14:51.746136+0000","last_undegraded":"2024-04-22T01:14:51.746136+0000","last_fullsized":"2024-04-22T01:14:51.746136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:48.053524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444098+0000","last_change":"2024-04-22T01:14:45.696794+0000","last_active":"2024-04-22T01:14:53.444098+0000","last_peered":"2024-04-22T01:14:53.444098+0000","last_clean":"2024-04-22T01:14:53.444098+0000","last_became_active":"2024-04-22T01:14:45.688246+0000","last_became_peered":"2024-04-22T01:14:45.688246+0000","last_unstale":"2024-04-22T01:14:53.444098+0000","last_undegraded":"2024-04-22T01:14:53.444098+0000","last_fullsized":"2024-04-22T01:14:53.444098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746641+0000","last_change":"2024-04-22T01:14:50.778437+0000","last_active":"2024-04-22T01:14:51.746641+0000","last_peered":"2024-04-22T01:14:51.746641+0000","last_clean":"2024-04-22T01:14:51.746641+0000","last_became_active":"2024-04-22T01:14:50.774011+0000","last_became_peered":"2024-04-22T01:14:50.774011+0000","last_unstale":"2024-04-22T01:14:51.746641+0000","last_undegraded":"2024-04-22T01:14:51.746641+0000","last_fullsized":"2024-04-22T01:14:51.746641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:35.973709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806149+0000","last_change":"2024-04-22T01:14:46.683735+0000","last_active":"2024-04-22T01:14:52.806149+0000","last_peered":"2024-04-22T01:14:52.806149+0000","last_clean":"2024-04-22T01:14:52.806149+0000","last_became_active":"2024-04-22T01:14:46.683395+0000","last_became_peered":"2024-04-22T01:14:46.683395+0000","last_unstale":"2024-04-22T01:14:52.806149+0000","last_undegraded":"2024-04-22T01:14:52.806149+0000","last_fullsized":"2024-04-22T01:14:52.806149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:32:35.242494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939744+0000","last_change":"2024-04-22T01:14:45.690873+0000","last_active":"2024-04-22T01:14:53.939744+0000","last_peered":"2024-04-22T01:14:53.939744+0000","last_clean":"2024-04-22T01:14:53.939744+0000","last_became_active":"2024-04-22T01:14:45.689980+0000","last_became_peered":"2024-04-22T01:14:45.689980+0000","last_unstale":"2024-04-22T01:14:53.939744+0000","last_undegraded":"2024-04-22T01:14:53.939744+0000","last_fullsized":"2024-04-22T01:14:53.939744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745320+0000","last_change":"2024-04-22T01:14:50.934553+0000","last_active":"2024-04-22T01:14:51.745320+0000","last_peered":"2024-04-22T01:14:51.745320+0000","last_clean":"2024-04-22T01:14:51.745320+0000","last_became_active":"2024-04-22T01:14:50.933460+0000","last_became_peered":"2024-04-22T01:14:50.933460+0000","last_unstale":"2024-04-22T01:14:51.745320+0000","last_undegraded":"2024-04-22T01:14:51.745320+0000","last_fullsized":"2024-04-22T01:14:51.745320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:14:00.624653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918403+0000","last_change":"2024-04-22T01:14:46.736500+0000","last_active":"2024-04-22T01:14:53.918403+0000","last_peered":"2024-04-22T01:14:53.918403+0000","last_clean":"2024-04-22T01:14:53.918403+0000","last_became_active":"2024-04-22T01:14:46.736199+0000","last_became_peered":"2024-04-22T01:14:46.736199+0000","last_unstale":"2024-04-22T01:14:53.918403+0000","last_undegraded":"2024-04-22T01:14:53.918403+0000","last_fullsized":"2024-04-22T01:14:53.918403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865748+0000","last_change":"2024-04-22T01:14:45.691876+0000","last_active":"2024-04-22T01:14:53.865748+0000","last_peered":"2024-04-22T01:14:53.865748+0000","last_clean":"2024-04-22T01:14:53.865748+0000","last_became_active":"2024-04-22T01:14:45.691301+0000","last_became_peered":"2024-04-22T01:14:45.691301+0000","last_unstale":"2024-04-22T01:14:53.865748+0000","last_undegraded":"2024-04-22T01:14:53.865748+0000","last_fullsized":"2024-04-22T01:14:53.865748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:09.675115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915922+0000","last_change":"2024-04-22T01:14:50.935981+0000","last_active":"2024-04-22T01:14:53.915922+0000","last_peered":"2024-04-22T01:14:53.915922+0000","last_clean":"2024-04-22T01:14:53.915922+0000","last_became_active":"2024-04-22T01:14:50.935496+0000","last_became_peered":"2024-04-22T01:14:50.935496+0000","last_unstale":"2024-04-22T01:14:53.915922+0000","last_undegraded":"2024-04-22T01:14:53.915922+0000","last_fullsized":"2024-04-22T01:14:53.915922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:24:46.014214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865369+0000","last_change":"2024-04-22T01:14:46.690549+0000","last_active":"2024-04-22T01:14:53.865369+0000","last_peered":"2024-04-22T01:14:53.865369+0000","last_clean":"2024-04-22T01:14:53.865369+0000","last_became_active":"2024-04-22T01:14:46.689835+0000","last_became_peered":"2024-04-22T01:14:46.689835+0000","last_unstale":"2024-04-22T01:14:53.865369+0000","last_undegraded":"2024-04-22T01:14:53.865369+0000","last_fullsized":"2024-04-22T01:14:53.865369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445817+0000","last_change":"2024-04-22T01:14:45.702632+0000","last_active":"2024-04-22T01:14:53.445817+0000","last_peered":"2024-04-22T01:14:53.445817+0000","last_clean":"2024-04-22T01:14:53.445817+0000","last_became_active":"2024-04-22T01:14:45.698621+0000","last_became_peered":"2024-04-22T01:14:45.698621+0000","last_unstale":"2024-04-22T01:14:53.445817+0000","last_undegraded":"2024-04-22T01:14:53.445817+0000","last_fullsized":"2024-04-22T01:14:53.445817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805742+0000","last_change":"2024-04-22T01:14:50.932335+0000","last_active":"2024-04-22T01:14:52.805742+0000","last_peered":"2024-04-22T01:14:52.805742+0000","last_clean":"2024-04-22T01:14:52.805742+0000","last_became_active":"2024-04-22T01:14:50.931895+0000","last_became_peered":"2024-04-22T01:14:50.931895+0000","last_unstale":"2024-04-22T01:14:52.805742+0000","last_undegraded":"2024-04-22T01:14:52.805742+0000","last_fullsized":"2024-04-22T01:14:52.805742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750227+0000","last_change":"2024-04-22T01:14:46.704411+0000","last_active":"2024-04-22T01:14:51.750227+0000","last_peered":"2024-04-22T01:14:51.750227+0000","last_clean":"2024-04-22T01:14:51.750227+0000","last_became_active":"2024-04-22T01:14:46.703696+0000","last_became_peered":"2024-04-22T01:14:46.703696+0000","last_unstale":"2024-04-22T01:14:51.750227+0000","last_undegraded":"2024-04-22T01:14:51.750227+0000","last_fullsized":"2024-04-22T01:14:51.750227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863380+0000","last_change":"2024-04-22T01:14:45.685209+0000","last_active":"2024-04-22T01:14:53.863380+0000","last_peered":"2024-04-22T01:14:53.863380+0000","last_clean":"2024-04-22T01:14:53.863380+0000","last_became_active":"2024-04-22T01:14:45.684801+0000","last_became_peered":"2024-04-22T01:14:45.684801+0000","last_unstale":"2024-04-22T01:14:53.863380+0000","last_undegraded":"2024-04-22T01:14:53.863380+0000","last_fullsized":"2024-04-22T01:14:53.863380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864672+0000","last_change":"2024-04-22T01:14:50.788944+0000","last_active":"2024-04-22T01:14:53.864672+0000","last_peered":"2024-04-22T01:14:53.864672+0000","last_clean":"2024-04-22T01:14:53.864672+0000","last_became_active":"2024-04-22T01:14:50.788405+0000","last_became_peered":"2024-04-22T01:14:50.788405+0000","last_unstale":"2024-04-22T01:14:53.864672+0000","last_undegraded":"2024-04-22T01:14:53.864672+0000","last_fullsized":"2024-04-22T01:14:53.864672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747528+0000","last_change":"2024-04-22T01:14:46.680069+0000","last_active":"2024-04-22T01:14:51.747528+0000","last_peered":"2024-04-22T01:14:51.747528+0000","last_clean":"2024-04-22T01:14:51.747528+0000","last_became_active":"2024-04-22T01:14:46.679311+0000","last_became_peered":"2024-04-22T01:14:46.679311+0000","last_unstale":"2024-04-22T01:14:51.747528+0000","last_undegraded":"2024-04-22T01:14:51.747528+0000","last_fullsized":"2024-04-22T01:14:51.747528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748679+0000","last_change":"2024-04-22T01:14:45.701963+0000","last_active":"2024-04-22T01:14:51.748679+0000","last_peered":"2024-04-22T01:14:51.748679+0000","last_clean":"2024-04-22T01:14:51.748679+0000","last_became_active":"2024-04-22T01:14:45.700246+0000","last_became_peered":"2024-04-22T01:14:45.700246+0000","last_unstale":"2024-04-22T01:14:51.748679+0000","last_undegraded":"2024-04-22T01:14:51.748679+0000","last_fullsized":"2024-04-22T01:14:51.748679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:11.433086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865444+0000","last_change":"2024-04-22T01:14:50.783908+0000","last_active":"2024-04-22T01:14:53.865444+0000","last_peered":"2024-04-22T01:14:53.865444+0000","last_clean":"2024-04-22T01:14:53.865444+0000","last_became_active":"2024-04-22T01:14:50.782537+0000","last_became_peered":"2024-04-22T01:14:50.782537+0000","last_unstale":"2024-04-22T01:14:53.865444+0000","last_undegraded":"2024-04-22T01:14:53.865444+0000","last_fullsized":"2024-04-22T01:14:53.865444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864361+0000","last_change":"2024-04-22T01:14:46.681168+0000","last_active":"2024-04-22T01:14:53.864361+0000","last_peered":"2024-04-22T01:14:53.864361+0000","last_clean":"2024-04-22T01:14:53.864361+0000","last_became_active":"2024-04-22T01:14:46.680888+0000","last_became_peered":"2024-04-22T01:14:46.680888+0000","last_unstale":"2024-04-22T01:14:53.864361+0000","last_undegraded":"2024-04-22T01:14:53.864361+0000","last_fullsized":"2024-04-22T01:14:53.864361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744510+0000","last_change":"2024-04-22T01:14:45.677217+0000","last_active":"2024-04-22T01:14:51.744510+0000","last_peered":"2024-04-22T01:14:51.744510+0000","last_clean":"2024-04-22T01:14:51.744510+0000","last_became_active":"2024-04-22T01:14:45.676630+0000","last_became_peered":"2024-04-22T01:14:45.676630+0000","last_unstale":"2024-04-22T01:14:51.744510+0000","last_undegraded":"2024-04-22T01:14:51.744510+0000","last_fullsized":"2024-04-22T01:14:51.744510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865873+0000","last_change":"2024-04-22T01:14:50.778827+0000","last_active":"2024-04-22T01:14:53.865873+0000","last_peered":"2024-04-22T01:14:53.865873+0000","last_clean":"2024-04-22T01:14:53.865873+0000","last_became_active":"2024-04-22T01:14:50.773642+0000","last_became_peered":"2024-04-22T01:14:50.773642+0000","last_unstale":"2024-04-22T01:14:53.865873+0000","last_undegraded":"2024-04-22T01:14:53.865873+0000","last_fullsized":"2024-04-22T01:14:53.865873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.804659+0000","last_change":"2024-04-22T01:14:46.688078+0000","last_active":"2024-04-22T01:14:52.804659+0000","last_peered":"2024-04-22T01:14:52.804659+0000","last_clean":"2024-04-22T01:14:52.804659+0000","last_became_active":"2024-04-22T01:14:46.687685+0000","last_became_peered":"2024-04-22T01:14:46.687685+0000","last_unstale":"2024-04-22T01:14:52.804659+0000","last_undegraded":"2024-04-22T01:14:52.804659+0000","last_fullsized":"2024-04-22T01:14:52.804659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444194+0000","last_change":"2024-04-22T01:14:45.687324+0000","last_active":"2024-04-22T01:14:53.444194+0000","last_peered":"2024-04-22T01:14:53.444194+0000","last_clean":"2024-04-22T01:14:53.444194+0000","last_became_active":"2024-04-22T01:14:45.686553+0000","last_became_peered":"2024-04-22T01:14:45.686553+0000","last_unstale":"2024-04-22T01:14:53.444194+0000","last_undegraded":"2024-04-22T01:14:53.444194+0000","last_fullsized":"2024-04-22T01:14:53.444194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806076+0000","last_change":"2024-04-22T01:14:50.931597+0000","last_active":"2024-04-22T01:14:52.806076+0000","last_peered":"2024-04-22T01:14:52.806076+0000","last_clean":"2024-04-22T01:14:52.806076+0000","last_became_active":"2024-04-22T01:14:50.931200+0000","last_became_peered":"2024-04-22T01:14:50.931200+0000","last_unstale":"2024-04-22T01:14:52.806076+0000","last_undegraded":"2024-04-22T01:14:52.806076+0000","last_fullsized":"2024-04-22T01:14:52.806076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:06:52.984950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916581+0000","last_change":"2024-04-22T01:14:46.694352+0000","last_active":"2024-04-22T01:14:53.916581+0000","last_peered":"2024-04-22T01:14:53.916581+0000","last_clean":"2024-04-22T01:14:53.916581+0000","last_became_active":"2024-04-22T01:14:46.693993+0000","last_became_peered":"2024-04-22T01:14:46.693993+0000","last_unstale":"2024-04-22T01:14:53.916581+0000","last_undegraded":"2024-04-22T01:14:53.916581+0000","last_fullsized":"2024-04-22T01:14:53.916581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749278+0000","last_change":"2024-04-22T01:14:45.692182+0000","last_active":"2024-04-22T01:14:51.749278+0000","last_peered":"2024-04-22T01:14:51.749278+0000","last_clean":"2024-04-22T01:14:51.749278+0000","last_became_active":"2024-04-22T01:14:45.670363+0000","last_became_peered":"2024-04-22T01:14:45.670363+0000","last_unstale":"2024-04-22T01:14:51.749278+0000","last_undegraded":"2024-04-22T01:14:51.749278+0000","last_fullsized":"2024-04-22T01:14:51.749278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444406+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:14:53.444406+0000","last_peered":"2024-04-22T01:14:53.444406+0000","last_clean":"2024-04-22T01:14:53.444406+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:14:53.444406+0000","last_undegraded":"2024-04-22T01:14:53.444406+0000","last_fullsized":"2024-04-22T01:14:53.444406+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941257+0000","last_change":"2024-04-22T01:14:50.933466+0000","last_active":"2024-04-22T01:14:53.941257+0000","last_peered":"2024-04-22T01:14:53.941257+0000","last_clean":"2024-04-22T01:14:53.941257+0000","last_became_active":"2024-04-22T01:14:50.932162+0000","last_became_peered":"2024-04-22T01:14:50.932162+0000","last_unstale":"2024-04-22T01:14:53.941257+0000","last_undegraded":"2024-04-22T01:14:53.941257+0000","last_fullsized":"2024-04-22T01:14:53.941257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:37.096507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445273+0000","last_change":"2024-04-22T01:14:46.726237+0000","last_active":"2024-04-22T01:14:53.445273+0000","last_peered":"2024-04-22T01:14:53.445273+0000","last_clean":"2024-04-22T01:14:53.445273+0000","last_became_active":"2024-04-22T01:14:46.725538+0000","last_became_peered":"2024-04-22T01:14:46.725538+0000","last_unstale":"2024-04-22T01:14:53.445273+0000","last_undegraded":"2024-04-22T01:14:53.445273+0000","last_fullsized":"2024-04-22T01:14:53.445273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744593+0000","last_change":"2024-04-22T01:14:45.695208+0000","last_active":"2024-04-22T01:14:51.744593+0000","last_peered":"2024-04-22T01:14:51.744593+0000","last_clean":"2024-04-22T01:14:51.744593+0000","last_became_active":"2024-04-22T01:14:45.694829+0000","last_became_peered":"2024-04-22T01:14:45.694829+0000","last_unstale":"2024-04-22T01:14:51.744593+0000","last_undegraded":"2024-04-22T01:14:51.744593+0000","last_fullsized":"2024-04-22T01:14:51.744593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940759+0000","last_change":"2024-04-22T01:14:50.925049+0000","last_active":"2024-04-22T01:14:53.940759+0000","last_peered":"2024-04-22T01:14:53.940759+0000","last_clean":"2024-04-22T01:14:53.940759+0000","last_became_active":"2024-04-22T01:14:50.923916+0000","last_became_peered":"2024-04-22T01:14:50.923916+0000","last_unstale":"2024-04-22T01:14:53.940759+0000","last_undegraded":"2024-04-22T01:14:53.940759+0000","last_fullsized":"2024-04-22T01:14:53.940759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749772+0000","last_change":"2024-04-22T01:14:46.705928+0000","last_active":"2024-04-22T01:14:51.749772+0000","last_peered":"2024-04-22T01:14:51.749772+0000","last_clean":"2024-04-22T01:14:51.749772+0000","last_became_active":"2024-04-22T01:14:46.703511+0000","last_became_peered":"2024-04-22T01:14:46.703511+0000","last_unstale":"2024-04-22T01:14:51.749772+0000","last_undegraded":"2024-04-22T01:14:51.749772+0000","last_fullsized":"2024-04-22T01:14:51.749772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863430+0000","last_change":"2024-04-22T01:14:45.690308+0000","last_active":"2024-04-22T01:14:53.863430+0000","last_peered":"2024-04-22T01:14:53.863430+0000","last_clean":"2024-04-22T01:14:53.863430+0000","last_became_active":"2024-04-22T01:14:45.689948+0000","last_became_peered":"2024-04-22T01:14:45.689948+0000","last_unstale":"2024-04-22T01:14:53.863430+0000","last_undegraded":"2024-04-22T01:14:53.863430+0000","last_fullsized":"2024-04-22T01:14:53.863430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805877+0000","last_change":"2024-04-22T01:14:50.789559+0000","last_active":"2024-04-22T01:14:52.805877+0000","last_peered":"2024-04-22T01:14:52.805877+0000","last_clean":"2024-04-22T01:14:52.805877+0000","last_became_active":"2024-04-22T01:14:50.789026+0000","last_became_peered":"2024-04-22T01:14:50.789026+0000","last_unstale":"2024-04-22T01:14:52.805877+0000","last_undegraded":"2024-04-22T01:14:52.805877+0000","last_fullsized":"2024-04-22T01:14:52.805877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:09.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.446055+0000","last_change":"2024-04-22T01:14:46.734875+0000","last_active":"2024-04-22T01:14:53.446055+0000","last_peered":"2024-04-22T01:14:53.446055+0000","last_clean":"2024-04-22T01:14:53.446055+0000","last_became_active":"2024-04-22T01:14:46.733035+0000","last_became_peered":"2024-04-22T01:14:46.733035+0000","last_unstale":"2024-04-22T01:14:53.446055+0000","last_undegraded":"2024-04-22T01:14:53.446055+0000","last_fullsized":"2024-04-22T01:14:53.446055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744490+0000","last_change":"2024-04-22T01:14:45.692886+0000","last_active":"2024-04-22T01:14:51.744490+0000","last_peered":"2024-04-22T01:14:51.744490+0000","last_clean":"2024-04-22T01:14:51.744490+0000","last_became_active":"2024-04-22T01:14:45.692578+0000","last_became_peered":"2024-04-22T01:14:45.692578+0000","last_unstale":"2024-04-22T01:14:51.744490+0000","last_undegraded":"2024-04-22T01:14:51.744490+0000","last_fullsized":"2024-04-22T01:14:51.744490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745112+0000","last_change":"2024-04-22T01:14:50.925456+0000","last_active":"2024-04-22T01:14:51.745112+0000","last_peered":"2024-04-22T01:14:51.745112+0000","last_clean":"2024-04-22T01:14:51.745112+0000","last_became_active":"2024-04-22T01:14:50.924327+0000","last_became_peered":"2024-04-22T01:14:50.924327+0000","last_unstale":"2024-04-22T01:14:51.745112+0000","last_undegraded":"2024-04-22T01:14:51.745112+0000","last_fullsized":"2024-04-22T01:14:51.745112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445557+0000","last_change":"2024-04-22T01:14:46.711269+0000","last_active":"2024-04-22T01:14:53.445557+0000","last_peered":"2024-04-22T01:14:53.445557+0000","last_clean":"2024-04-22T01:14:53.445557+0000","last_became_active":"2024-04-22T01:14:46.710502+0000","last_became_peered":"2024-04-22T01:14:46.710502+0000","last_unstale":"2024-04-22T01:14:53.445557+0000","last_undegraded":"2024-04-22T01:14:53.445557+0000","last_fullsized":"2024-04-22T01:14:53.445557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:16.233331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746774+0000","last_change":"2024-04-22T01:14:45.679532+0000","last_active":"2024-04-22T01:14:51.746774+0000","last_peered":"2024-04-22T01:14:51.746774+0000","last_clean":"2024-04-22T01:14:51.746774+0000","last_became_active":"2024-04-22T01:14:45.678724+0000","last_became_peered":"2024-04-22T01:14:45.678724+0000","last_unstale":"2024-04-22T01:14:51.746774+0000","last_undegraded":"2024-04-22T01:14:51.746774+0000","last_fullsized":"2024-04-22T01:14:51.746774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:09.261855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744892+0000","last_change":"2024-04-22T01:14:50.780337+0000","last_active":"2024-04-22T01:14:51.744892+0000","last_peered":"2024-04-22T01:14:51.744892+0000","last_clean":"2024-04-22T01:14:51.744892+0000","last_became_active":"2024-04-22T01:14:50.779283+0000","last_became_peered":"2024-04-22T01:14:50.779283+0000","last_unstale":"2024-04-22T01:14:51.744892+0000","last_undegraded":"2024-04-22T01:14:51.744892+0000","last_fullsized":"2024-04-22T01:14:51.744892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:34.466322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444639+0000","last_change":"2024-04-22T01:14:46.719676+0000","last_active":"2024-04-22T01:14:53.444639+0000","last_peered":"2024-04-22T01:14:53.444639+0000","last_clean":"2024-04-22T01:14:53.444639+0000","last_became_active":"2024-04-22T01:14:46.712569+0000","last_became_peered":"2024-04-22T01:14:46.712569+0000","last_unstale":"2024-04-22T01:14:53.444639+0000","last_undegraded":"2024-04-22T01:14:53.444639+0000","last_fullsized":"2024-04-22T01:14:53.444639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744016+0000","last_change":"2024-04-22T01:14:45.681046+0000","last_active":"2024-04-22T01:14:51.744016+0000","last_peered":"2024-04-22T01:14:51.744016+0000","last_clean":"2024-04-22T01:14:51.744016+0000","last_became_active":"2024-04-22T01:14:45.680666+0000","last_became_peered":"2024-04-22T01:14:45.680666+0000","last_unstale":"2024-04-22T01:14:51.744016+0000","last_undegraded":"2024-04-22T01:14:51.744016+0000","last_fullsized":"2024-04-22T01:14:51.744016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805846+0000","last_change":"2024-04-22T01:14:50.767393+0000","last_active":"2024-04-22T01:14:52.805846+0000","last_peered":"2024-04-22T01:14:52.805846+0000","last_clean":"2024-04-22T01:14:52.805846+0000","last_became_active":"2024-04-22T01:14:50.766951+0000","last_became_peered":"2024-04-22T01:14:50.766951+0000","last_unstale":"2024-04-22T01:14:52.805846+0000","last_undegraded":"2024-04-22T01:14:52.805846+0000","last_fullsized":"2024-04-22T01:14:52.805846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:08.974774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940415+0000","last_change":"2024-04-22T01:14:46.706644+0000","last_active":"2024-04-22T01:14:53.940415+0000","last_peered":"2024-04-22T01:14:53.940415+0000","last_clean":"2024-04-22T01:14:53.940415+0000","last_became_active":"2024-04-22T01:14:46.705447+0000","last_became_peered":"2024-04-22T01:14:46.705447+0000","last_unstale":"2024-04-22T01:14:53.940415+0000","last_undegraded":"2024-04-22T01:14:53.940415+0000","last_fullsized":"2024-04-22T01:14:53.940415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939587+0000","last_change":"2024-04-22T01:14:45.692258+0000","last_active":"2024-04-22T01:14:53.939587+0000","last_peered":"2024-04-22T01:14:53.939587+0000","last_clean":"2024-04-22T01:14:53.939587+0000","last_became_active":"2024-04-22T01:14:45.691375+0000","last_became_peered":"2024-04-22T01:14:45.691375+0000","last_unstale":"2024-04-22T01:14:53.939587+0000","last_undegraded":"2024-04-22T01:14:53.939587+0000","last_fullsized":"2024-04-22T01:14:53.939587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:21.679723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918239+0000","last_change":"2024-04-22T01:14:50.932861+0000","last_active":"2024-04-22T01:14:53.918239+0000","last_peered":"2024-04-22T01:14:53.918239+0000","last_clean":"2024-04-22T01:14:53.918239+0000","last_became_active":"2024-04-22T01:14:50.932500+0000","last_became_peered":"2024-04-22T01:14:50.932500+0000","last_unstale":"2024-04-22T01:14:53.918239+0000","last_undegraded":"2024-04-22T01:14:53.918239+0000","last_fullsized":"2024-04-22T01:14:53.918239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:13.452468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445087+0000","last_change":"2024-04-22T01:14:46.733784+0000","last_active":"2024-04-22T01:14:53.445087+0000","last_peered":"2024-04-22T01:14:53.445087+0000","last_clean":"2024-04-22T01:14:53.445087+0000","last_became_active":"2024-04-22T01:14:46.732753+0000","last_became_peered":"2024-04-22T01:14:46.732753+0000","last_unstale":"2024-04-22T01:14:53.445087+0000","last_undegraded":"2024-04-22T01:14:53.445087+0000","last_fullsized":"2024-04-22T01:14:53.445087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:48:31.897007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.750181+0000","last_change":"2024-04-22T01:14:45.706819+0000","last_active":"2024-04-22T01:14:51.750181+0000","last_peered":"2024-04-22T01:14:51.750181+0000","last_clean":"2024-04-22T01:14:51.750181+0000","last_became_active":"2024-04-22T01:14:45.705704+0000","last_became_peered":"2024-04-22T01:14:45.705704+0000","last_unstale":"2024-04-22T01:14:51.750181+0000","last_undegraded":"2024-04-22T01:14:51.750181+0000","last_fullsized":"2024-04-22T01:14:51.750181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:34.042697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917603+0000","last_change":"2024-04-22T01:14:50.925247+0000","last_active":"2024-04-22T01:14:53.917603+0000","last_peered":"2024-04-22T01:14:53.917603+0000","last_clean":"2024-04-22T01:14:53.917603+0000","last_became_active":"2024-04-22T01:14:50.924739+0000","last_became_peered":"2024-04-22T01:14:50.924739+0000","last_unstale":"2024-04-22T01:14:53.917603+0000","last_undegraded":"2024-04-22T01:14:53.917603+0000","last_fullsized":"2024-04-22T01:14:53.917603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:36.206609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805145+0000","last_change":"2024-04-22T01:14:46.686331+0000","last_active":"2024-04-22T01:14:52.805145+0000","last_peered":"2024-04-22T01:14:52.805145+0000","last_clean":"2024-04-22T01:14:52.805145+0000","last_became_active":"2024-04-22T01:14:46.686010+0000","last_became_peered":"2024-04-22T01:14:46.686010+0000","last_unstale":"2024-04-22T01:14:52.805145+0000","last_undegraded":"2024-04-22T01:14:52.805145+0000","last_fullsized":"2024-04-22T01:14:52.805145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806145+0000","last_change":"2024-04-22T01:14:45.686652+0000","last_active":"2024-04-22T01:14:52.806145+0000","last_peered":"2024-04-22T01:14:52.806145+0000","last_clean":"2024-04-22T01:14:52.806145+0000","last_became_active":"2024-04-22T01:14:45.686365+0000","last_became_peered":"2024-04-22T01:14:45.686365+0000","last_unstale":"2024-04-22T01:14:52.806145+0000","last_undegraded":"2024-04-22T01:14:52.806145+0000","last_fullsized":"2024-04-22T01:14:52.806145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749848+0000","last_change":"2024-04-22T01:14:50.925756+0000","last_active":"2024-04-22T01:14:51.749848+0000","last_peered":"2024-04-22T01:14:51.749848+0000","last_clean":"2024-04-22T01:14:51.749848+0000","last_became_active":"2024-04-22T01:14:50.924521+0000","last_became_peered":"2024-04-22T01:14:50.924521+0000","last_unstale":"2024-04-22T01:14:51.749848+0000","last_undegraded":"2024-04-22T01:14:51.749848+0000","last_fullsized":"2024-04-22T01:14:51.749848+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444945+0000","last_change":"2024-04-22T01:14:46.717218+0000","last_active":"2024-04-22T01:14:53.444945+0000","last_peered":"2024-04-22T01:14:53.444945+0000","last_clean":"2024-04-22T01:14:53.444945+0000","last_became_active":"2024-04-22T01:14:46.716342+0000","last_became_peered":"2024-04-22T01:14:46.716342+0000","last_unstale":"2024-04-22T01:14:53.444945+0000","last_undegraded":"2024-04-22T01:14:53.444945+0000","last_fullsized":"2024-04-22T01:14:53.444945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:57.350819+0000","last_change":"2024-04-22T01:14:45.688781+0000","last_active":"2024-04-22T01:14:57.350819+0000","last_peered":"2024-04-22T01:14:57.350819+0000","last_clean":"2024-04-22T01:14:57.350819+0000","last_became_active":"2024-04-22T01:14:45.688153+0000","last_became_peered":"2024-04-22T01:14:45.688153+0000","last_unstale":"2024-04-22T01:14:57.350819+0000","last_undegraded":"2024-04-22T01:14:57.350819+0000","last_fullsized":"2024-04-22T01:14:57.350819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864909+0000","last_change":"2024-04-22T01:14:50.778877+0000","last_active":"2024-04-22T01:14:53.864909+0000","last_peered":"2024-04-22T01:14:53.864909+0000","last_clean":"2024-04-22T01:14:53.864909+0000","last_became_active":"2024-04-22T01:14:50.773623+0000","last_became_peered":"2024-04-22T01:14:50.773623+0000","last_unstale":"2024-04-22T01:14:53.864909+0000","last_undegraded":"2024-04-22T01:14:53.864909+0000","last_fullsized":"2024-04-22T01:14:53.864909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:44.931605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746481+0000","last_change":"2024-04-22T01:14:46.727618+0000","last_active":"2024-04-22T01:14:51.746481+0000","last_peered":"2024-04-22T01:14:51.746481+0000","last_clean":"2024-04-22T01:14:51.746481+0000","last_became_active":"2024-04-22T01:14:46.727055+0000","last_became_peered":"2024-04-22T01:14:46.727055+0000","last_unstale":"2024-04-22T01:14:51.746481+0000","last_undegraded":"2024-04-22T01:14:51.746481+0000","last_fullsized":"2024-04-22T01:14:51.746481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:15.551380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.863451+0000","last_change":"2024-04-22T01:14:45.695088+0000","last_active":"2024-04-22T01:14:53.863451+0000","last_peered":"2024-04-22T01:14:53.863451+0000","last_clean":"2024-04-22T01:14:53.863451+0000","last_became_active":"2024-04-22T01:14:45.694765+0000","last_became_peered":"2024-04-22T01:14:45.694765+0000","last_unstale":"2024-04-22T01:14:53.863451+0000","last_undegraded":"2024-04-22T01:14:53.863451+0000","last_fullsized":"2024-04-22T01:14:53.863451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.747096+0000","last_change":"2024-04-22T01:14:50.782280+0000","last_active":"2024-04-22T01:14:51.747096+0000","last_peered":"2024-04-22T01:14:51.747096+0000","last_clean":"2024-04-22T01:14:51.747096+0000","last_became_active":"2024-04-22T01:14:50.779944+0000","last_became_peered":"2024-04-22T01:14:50.779944+0000","last_unstale":"2024-04-22T01:14:51.747096+0000","last_undegraded":"2024-04-22T01:14:51.747096+0000","last_fullsized":"2024-04-22T01:14:51.747096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:36:20.310033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805792+0000","last_change":"2024-04-22T01:14:46.688527+0000","last_active":"2024-04-22T01:14:52.805792+0000","last_peered":"2024-04-22T01:14:52.805792+0000","last_clean":"2024-04-22T01:14:52.805792+0000","last_became_active":"2024-04-22T01:14:46.687989+0000","last_became_peered":"2024-04-22T01:14:46.687989+0000","last_unstale":"2024-04-22T01:14:52.805792+0000","last_undegraded":"2024-04-22T01:14:52.805792+0000","last_fullsized":"2024-04-22T01:14:52.805792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749455+0000","last_change":"2024-04-22T01:14:45.699988+0000","last_active":"2024-04-22T01:14:51.749455+0000","last_peered":"2024-04-22T01:14:51.749455+0000","last_clean":"2024-04-22T01:14:51.749455+0000","last_became_active":"2024-04-22T01:14:45.692830+0000","last_became_peered":"2024-04-22T01:14:45.692830+0000","last_unstale":"2024-04-22T01:14:51.749455+0000","last_undegraded":"2024-04-22T01:14:51.749455+0000","last_fullsized":"2024-04-22T01:14:51.749455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916418+0000","last_change":"2024-04-22T01:14:50.781549+0000","last_active":"2024-04-22T01:14:53.916418+0000","last_peered":"2024-04-22T01:14:53.916418+0000","last_clean":"2024-04-22T01:14:53.916418+0000","last_became_active":"2024-04-22T01:14:50.780774+0000","last_became_peered":"2024-04-22T01:14:50.780774+0000","last_unstale":"2024-04-22T01:14:53.916418+0000","last_undegraded":"2024-04-22T01:14:53.916418+0000","last_fullsized":"2024-04-22T01:14:53.916418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:51.586134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749318+0000","last_change":"2024-04-22T01:14:46.686118+0000","last_active":"2024-04-22T01:14:51.749318+0000","last_peered":"2024-04-22T01:14:51.749318+0000","last_clean":"2024-04-22T01:14:51.749318+0000","last_became_active":"2024-04-22T01:14:46.685329+0000","last_became_peered":"2024-04-22T01:14:46.685329+0000","last_unstale":"2024-04-22T01:14:51.749318+0000","last_undegraded":"2024-04-22T01:14:51.749318+0000","last_fullsized":"2024-04-22T01:14:51.749318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939780+0000","last_change":"2024-04-22T01:14:45.690257+0000","last_active":"2024-04-22T01:14:53.939780+0000","last_peered":"2024-04-22T01:14:53.939780+0000","last_clean":"2024-04-22T01:14:53.939780+0000","last_became_active":"2024-04-22T01:14:45.688794+0000","last_became_peered":"2024-04-22T01:14:45.688794+0000","last_unstale":"2024-04-22T01:14:53.939780+0000","last_undegraded":"2024-04-22T01:14:53.939780+0000","last_fullsized":"2024-04-22T01:14:53.939780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:47.043705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916549+0000","last_change":"2024-04-22T01:14:50.930535+0000","last_active":"2024-04-22T01:14:53.916549+0000","last_peered":"2024-04-22T01:14:53.916549+0000","last_clean":"2024-04-22T01:14:53.916549+0000","last_became_active":"2024-04-22T01:14:50.930103+0000","last_became_peered":"2024-04-22T01:14:50.930103+0000","last_unstale":"2024-04-22T01:14:53.916549+0000","last_undegraded":"2024-04-22T01:14:53.916549+0000","last_fullsized":"2024-04-22T01:14:53.916549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:48.417636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805736+0000","last_change":"2024-04-22T01:14:50.932710+0000","last_active":"2024-04-22T01:14:52.805736+0000","last_peered":"2024-04-22T01:14:52.805736+0000","last_clean":"2024-04-22T01:14:52.805736+0000","last_became_active":"2024-04-22T01:14:50.931957+0000","last_became_peered":"2024-04-22T01:14:50.931957+0000","last_unstale":"2024-04-22T01:14:52.805736+0000","last_undegraded":"2024-04-22T01:14:52.805736+0000","last_fullsized":"2024-04-22T01:14:52.805736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864231+0000","last_change":"2024-04-22T01:14:46.697192+0000","last_active":"2024-04-22T01:14:53.864231+0000","last_peered":"2024-04-22T01:14:53.864231+0000","last_clean":"2024-04-22T01:14:53.864231+0000","last_became_active":"2024-04-22T01:14:46.696894+0000","last_became_peered":"2024-04-22T01:14:46.696894+0000","last_unstale":"2024-04-22T01:14:53.864231+0000","last_undegraded":"2024-04-22T01:14:53.864231+0000","last_fullsized":"2024-04-22T01:14:53.864231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745866+0000","last_change":"2024-04-22T01:14:45.701127+0000","last_active":"2024-04-22T01:14:51.745866+0000","last_peered":"2024-04-22T01:14:51.745866+0000","last_clean":"2024-04-22T01:14:51.745866+0000","last_became_active":"2024-04-22T01:14:45.699950+0000","last_became_peered":"2024-04-22T01:14:45.699950+0000","last_unstale":"2024-04-22T01:14:51.745866+0000","last_undegraded":"2024-04-22T01:14:51.745866+0000","last_fullsized":"2024-04-22T01:14:51.745866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:28:43.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941786+0000","last_change":"2024-04-22T01:14:50.781112+0000","last_active":"2024-04-22T01:14:53.941786+0000","last_peered":"2024-04-22T01:14:53.941786+0000","last_clean":"2024-04-22T01:14:53.941786+0000","last_became_active":"2024-04-22T01:14:50.780333+0000","last_became_peered":"2024-04-22T01:14:50.780333+0000","last_unstale":"2024-04-22T01:14:53.941786+0000","last_undegraded":"2024-04-22T01:14:53.941786+0000","last_fullsized":"2024-04-22T01:14:53.941786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:04.402183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916408+0000","last_change":"2024-04-22T01:14:46.699240+0000","last_active":"2024-04-22T01:14:53.916408+0000","last_peered":"2024-04-22T01:14:53.916408+0000","last_clean":"2024-04-22T01:14:53.916408+0000","last_became_active":"2024-04-22T01:14:46.698319+0000","last_became_peered":"2024-04-22T01:14:46.698319+0000","last_unstale":"2024-04-22T01:14:53.916408+0000","last_undegraded":"2024-04-22T01:14:53.916408+0000","last_fullsized":"2024-04-22T01:14:53.916408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916909+0000","last_change":"2024-04-22T01:14:45.692605+0000","last_active":"2024-04-22T01:14:53.916909+0000","last_peered":"2024-04-22T01:14:53.916909+0000","last_clean":"2024-04-22T01:14:53.916909+0000","last_became_active":"2024-04-22T01:14:45.692201+0000","last_became_peered":"2024-04-22T01:14:45.692201+0000","last_unstale":"2024-04-22T01:14:53.916909+0000","last_undegraded":"2024-04-22T01:14:53.916909+0000","last_fullsized":"2024-04-22T01:14:53.916909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:49:16.518570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.939598+0000","last_change":"2024-04-22T01:14:45.690896+0000","last_active":"2024-04-22T01:14:53.939598+0000","last_peered":"2024-04-22T01:14:53.939598+0000","last_clean":"2024-04-22T01:14:53.939598+0000","last_became_active":"2024-04-22T01:14:45.689871+0000","last_became_peered":"2024-04-22T01:14:45.689871+0000","last_unstale":"2024-04-22T01:14:53.939598+0000","last_undegraded":"2024-04-22T01:14:53.939598+0000","last_fullsized":"2024-04-22T01:14:53.939598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:23:21.713229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.806029+0000","last_change":"2024-04-22T01:14:46.685575+0000","last_active":"2024-04-22T01:14:52.806029+0000","last_peered":"2024-04-22T01:14:52.806029+0000","last_clean":"2024-04-22T01:14:52.806029+0000","last_became_active":"2024-04-22T01:14:46.685301+0000","last_became_peered":"2024-04-22T01:14:46.685301+0000","last_unstale":"2024-04-22T01:14:52.806029+0000","last_undegraded":"2024-04-22T01:14:52.806029+0000","last_fullsized":"2024-04-22T01:14:52.806029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:25.348663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744699+0000","last_change":"2024-04-22T01:14:50.775646+0000","last_active":"2024-04-22T01:14:51.744699+0000","last_peered":"2024-04-22T01:14:51.744699+0000","last_clean":"2024-04-22T01:14:51.744699+0000","last_became_active":"2024-04-22T01:14:50.775153+0000","last_became_peered":"2024-04-22T01:14:50.775153+0000","last_unstale":"2024-04-22T01:14:51.744699+0000","last_undegraded":"2024-04-22T01:14:51.744699+0000","last_fullsized":"2024-04-22T01:14:51.744699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:46.711296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.915399+0000","last_change":"2024-04-22T01:14:45.700287+0000","last_active":"2024-04-22T01:14:53.915399+0000","last_peered":"2024-04-22T01:14:53.915399+0000","last_clean":"2024-04-22T01:14:53.915399+0000","last_became_active":"2024-04-22T01:14:45.700063+0000","last_became_peered":"2024-04-22T01:14:45.700063+0000","last_unstale":"2024-04-22T01:14:53.915399+0000","last_undegraded":"2024-04-22T01:14:53.915399+0000","last_fullsized":"2024-04-22T01:14:53.915399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:44.640283+0000","last_clean_scrub_stamp":"2024-04-22T01:14:44.640283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:28:52.693628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745633+0000","last_change":"2024-04-22T01:14:46.694633+0000","last_active":"2024-04-22T01:14:51.745633+0000","last_peered":"2024-04-22T01:14:51.745633+0000","last_clean":"2024-04-22T01:14:51.745633+0000","last_became_active":"2024-04-22T01:14:46.690904+0000","last_became_peered":"2024-04-22T01:14:46.690904+0000","last_unstale":"2024-04-22T01:14:51.745633+0000","last_undegraded":"2024-04-22T01:14:51.745633+0000","last_fullsized":"2024-04-22T01:14:51.745633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:35.052516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917858+0000","last_change":"2024-04-22T01:14:50.775487+0000","last_active":"2024-04-22T01:14:53.917858+0000","last_peered":"2024-04-22T01:14:53.917858+0000","last_clean":"2024-04-22T01:14:53.917858+0000","last_became_active":"2024-04-22T01:14:50.774770+0000","last_became_peered":"2024-04-22T01:14:50.774770+0000","last_unstale":"2024-04-22T01:14:53.917858+0000","last_undegraded":"2024-04-22T01:14:53.917858+0000","last_fullsized":"2024-04-22T01:14:53.917858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:49.726804+0000","last_clean_scrub_stamp":"2024-04-22T01:14:49.726804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:59:18.378937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.940165+0000","last_change":"2024-04-22T01:14:46.715466+0000","last_active":"2024-04-22T01:14:53.940165+0000","last_peered":"2024-04-22T01:14:53.940165+0000","last_clean":"2024-04-22T01:14:53.940165+0000","last_became_active":"2024-04-22T01:14:46.706194+0000","last_became_peered":"2024-04-22T01:14:46.706194+0000","last_unstale":"2024-04-22T01:14:53.940165+0000","last_undegraded":"2024-04-22T01:14:53.940165+0000","last_fullsized":"2024-04-22T01:14:53.940165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744256+0000","last_change":"2024-04-22T01:14:46.695116+0000","last_active":"2024-04-22T01:14:51.744256+0000","last_peered":"2024-04-22T01:14:51.744256+0000","last_clean":"2024-04-22T01:14:51.744256+0000","last_became_active":"2024-04-22T01:14:46.694817+0000","last_became_peered":"2024-04-22T01:14:46.694817+0000","last_unstale":"2024-04-22T01:14:51.744256+0000","last_undegraded":"2024-04-22T01:14:51.744256+0000","last_fullsized":"2024-04-22T01:14:51.744256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748407+0000","last_change":"2024-04-22T01:14:46.697812+0000","last_active":"2024-04-22T01:14:51.748407+0000","last_peered":"2024-04-22T01:14:51.748407+0000","last_clean":"2024-04-22T01:14:51.748407+0000","last_became_active":"2024-04-22T01:14:46.697056+0000","last_became_peered":"2024-04-22T01:14:46.697056+0000","last_unstale":"2024-04-22T01:14:51.748407+0000","last_undegraded":"2024-04-22T01:14:51.748407+0000","last_fullsized":"2024-04-22T01:14:51.748407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805883+0000","last_change":"2024-04-22T01:14:46.696293+0000","last_active":"2024-04-22T01:14:52.805883+0000","last_peered":"2024-04-22T01:14:52.805883+0000","last_clean":"2024-04-22T01:14:52.805883+0000","last_became_active":"2024-04-22T01:14:46.695950+0000","last_became_peered":"2024-04-22T01:14:46.695950+0000","last_unstale":"2024-04-22T01:14:52.805883+0000","last_undegraded":"2024-04-22T01:14:52.805883+0000","last_fullsized":"2024-04-22T01:14:52.805883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.917964+0000","last_change":"2024-04-22T01:14:46.705052+0000","last_active":"2024-04-22T01:14:53.917964+0000","last_peered":"2024-04-22T01:14:53.917964+0000","last_clean":"2024-04-22T01:14:53.917964+0000","last_became_active":"2024-04-22T01:14:46.704239+0000","last_became_peered":"2024-04-22T01:14:46.704239+0000","last_unstale":"2024-04-22T01:14:53.917964+0000","last_undegraded":"2024-04-22T01:14:53.917964+0000","last_fullsized":"2024-04-22T01:14:53.917964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865529+0000","last_change":"2024-04-22T01:14:46.702587+0000","last_active":"2024-04-22T01:14:53.865529+0000","last_peered":"2024-04-22T01:14:53.865529+0000","last_clean":"2024-04-22T01:14:53.865529+0000","last_became_active":"2024-04-22T01:14:46.702257+0000","last_became_peered":"2024-04-22T01:14:46.702257+0000","last_unstale":"2024-04-22T01:14:53.865529+0000","last_undegraded":"2024-04-22T01:14:53.865529+0000","last_fullsized":"2024-04-22T01:14:53.865529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941652+0000","last_change":"2024-04-22T01:14:46.729745+0000","last_active":"2024-04-22T01:14:53.941652+0000","last_peered":"2024-04-22T01:14:53.941652+0000","last_clean":"2024-04-22T01:14:53.941652+0000","last_became_active":"2024-04-22T01:14:46.729165+0000","last_became_peered":"2024-04-22T01:14:46.729165+0000","last_unstale":"2024-04-22T01:14:53.941652+0000","last_undegraded":"2024-04-22T01:14:53.941652+0000","last_fullsized":"2024-04-22T01:14:53.941652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.918240+0000","last_change":"2024-04-22T01:14:46.707058+0000","last_active":"2024-04-22T01:14:53.918240+0000","last_peered":"2024-04-22T01:14:53.918240+0000","last_clean":"2024-04-22T01:14:53.918240+0000","last_became_active":"2024-04-22T01:14:46.706772+0000","last_became_peered":"2024-04-22T01:14:46.706772+0000","last_unstale":"2024-04-22T01:14:53.918240+0000","last_undegraded":"2024-04-22T01:14:53.918240+0000","last_fullsized":"2024-04-22T01:14:53.918240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745635+0000","last_change":"2024-04-22T01:14:46.702309+0000","last_active":"2024-04-22T01:14:51.745635+0000","last_peered":"2024-04-22T01:14:51.745635+0000","last_clean":"2024-04-22T01:14:51.745635+0000","last_became_active":"2024-04-22T01:14:46.701437+0000","last_became_peered":"2024-04-22T01:14:46.701437+0000","last_unstale":"2024-04-22T01:14:51.745635+0000","last_undegraded":"2024-04-22T01:14:51.745635+0000","last_fullsized":"2024-04-22T01:14:51.745635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916675+0000","last_change":"2024-04-22T01:14:46.706939+0000","last_active":"2024-04-22T01:14:53.916675+0000","last_peered":"2024-04-22T01:14:53.916675+0000","last_clean":"2024-04-22T01:14:53.916675+0000","last_became_active":"2024-04-22T01:14:46.706649+0000","last_became_peered":"2024-04-22T01:14:46.706649+0000","last_unstale":"2024-04-22T01:14:53.916675+0000","last_undegraded":"2024-04-22T01:14:53.916675+0000","last_fullsized":"2024-04-22T01:14:53.916675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748360+0000","last_change":"2024-04-22T01:14:46.708136+0000","last_active":"2024-04-22T01:14:51.748360+0000","last_peered":"2024-04-22T01:14:51.748360+0000","last_clean":"2024-04-22T01:14:51.748360+0000","last_became_active":"2024-04-22T01:14:46.704481+0000","last_became_peered":"2024-04-22T01:14:46.704481+0000","last_unstale":"2024-04-22T01:14:51.748360+0000","last_undegraded":"2024-04-22T01:14:51.748360+0000","last_fullsized":"2024-04-22T01:14:51.748360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:13:04.898072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805288+0000","last_change":"2024-04-22T01:14:46.730861+0000","last_active":"2024-04-22T01:14:52.805288+0000","last_peered":"2024-04-22T01:14:52.805288+0000","last_clean":"2024-04-22T01:14:52.805288+0000","last_became_active":"2024-04-22T01:14:46.730520+0000","last_became_peered":"2024-04-22T01:14:46.730520+0000","last_unstale":"2024-04-22T01:14:52.805288+0000","last_undegraded":"2024-04-22T01:14:52.805288+0000","last_fullsized":"2024-04-22T01:14:52.805288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:42.634594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941258+0000","last_change":"2024-04-22T01:14:46.724974+0000","last_active":"2024-04-22T01:14:53.941258+0000","last_peered":"2024-04-22T01:14:53.941258+0000","last_clean":"2024-04-22T01:14:53.941258+0000","last_became_active":"2024-04-22T01:14:46.717427+0000","last_became_peered":"2024-04-22T01:14:46.717427+0000","last_unstale":"2024-04-22T01:14:53.941258+0000","last_undegraded":"2024-04-22T01:14:53.941258+0000","last_fullsized":"2024-04-22T01:14:53.941258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:50.094942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445528+0000","last_change":"2024-04-22T01:14:46.724990+0000","last_active":"2024-04-22T01:14:53.445528+0000","last_peered":"2024-04-22T01:14:53.445528+0000","last_clean":"2024-04-22T01:14:53.445528+0000","last_became_active":"2024-04-22T01:14:46.724307+0000","last_became_peered":"2024-04-22T01:14:46.724307+0000","last_unstale":"2024-04-22T01:14:53.445528+0000","last_undegraded":"2024-04-22T01:14:53.445528+0000","last_fullsized":"2024-04-22T01:14:53.445528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:33.019805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.744480+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.744480+0000","last_peered":"2024-04-22T01:14:51.744480+0000","last_clean":"2024-04-22T01:14:51.744480+0000","last_became_active":"2024-04-22T01:14:46.719087+0000","last_became_peered":"2024-04-22T01:14:46.719087+0000","last_unstale":"2024-04-22T01:14:51.744480+0000","last_undegraded":"2024-04-22T01:14:51.744480+0000","last_fullsized":"2024-04-22T01:14:51.744480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:12.442905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749914+0000","last_change":"2024-04-22T01:14:46.726617+0000","last_active":"2024-04-22T01:14:51.749914+0000","last_peered":"2024-04-22T01:14:51.749914+0000","last_clean":"2024-04-22T01:14:51.749914+0000","last_became_active":"2024-04-22T01:14:46.725959+0000","last_became_peered":"2024-04-22T01:14:46.725959+0000","last_unstale":"2024-04-22T01:14:51.749914+0000","last_undegraded":"2024-04-22T01:14:51.749914+0000","last_fullsized":"2024-04-22T01:14:51.749914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:05.654566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.745404+0000","last_change":"2024-04-22T01:14:46.678143+0000","last_active":"2024-04-22T01:14:51.745404+0000","last_peered":"2024-04-22T01:14:51.745404+0000","last_clean":"2024-04-22T01:14:51.745404+0000","last_became_active":"2024-04-22T01:14:46.677398+0000","last_became_peered":"2024-04-22T01:14:46.677398+0000","last_unstale":"2024-04-22T01:14:51.745404+0000","last_undegraded":"2024-04-22T01:14:51.745404+0000","last_fullsized":"2024-04-22T01:14:51.745404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.864510+0000","last_change":"2024-04-22T01:14:46.697838+0000","last_active":"2024-04-22T01:14:53.864510+0000","last_peered":"2024-04-22T01:14:53.864510+0000","last_clean":"2024-04-22T01:14:53.864510+0000","last_became_active":"2024-04-22T01:14:46.697564+0000","last_became_peered":"2024-04-22T01:14:46.697564+0000","last_unstale":"2024-04-22T01:14:53.864510+0000","last_undegraded":"2024-04-22T01:14:53.864510+0000","last_fullsized":"2024-04-22T01:14:53.864510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805023+0000","last_change":"2024-04-22T01:14:46.679516+0000","last_active":"2024-04-22T01:14:52.805023+0000","last_peered":"2024-04-22T01:14:52.805023+0000","last_clean":"2024-04-22T01:14:52.805023+0000","last_became_active":"2024-04-22T01:14:46.679223+0000","last_became_peered":"2024-04-22T01:14:46.679223+0000","last_unstale":"2024-04-22T01:14:52.805023+0000","last_undegraded":"2024-04-22T01:14:52.805023+0000","last_fullsized":"2024-04-22T01:14:52.805023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:30:12.795597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916981+0000","last_change":"2024-04-22T01:14:46.709145+0000","last_active":"2024-04-22T01:14:53.916981+0000","last_peered":"2024-04-22T01:14:53.916981+0000","last_clean":"2024-04-22T01:14:53.916981+0000","last_became_active":"2024-04-22T01:14:46.708806+0000","last_became_peered":"2024-04-22T01:14:46.708806+0000","last_unstale":"2024-04-22T01:14:53.916981+0000","last_undegraded":"2024-04-22T01:14:53.916981+0000","last_fullsized":"2024-04-22T01:14:53.916981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:10:52.005704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.746371+0000","last_change":"2024-04-22T01:14:46.719308+0000","last_active":"2024-04-22T01:14:51.746371+0000","last_peered":"2024-04-22T01:14:51.746371+0000","last_clean":"2024-04-22T01:14:51.746371+0000","last_became_active":"2024-04-22T01:14:46.718623+0000","last_became_peered":"2024-04-22T01:14:46.718623+0000","last_unstale":"2024-04-22T01:14:51.746371+0000","last_undegraded":"2024-04-22T01:14:51.746371+0000","last_fullsized":"2024-04-22T01:14:51.746371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:10.684934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941253+0000","last_change":"2024-04-22T01:14:46.705553+0000","last_active":"2024-04-22T01:14:53.941253+0000","last_peered":"2024-04-22T01:14:53.941253+0000","last_clean":"2024-04-22T01:14:53.941253+0000","last_became_active":"2024-04-22T01:14:46.704621+0000","last_became_peered":"2024-04-22T01:14:46.704621+0000","last_unstale":"2024-04-22T01:14:53.941253+0000","last_undegraded":"2024-04-22T01:14:53.941253+0000","last_fullsized":"2024-04-22T01:14:53.941253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:59.565039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.941576+0000","last_change":"2024-04-22T01:14:46.705656+0000","last_active":"2024-04-22T01:14:53.941576+0000","last_peered":"2024-04-22T01:14:53.941576+0000","last_clean":"2024-04-22T01:14:53.941576+0000","last_became_active":"2024-04-22T01:14:46.704857+0000","last_became_peered":"2024-04-22T01:14:46.704857+0000","last_unstale":"2024-04-22T01:14:53.941576+0000","last_undegraded":"2024-04-22T01:14:53.941576+0000","last_fullsized":"2024-04-22T01:14:53.941576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:02.845001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445983+0000","last_change":"2024-04-22T01:14:46.735811+0000","last_active":"2024-04-22T01:14:53.445983+0000","last_peered":"2024-04-22T01:14:53.445983+0000","last_clean":"2024-04-22T01:14:53.445983+0000","last_became_active":"2024-04-22T01:14:46.727833+0000","last_became_peered":"2024-04-22T01:14:46.727833+0000","last_unstale":"2024-04-22T01:14:53.445983+0000","last_undegraded":"2024-04-22T01:14:53.445983+0000","last_fullsized":"2024-04-22T01:14:53.445983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:52:40.854903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.916197+0000","last_change":"2024-04-22T01:14:46.699497+0000","last_active":"2024-04-22T01:14:53.916197+0000","last_peered":"2024-04-22T01:14:53.916197+0000","last_clean":"2024-04-22T01:14:53.916197+0000","last_became_active":"2024-04-22T01:14:46.698632+0000","last_became_peered":"2024-04-22T01:14:46.698632+0000","last_unstale":"2024-04-22T01:14:53.916197+0000","last_undegraded":"2024-04-22T01:14:53.916197+0000","last_fullsized":"2024-04-22T01:14:53.916197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:11:03.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444562+0000","last_change":"2024-04-22T01:14:46.699917+0000","last_active":"2024-04-22T01:14:53.444562+0000","last_peered":"2024-04-22T01:14:53.444562+0000","last_clean":"2024-04-22T01:14:53.444562+0000","last_became_active":"2024-04-22T01:14:46.684087+0000","last_became_peered":"2024-04-22T01:14:46.684087+0000","last_unstale":"2024-04-22T01:14:53.444562+0000","last_undegraded":"2024-04-22T01:14:53.444562+0000","last_fullsized":"2024-04-22T01:14:53.444562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:01.084620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.748148+0000","last_change":"2024-04-22T01:14:46.683759+0000","last_active":"2024-04-22T01:14:51.748148+0000","last_peered":"2024-04-22T01:14:51.748148+0000","last_clean":"2024-04-22T01:14:51.748148+0000","last_became_active":"2024-04-22T01:14:46.682687+0000","last_became_peered":"2024-04-22T01:14:46.682687+0000","last_unstale":"2024-04-22T01:14:51.748148+0000","last_undegraded":"2024-04-22T01:14:51.748148+0000","last_fullsized":"2024-04-22T01:14:51.748148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:40:46.482269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.444815+0000","last_change":"2024-04-22T01:14:46.732970+0000","last_active":"2024-04-22T01:14:53.444815+0000","last_peered":"2024-04-22T01:14:53.444815+0000","last_clean":"2024-04-22T01:14:53.444815+0000","last_became_active":"2024-04-22T01:14:46.727597+0000","last_became_peered":"2024-04-22T01:14:46.727597+0000","last_unstale":"2024-04-22T01:14:53.444815+0000","last_undegraded":"2024-04-22T01:14:53.444815+0000","last_fullsized":"2024-04-22T01:14:53.444815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:06.552653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:51.749390+0000","last_change":"2024-04-22T01:14:46.705305+0000","last_active":"2024-04-22T01:14:51.749390+0000","last_peered":"2024-04-22T01:14:51.749390+0000","last_clean":"2024-04-22T01:14:51.749390+0000","last_became_active":"2024-04-22T01:14:46.704267+0000","last_became_peered":"2024-04-22T01:14:46.704267+0000","last_unstale":"2024-04-22T01:14:51.749390+0000","last_undegraded":"2024-04-22T01:14:51.749390+0000","last_fullsized":"2024-04-22T01:14:51.749390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:56.547951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:52.805327+0000","last_change":"2024-04-22T01:14:46.678217+0000","last_active":"2024-04-22T01:14:52.805327+0000","last_peered":"2024-04-22T01:14:52.805327+0000","last_clean":"2024-04-22T01:14:52.805327+0000","last_became_active":"2024-04-22T01:14:46.677881+0000","last_became_peered":"2024-04-22T01:14:46.677881+0000","last_unstale":"2024-04-22T01:14:52.805327+0000","last_undegraded":"2024-04-22T01:14:52.805327+0000","last_fullsized":"2024-04-22T01:14:52.805327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.865661+0000","last_change":"2024-04-22T01:14:46.682640+0000","last_active":"2024-04-22T01:14:53.865661+0000","last_peered":"2024-04-22T01:14:53.865661+0000","last_clean":"2024-04-22T01:14:53.865661+0000","last_became_active":"2024-04-22T01:14:46.682389+0000","last_became_peered":"2024-04-22T01:14:46.682389+0000","last_unstale":"2024-04-22T01:14:53.865661+0000","last_undegraded":"2024-04-22T01:14:53.865661+0000","last_fullsized":"2024-04-22T01:14:53.865661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:30.389620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-22T01:14:53.445838+0000","last_change":"2024-04-22T01:14:46.727060+0000","last_active":"2024-04-22T01:14:53.445838+0000","last_peered":"2024-04-22T01:14:53.445838+0000","last_clean":"2024-04-22T01:14:53.445838+0000","last_became_active":"2024-04-22T01:14:46.726485+0000","last_became_peered":"2024-04-22T01:14:46.726485+0000","last_unstale":"2024-04-22T01:14:53.445838+0000","last_undegraded":"2024-04-22T01:14:53.445838+0000","last_fullsized":"2024-04-22T01:14:53.445838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:45.650102+0000","last_clean_scrub_stamp":"2024-04-22T01:14:45.650102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:00.325481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:15:13.842 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T01:15:13.842 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T01:15:13.842 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T01:15:13.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T01:15:13.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 Processor -- start 2024-04-22T01:15:13.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 -- start start 2024-04-22T01:15:13.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8b40b3cd0 0x7fb8b4072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b406ed40 0x7fb8b4161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:13.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b415bf20 con 0x7fb8b415bb40 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b406e8b0 con 0x7fb8b406ed40 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8bb5ba640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b40aec30 con 0x7fb8b40b3cd0 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b406ed40 0x7fb8b4161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b406ed40 0x7fb8b4161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54848/0 (socket says 172.21.15.70:54848) 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 -- 172.21.15.70:0/1460322850 learned_addr learned my addr 172.21.15.70:0/1460322850 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b8b55640 1 --2- 172.21.15.70:0/1460322850 >> v2:172.21.15.98:3300/0 conn(0x7fb8b40b3cd0 0x7fb8b4072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 -- 172.21.15.70:0/1460322850 >> v2:172.21.15.98:3300/0 conn(0x7fb8b40b3cd0 msgr2=0x7fb8b4072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 --2- 172.21.15.70:0/1460322850 >> v2:172.21.15.98:3300/0 conn(0x7fb8b40b3cd0 0x7fb8b4072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 -- 172.21.15.70:0/1460322850 >> v1:172.21.15.98:6789/0 conn(0x7fb8b415bb40 legacy=0x7fb8b40b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:13.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.994+0000 7fb8b9b57640 1 -- 172.21.15.70:0/1460322850 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b40aef10 con 0x7fb8b406ed40 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8b9b57640 1 --2- 172.21.15.70:0/1460322850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b406ed40 0x7fb8b4161ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40b2d10 tx=0x7fb8ac01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6630ab007196f14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8abfff640 1 -- 172.21.15.70:0/1460322850 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0a5840 con 0x7fb8b406ed40 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8abfff640 1 -- 172.21.15.70:0/1460322850 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ac0a59e0 con 0x7fb8b406ed40 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8abfff640 1 -- 172.21.15.70:0/1460322850 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0a5db0 con 0x7fb8b406ed40 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/1460322850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b406ed40 msgr2=0x7fb8b4161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/1460322850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b406ed40 0x7fb8b4161ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40b2d10 tx=0x7fb8ac01c980 comp rx=0 tx=0).stop 2024-04-22T01:15:13.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/1460322850 shutdown_connections 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/1460322850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b406ed40 0x7fb8b4161ac0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/1460322850 >> v2:172.21.15.98:3300/0 conn(0x7fb8b40b3cd0 0x7fb8b4072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/1460322850 >> 172.21.15.70:0/1460322850 conn(0x7fb8b406b1f0 msgr2=0x7fb8b406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/1460322850 shutdown_connections 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/1460322850 wait complete. 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 Processor -- start 2024-04-22T01:15:13.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:13.998+0000 7fb8bb5ba640 1 -- start start 2024-04-22T01:15:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b406ed40 0x7fb8b40b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 0x7fb8b40b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b415bb40 0x7fb8b4147760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b406f740 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b406f5c0 con 0x7fb8b415bb40 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b416ae60 con 0x7fb8b406ed40 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 0x7fb8b40b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 0x7fb8b40b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54860/0 (socket says 172.21.15.70:54860) 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 -- 172.21.15.70:0/2826524888 learned_addr learned my addr 172.21.15.70:0/2826524888 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b9356640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b406ed40 0x7fb8b40b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 -- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b406ed40 msgr2=0x7fb8b40b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b406ed40 0x7fb8b40b0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 -- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b415bb40 msgr2=0x7fb8b4147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b9b57640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b415bb40 0x7fb8b4147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b415bb40 0x7fb8b4147760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 -- 172.21.15.70:0/2826524888 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b414ba80 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b8b55640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 0x7fb8b40b1fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c820 tx=0x7fb8a400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4015070 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a4004240 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b414acc0 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400ce10 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b414b280 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b4068210 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb8a4007830 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8a9ffb640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8980678c0 0x7fb898069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b9356640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8980678c0 0x7fb898069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb8a4085e10 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.002+0000 7fb8b9356640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8980678c0 0x7fb898069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40cef00 tx=0x7fb8b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.006+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb8a40566a0 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb8b406e040 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8a9ffb640 1 -- 172.21.15.70:0/2826524888 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb8a4055fa0 con 0x7fb8b40b3cd0 2024-04-22T01:15:14.189 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:14.189 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8980678c0 msgr2=0x7fb898069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8980678c0 0x7fb898069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40cef00 tx=0x7fb8b0008040 comp rx=0 tx=0).stop 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 msgr2=0x7fb8b40b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 0x7fb8b40b1fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c820 tx=0x7fb8a400cce0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 shutdown_connections 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8980678c0 0x7fb898069d80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b415bb40 0x7fb8b4147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b40b3cd0 0x7fb8b40b1fe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 --2- 172.21.15.70:0/2826524888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b406ed40 0x7fb8b40b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 >> 172.21.15.70:0/2826524888 conn(0x7fb8b406b1f0 msgr2=0x7fb8b406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 shutdown_connections 2024-04-22T01:15:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.190+0000 7fb8bb5ba640 1 -- 172.21.15.70:0/2826524888 wait complete. 2024-04-22T01:15:14.206 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T01:15:14.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:14.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 Processor -- start 2024-04-22T01:15:14.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 -- start start 2024-04-22T01:15:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1201455b0 0x7ff1200cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1200cc150 0x7ff120071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff120072060 0x7ff12015f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff1200724d0 con 0x7ff1200cc150 2024-04-22T01:15:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff1200d0250 con 0x7ff120072060 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff12562f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff1200cd7b0 con 0x7ff1201455b0 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1200cc150 0x7ff120071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1200cc150 0x7ff120071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54872/0 (socket says 172.21.15.70:54872) 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 -- 172.21.15.70:0/986302507 learned_addr learned my addr 172.21.15.70:0/986302507 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/986302507 >> v2:172.21.15.98:3300/0 conn(0x7ff120072060 0x7ff12015f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff11effd640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1201455b0 0x7ff1200cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 -- 172.21.15.70:0/986302507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1201455b0 msgr2=0x7ff1200cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1201455b0 0x7ff1200cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 -- 172.21.15.70:0/986302507 >> v2:172.21.15.98:3300/0 conn(0x7ff120072060 msgr2=0x7ff12015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 --2- 172.21.15.70:0/986302507 >> v2:172.21.15.98:3300/0 conn(0x7ff120072060 0x7ff12015f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 -- 172.21.15.70:0/986302507 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1200cc710 con 0x7ff1200cc150 2024-04-22T01:15:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff11effd640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1201455b0 0x7ff1200cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.322+0000 7ff1167fc640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cc150 0x7ff120071b20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1200b1580 tx=0x7ff10801c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1f8f5aa57c697a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11e7fc640 1 -- 172.21.15.70:0/986302507 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1080b1070 con 0x7ff1200cc150 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11e7fc640 1 -- 172.21.15.70:0/986302507 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1080ac440 con 0x7ff1200cc150 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11e7fc640 1 -- 172.21.15.70:0/986302507 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1080ac810 con 0x7ff1200cc150 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/986302507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cc150 msgr2=0x7ff120071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cc150 0x7ff120071b20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1200b1580 tx=0x7ff10801c140 comp rx=0 tx=0).stop 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/986302507 shutdown_connections 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- 172.21.15.70:0/986302507 >> v2:172.21.15.98:3300/0 conn(0x7ff120072060 0x7ff12015f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cc150 0x7ff120071b20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- 172.21.15.70:0/986302507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1201455b0 0x7ff1200cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/986302507 >> 172.21.15.70:0/986302507 conn(0x7ff12006b1f0 msgr2=0x7ff12006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/986302507 shutdown_connections 2024-04-22T01:15:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/986302507 wait complete. 2024-04-22T01:15:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 Processor -- start 2024-04-22T01:15:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- start start 2024-04-22T01:15:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 0x7ff1201f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200cc150 0x7ff120129320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1201455b0 0x7ff120143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff12006e380 con 0x7ff120072060 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff12006e200 con 0x7ff1201455b0 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff12006e500 con 0x7ff1200cc150 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 0x7ff1201f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 0x7ff1201f81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54884/0 (socket says 172.21.15.70:54884) 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 -- 172.21.15.70:0/4113672724 learned_addr learned my addr 172.21.15.70:0/4113672724 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff1167fc640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200cc150 0x7ff120129320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1201455b0 0x7ff120143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 -- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200cc150 msgr2=0x7ff120129320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200cc150 0x7ff120129320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 -- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1201455b0 msgr2=0x7ff120143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1201455b0 0x7ff120143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 -- 172.21.15.70:0/4113672724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff120129860 con 0x7ff120072060 2024-04-22T01:15:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1201455b0 0x7ff120143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff11effd640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 0x7ff1201f81d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff11000c7f0 tx=0x7ff11000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110015070 con 0x7ff120072060 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff110007820 con 0x7ff120072060 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12013ae30 con 0x7ff120072060 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110007c10 con 0x7ff120072060 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.326+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12013b3f0 con 0x7ff120072060 2024-04-22T01:15:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.330+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff12013bb10 con 0x7ff120072060 2024-04-22T01:15:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.330+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff110018070 con 0x7ff120072060 2024-04-22T01:15:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.330+0000 7ff117fff640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.330+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff110086070 con 0x7ff120072060 2024-04-22T01:15:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.330+0000 7ff1167fc640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.330+0000 7ff1167fc640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff1080ae000 tx=0x7ff1080a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.334+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff110056900 con 0x7ff120072060 2024-04-22T01:15:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.482+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff120068210 con 0x7ff120072060 2024-04-22T01:15:14.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.482+0000 7ff117fff640 1 -- 172.21.15.70:0/4113672724 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7ff110056200 con 0x7ff120072060 2024-04-22T01:15:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:14.484 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 msgr2=0x7ff100069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff1080ae000 tx=0x7ff1080a6070 comp rx=0 tx=0).stop 2024-04-22T01:15:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 msgr2=0x7ff1201f81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 0x7ff1201f81d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff11000c7f0 tx=0x7ff11000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 shutdown_connections 2024-04-22T01:15:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1201455b0 0x7ff120143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200cc150 0x7ff120129320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 --2- 172.21.15.70:0/4113672724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff120072060 0x7ff1201f81d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 >> 172.21.15.70:0/4113672724 conn(0x7ff12006b1f0 msgr2=0x7ff12015cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 shutdown_connections 2024-04-22T01:15:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.486+0000 7ff12562f640 1 -- 172.21.15.70:0/4113672724 wait complete. 2024-04-22T01:15:14.501 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:14.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.610+0000 7f74270aa640 1 Processor -- start 2024-04-22T01:15:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.610+0000 7f74270aa640 1 -- start start 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f74270aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74200cd220 0x7f7420166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f74270aa640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742015c290 con 0x7f74200b4450 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f74270aa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742015c570 con 0x7f7420072280 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f74270aa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f74200b3a50 con 0x7f74200cd220 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7424e46640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f74200b4450 0x7f7420071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40858/0 (socket says 172.21.15.70:40858) 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7424e46640 1 -- 172.21.15.70:0/1283792524 learned_addr learned my addr 172.21.15.70:0/1283792524 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1657564433 0 0) 0x7f742015c290 con 0x7f74200b4450 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 --2- 172.21.15.70:0/1283792524 >> v2:172.21.15.98:3300/0 conn(0x7f74200cd220 0x7f7420166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7408003530 con 0x7f74200b4450 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2788582441 0 0) 0x7f7408003530 con 0x7f74200b4450 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 >> v2:172.21.15.98:3300/0 conn(0x7f74200cd220 msgr2=0x7f7420166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 --2- 172.21.15.70:0/1283792524 >> v2:172.21.15.98:3300/0 conn(0x7f74200cd220 0x7f7420166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 >> v1:172.21.15.98:6789/0 conn(0x7f7420072280 legacy=0x7f74200ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742015c850 con 0x7f74200b4450 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7410001680 con 0x7f74200b4450 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 >> v1:172.21.15.70:6789/0 conn(0x7f74200b4450 legacy=0x7f7420071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74080039c0 0x7f7408003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7408004810 0x7f7408004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f740800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74200b3a50 con 0x7f74080051d0 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f740800c100 con 0x7f74080039c0 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f740800c300 con 0x7f7408004810 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f740800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7424e46640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74080039c0 0x7f7408003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f741ffff640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7408004810 0x7f7408004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 -- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7408004810 msgr2=0x7f7408004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7408004810 0x7f7408004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 -- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74080039c0 msgr2=0x7f7408003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74080039c0 0x7f7408003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 -- 172.21.15.70:0/1283792524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7408003530 con 0x7f74080051d0 2024-04-22T01:15:14.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7424e46640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74080039c0 0x7f7408003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.614+0000 7f7425647640 1 -- 172.21.15.70:0/1283792524 reap_dead start 2024-04-22T01:15:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7425647640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f740800b9d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7418007330 tx=0x7f741801ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772e4af4a3becc51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418009be0 con 0x7f74080051d0 2024-04-22T01:15:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74180a5d60 con 0x7f74080051d0 2024-04-22T01:15:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741f7fe640 1 -- 172.21.15.70:0/1283792524 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418003be0 con 0x7f74080051d0 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/1283792524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 msgr2=0x7f740800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f740800b9d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7418007330 tx=0x7f741801ce60 comp rx=0 tx=0).stop 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/1283792524 shutdown_connections 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 --2- 172.21.15.70:0/1283792524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f740800b9d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/1283792524 >> 172.21.15.70:0/1283792524 conn(0x7f74200b1d40 msgr2=0x7f74200cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/1283792524 shutdown_connections 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/1283792524 wait complete. 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 Processor -- start 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- start start 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f742013e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74201f49c0 0x7f742015de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f742015e380 0x7f74201f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74200b4830 con 0x7f74080051d0 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f742015d050 con 0x7f74201f49c0 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f74200b49b0 con 0x7f742015e380 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f742013e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f742013e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54912/0 (socket says 172.21.15.70:54912) 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 -- 172.21.15.70:0/3090997072 learned_addr learned my addr 172.21.15.70:0/3090997072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741ffff640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74201f49c0 0x7f742015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 -- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f742015e380 msgr2=0x7f74201f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f742015e380 0x7f74201f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 -- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74201f49c0 msgr2=0x7f742015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74201f49c0 0x7f742015de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 -- 172.21.15.70:0/3090997072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7420152210 con 0x7f74080051d0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f7424e46640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f742013e560 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f741000d810 tx=0x7f741000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7410019070 con 0x7f74080051d0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74100074e0 con 0x7f74080051d0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74201524b0 con 0x7f74080051d0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74100049f0 con 0x7f74080051d0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.618+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7420152a70 con 0x7f74080051d0 2024-04-22T01:15:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.622+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7420068210 con 0x7f74080051d0 2024-04-22T01:15:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.622+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f741001c070 con 0x7f74080051d0 2024-04-22T01:15:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.622+0000 7f741d7fa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f740c0678f0 0x7f740c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.622+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7410085e20 con 0x7f74080051d0 2024-04-22T01:15:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.622+0000 7f741ffff640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f740c0678f0 0x7f740c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.622+0000 7f741ffff640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f740c0678f0 0x7f740c069db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f74201f4de0 tx=0x7f7414009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.626+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7410023020 con 0x7f74080051d0 2024-04-22T01:15:14.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.774+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74200d01e0 con 0x7f74080051d0 2024-04-22T01:15:14.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f741d7fa640 1 -- 172.21.15.70:0/3090997072 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f7410056670 con 0x7f74080051d0 2024-04-22T01:15:14.776 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f740c0678f0 msgr2=0x7f740c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f740c0678f0 0x7f740c069db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f74201f4de0 tx=0x7f7414009040 comp rx=0 tx=0).stop 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 msgr2=0x7f742013e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f742013e560 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f741000d810 tx=0x7f741000dcd0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 shutdown_connections 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f740c0678f0 0x7f740c069db0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f742015e380 0x7f74201f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74201f49c0 0x7f742015de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 --2- 172.21.15.70:0/3090997072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74080051d0 0x7f742013e560 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 >> 172.21.15.70:0/3090997072 conn(0x7f74200b1d40 msgr2=0x7f74200cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 shutdown_connections 2024-04-22T01:15:14.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.778+0000 7f74270aa640 1 -- 172.21.15.70:0/3090997072 wait complete. 2024-04-22T01:15:14.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 Processor -- start 2024-04-22T01:15:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 -- start start 2024-04-22T01:15:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbcec0b4450 0x7fbcec071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcec0720b0 0x7fbcec0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcec15c290 con 0x7fbcec0cd050 2024-04-22T01:15:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbcec0b3a50 con 0x7fbcec0b4450 2024-04-22T01:15:14.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf3ea6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbcec15c570 con 0x7fbcec0720b0 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf1c42640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbcec0b4450 0x7fbcec071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf2443640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbcec0cd050 0x7fbcec15f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40862/0 (socket says 172.21.15.70:40862) 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.906+0000 7fbcf2443640 1 -- 172.21.15.70:0/1409353091 learned_addr learned my addr 172.21.15.70:0/1409353091 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1441640 1 --2- 172.21.15.70:0/1409353091 >> v2:172.21.15.98:3300/0 conn(0x7fbcec0720b0 0x7fbcec0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1c42640 1 -- 172.21.15.70:0/1409353091 >> v2:172.21.15.98:3300/0 conn(0x7fbcec0720b0 msgr2=0x7fbcec0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1c42640 1 --2- 172.21.15.70:0/1409353091 >> v2:172.21.15.98:3300/0 conn(0x7fbcec0720b0 0x7fbcec0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1c42640 1 -- 172.21.15.70:0/1409353091 >> v1:172.21.15.70:6789/0 conn(0x7fbcec0cd050 legacy=0x7fbcec15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1c42640 1 -- 172.21.15.70:0/1409353091 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcec15c850 con 0x7fbcec0b4450 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf0c40640 1 -- 172.21.15.70:0/1409353091 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387460896 0 0) 0x7fbcec15c290 con 0x7fbcec0cd050 2024-04-22T01:15:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1441640 1 --2- 172.21.15.70:0/1409353091 >> v2:172.21.15.98:3300/0 conn(0x7fbcec0720b0 0x7fbcec0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf1c42640 1 --2- 172.21.15.70:0/1409353091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec071b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0cf9a0 tx=0x7fbce801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c08afc52c97281 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf0c40640 1 -- 172.21.15.70:0/1409353091 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce80b0070 con 0x7fbcec0b4450 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf0c40640 1 -- 172.21.15.70:0/1409353091 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce80ab600 con 0x7fbcec0b4450 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf0c40640 1 -- 172.21.15.70:0/1409353091 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce80ab9d0 con 0x7fbcec0b4450 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1409353091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 msgr2=0x7fbcec071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1409353091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec071b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0cf9a0 tx=0x7fbce801c480 comp rx=0 tx=0).stop 2024-04-22T01:15:14.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1409353091 shutdown_connections 2024-04-22T01:15:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1409353091 >> v2:172.21.15.98:3300/0 conn(0x7fbcec0720b0 0x7fbcec0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1409353091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec071b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1409353091 >> 172.21.15.70:0/1409353091 conn(0x7fbcec0b1d40 msgr2=0x7fbcec0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1409353091 shutdown_connections 2024-04-22T01:15:14.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.910+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1409353091 wait complete. 2024-04-22T01:15:14.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 Processor -- start 2024-04-22T01:15:14.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- start start 2024-04-22T01:15:14.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcec0720b0 0x7fbcec14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec0cd050 0x7fbcec13d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec0b49b0 con 0x7fbcec0b4450 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec0b4830 con 0x7fbcec0cd050 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbcec0cd800 con 0x7fbcec0720b0 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54934/0 (socket says 172.21.15.70:54934) 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 -- 172.21.15.70:0/1885467143 learned_addr learned my addr 172.21.15.70:0/1885467143 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 -- 172.21.15.70:0/1885467143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcec0720b0 msgr2=0x7fbcec14c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcec0720b0 0x7fbcec14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 -- 172.21.15.70:0/1885467143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec0cd050 msgr2=0x7fbcec13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec0cd050 0x7fbcec13d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 -- 172.21.15.70:0/1885467143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcec14c600 con 0x7fbcec0b4450 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1441640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec159000 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00c7f0 tx=0x7fbcdc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc015070 con 0x7fbcec0b4450 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc007820 con 0x7fbcec0b4450 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcec159790 con 0x7fbcec0b4450 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc007c10 con 0x7fbcec0b4450 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcec159d50 con 0x7fbcec0b4450 2024-04-22T01:15:14.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcec068210 con 0x7fbcec0b4450 2024-04-22T01:15:14.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbcdc018070 con 0x7fbcec0b4450 2024-04-22T01:15:14.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbce27fc640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbcd00678c0 0x7fbcd0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:14.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbcf1c42640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbcd00678c0 0x7fbcd0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:14.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.914+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbcdc086070 con 0x7fbcec0b4450 2024-04-22T01:15:14.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.918+0000 7fbcf1c42640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbcd00678c0 0x7fbcd0069d80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbce80ad000 tx=0x7fbce8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:14.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:14.918+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbcdc056900 con 0x7fbcec0b4450 2024-04-22T01:15:15.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.074+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbcec0d01e0 con 0x7fbcec0b4450 2024-04-22T01:15:15.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.074+0000 7fbce27fc640 1 -- 172.21.15.70:0/1885467143 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fbcdc056200 con 0x7fbcec0b4450 2024-04-22T01:15:15.076 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:15.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbcd00678c0 msgr2=0x7fbcd0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbcd00678c0 0x7fbcd0069d80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbce80ad000 tx=0x7fbce8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:15.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 msgr2=0x7fbcec159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec159000 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00c7f0 tx=0x7fbcdc00ccb0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 shutdown_connections 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbcd00678c0 0x7fbcd0069d80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcec0cd050 0x7fbcec13d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcec0b4450 0x7fbcec159000 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 --2- 172.21.15.70:0/1885467143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcec0720b0 0x7fbcec14c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 >> 172.21.15.70:0/1885467143 conn(0x7fbcec0b1d40 msgr2=0x7fbcec155c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 shutdown_connections 2024-04-22T01:15:15.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.078+0000 7fbcf3ea6640 1 -- 172.21.15.70:0/1885467143 wait complete. 2024-04-22T01:15:15.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:15.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 Processor -- start 2024-04-22T01:15:15.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 -- start start 2024-04-22T01:15:15.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f078c0ce920 0x7f078c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c0cedd0 0x7f078c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078c0b3800 con 0x7f078c15e6e0 2024-04-22T01:15:15.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f078c0b3ae0 con 0x7f078c0ce920 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0790850640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f078c0cde40 con 0x7f078c0cedd0 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f078c0ce920 0x7f078c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f078c0ce920 0x7f078c154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54938/0 (socket says 172.21.15.70:54938) 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 -- 172.21.15.70:0/3048348827 learned_addr learned my addr 172.21.15.70:0/3048348827 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f078ad9d640 1 --2- 172.21.15.70:0/3048348827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c0cedd0 0x7f078c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 -- 172.21.15.70:0/3048348827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c0cedd0 msgr2=0x7f078c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 --2- 172.21.15.70:0/3048348827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c0cedd0 0x7f078c1551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 -- 172.21.15.70:0/3048348827 >> v1:172.21.15.98:6789/0 conn(0x7f078c15e6e0 legacy=0x7f078c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.206+0000 7f0789d9b640 1 -- 172.21.15.70:0/3048348827 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c0ce120 con 0x7f078c0ce920 2024-04-22T01:15:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0789d9b640 1 --2- 172.21.15.70:0/3048348827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c154c60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f078000bb10 tx=0x7f078001ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a54131002100577 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f078959a640 1 -- 172.21.15.70:0/3048348827 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07800b4070 con 0x7f078c0ce920 2024-04-22T01:15:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f078959a640 1 -- 172.21.15.70:0/3048348827 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07800a7c70 con 0x7f078c0ce920 2024-04-22T01:15:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f078959a640 1 -- 172.21.15.70:0/3048348827 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07800af660 con 0x7f078c0ce920 2024-04-22T01:15:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 -- 172.21.15.70:0/3048348827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 msgr2=0x7f078c154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 --2- 172.21.15.70:0/3048348827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c154c60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f078000bb10 tx=0x7f078001ebc0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 -- 172.21.15.70:0/3048348827 shutdown_connections 2024-04-22T01:15:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 --2- 172.21.15.70:0/3048348827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c0cedd0 0x7f078c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 --2- 172.21.15.70:0/3048348827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c154c60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 -- 172.21.15.70:0/3048348827 >> 172.21.15.70:0/3048348827 conn(0x7f078c0b1d40 msgr2=0x7f078c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 -- 172.21.15.70:0/3048348827 shutdown_connections 2024-04-22T01:15:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 -- 172.21.15.70:0/3048348827 wait complete. 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.210+0000 7f0790850640 1 Processor -- start 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- start start 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c1549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c0cedd0 0x7f078c14f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c15e6e0 0x7f078c156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0b2e40 con 0x7f078c0ce920 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0b2cc0 con 0x7f078c0cedd0 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f078c0b2fc0 con 0x7f078c15e6e0 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c1549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c1549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54944/0 (socket says 172.21.15.70:54944) 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 -- 172.21.15.70:0/2741323146 learned_addr learned my addr 172.21.15.70:0/2741323146 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 -- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c15e6e0 msgr2=0x7f078c156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078ad9d640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c15e6e0 0x7f078c156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0789d9b640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c0cedd0 0x7f078c14f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c15e6e0 0x7f078c156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 -- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c0cedd0 msgr2=0x7f078c14f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c0cedd0 0x7f078c14f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 -- 172.21.15.70:0/2741323146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c155080 con 0x7f078c0ce920 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f078a59c640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c1549a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f077c00b440 tx=0x7f077c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c015020 con 0x7f078c0ce920 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077c011890 con 0x7f078c0ce920 2024-04-22T01:15:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078c155360 con 0x7f078c0ce920 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c011c80 con 0x7f078c0ce920 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078c155920 con 0x7f078c0ce920 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078c068210 con 0x7f078c0ce920 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f077c018020 con 0x7f078c0ce920 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f077affd640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f076c0678c0 0x7f076c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f077c0861c0 con 0x7f078c0ce920 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0789d9b640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f076c0678c0 0x7f076c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.214+0000 7f0789d9b640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f076c0678c0 0x7f076c069d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07800b1000 tx=0x7f07800a9070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.218+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f077c01f020 con 0x7f078c0ce920 2024-04-22T01:15:15.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.366+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f078c0b4db0 con 0x7f078c0ce920 2024-04-22T01:15:15.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.366+0000 7f077affd640 1 -- 172.21.15.70:0/2741323146 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f077c00d040 con 0x7f078c0ce920 2024-04-22T01:15:15.366 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:15.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.366+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f076c0678c0 msgr2=0x7f076c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.366+0000 7f0790850640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f076c0678c0 0x7f076c069d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07800b1000 tx=0x7f07800a9070 comp rx=0 tx=0).stop 2024-04-22T01:15:15.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.366+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 msgr2=0x7f078c1549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.366+0000 7f0790850640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c1549a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f077c00b440 tx=0x7f077c00b900 comp rx=0 tx=0).stop 2024-04-22T01:15:15.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 shutdown_connections 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f076c0678c0 0x7f076c069d80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f078c15e6e0 0x7f078c156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f078c0cedd0 0x7f078c14f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 --2- 172.21.15.70:0/2741323146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078c0ce920 0x7f078c1549a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 >> 172.21.15.70:0/2741323146 conn(0x7f078c0b1d40 msgr2=0x7f078c0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 shutdown_connections 2024-04-22T01:15:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.370+0000 7f0790850640 1 -- 172.21.15.70:0/2741323146 wait complete. 2024-04-22T01:15:15.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.506+0000 7f5316f3a640 1 Processor -- start 2024-04-22T01:15:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.506+0000 7f5316f3a640 1 -- start start 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f531015e6e0 0x7f531015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53100ce920 0x7f5310154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f53100ceda0 0x7f53101551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f531015f000 con 0x7f53100ceda0 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f53100b3800 con 0x7f53100ce920 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f53100b3ae0 con 0x7f531015e6e0 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f53100ceda0 0x7f53101551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f53100ceda0 0x7f53101551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54950/0 (socket says 172.21.15.70:54950) 2024-04-22T01:15:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 -- 172.21.15.70:0/1082167538 learned_addr learned my addr 172.21.15.70:0/1082167538 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5314cd6640 1 --2- 172.21.15.70:0/1082167538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f531015e6e0 0x7f531015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f530ffff640 1 --2- 172.21.15.70:0/1082167538 >> v2:172.21.15.98:3300/0 conn(0x7f53100ce920 0x7f5310154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 -- 172.21.15.70:0/1082167538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f531015e6e0 msgr2=0x7f531015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 --2- 172.21.15.70:0/1082167538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f531015e6e0 0x7f531015eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 -- 172.21.15.70:0/1082167538 >> v2:172.21.15.98:3300/0 conn(0x7f53100ce920 msgr2=0x7f5310154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 --2- 172.21.15.70:0/1082167538 >> v2:172.21.15.98:3300/0 conn(0x7f53100ce920 0x7f5310154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 -- 172.21.15.70:0/1082167538 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53100cde40 con 0x7f53100ceda0 2024-04-22T01:15:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f53154d7640 1 --2- 172.21.15.70:0/1082167538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53100ceda0 0x7f53101551a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5310072e50 tx=0x7f530801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e3dfcd23654479 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f530f7fe640 1 -- 172.21.15.70:0/1082167538 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080a5840 con 0x7f53100ceda0 2024-04-22T01:15:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f530f7fe640 1 -- 172.21.15.70:0/1082167538 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53080a59e0 con 0x7f53100ceda0 2024-04-22T01:15:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f530f7fe640 1 -- 172.21.15.70:0/1082167538 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080a5db0 con 0x7f53100ceda0 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 -- 172.21.15.70:0/1082167538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53100ceda0 msgr2=0x7f53101551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 --2- 172.21.15.70:0/1082167538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53100ceda0 0x7f53101551a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5310072e50 tx=0x7f530801c9c0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 -- 172.21.15.70:0/1082167538 shutdown_connections 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 --2- 172.21.15.70:0/1082167538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53100ceda0 0x7f53101551a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.510+0000 7f5316f3a640 1 --2- 172.21.15.70:0/1082167538 >> v2:172.21.15.98:3300/0 conn(0x7f53100ce920 0x7f5310154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 --2- 172.21.15.70:0/1082167538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f531015e6e0 0x7f531015eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- 172.21.15.70:0/1082167538 >> 172.21.15.70:0/1082167538 conn(0x7f53100b1d40 msgr2=0x7f53100cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:15.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- 172.21.15.70:0/1082167538 shutdown_connections 2024-04-22T01:15:15.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- 172.21.15.70:0/1082167538 wait complete. 2024-04-22T01:15:15.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 Processor -- start 2024-04-22T01:15:15.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- start start 2024-04-22T01:15:15.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53100ce920 0x7f5310146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53100ceda0 0x7f531014e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 0x7f531014c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53100b2c30 con 0x7f531015e6e0 2024-04-22T01:15:15.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53100b2ab0 con 0x7f53100ce920 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f53100b2db0 con 0x7f53100ceda0 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 0x7f531014c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 0x7f531014c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54954/0 (socket says 172.21.15.70:54954) 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 -- 172.21.15.70:0/2045864513 learned_addr learned my addr 172.21.15.70:0/2045864513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f530ffff640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53100ceda0 0x7f531014e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5314cd6640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53100ce920 0x7f5310146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 -- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53100ceda0 msgr2=0x7f531014e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53100ceda0 0x7f531014e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 -- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53100ce920 msgr2=0x7f5310146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53100ce920 0x7f5310146b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 -- 172.21.15.70:0/2045864513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53100b0af0 con 0x7f531015e6e0 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f53154d7640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 0x7f531014c1e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f530801b070 tx=0x7f5308019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080b8670 con 0x7f531015e6e0 2024-04-22T01:15:15.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53080be070 con 0x7f531015e6e0 2024-04-22T01:15:15.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080b8a50 con 0x7f531015e6e0 2024-04-22T01:15:15.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53100b0e30 con 0x7f531015e6e0 2024-04-22T01:15:15.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.514+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53101470c0 con 0x7f531015e6e0 2024-04-22T01:15:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.518+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5310068210 con 0x7f531015e6e0 2024-04-22T01:15:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.518+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f53080b8bf0 con 0x7f531015e6e0 2024-04-22T01:15:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.518+0000 7f530d7fa640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f52f40678c0 0x7f52f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.518+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f53081212b0 con 0x7f531015e6e0 2024-04-22T01:15:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.518+0000 7f5314cd6640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f52f40678c0 0x7f52f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.518+0000 7f5314cd6640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f52f40678c0 0x7f52f4069d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f530400b930 tx=0x7f5304008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.522+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f53100b0e30 con 0x7f531015e6e0 2024-04-22T01:15:15.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f53101479e0 con 0x7f531015e6e0 2024-04-22T01:15:15.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f530d7fa640 1 -- 172.21.15.70:0/2045864513 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f53080165e0 con 0x7f531015e6e0 2024-04-22T01:15:15.681 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:15.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f52f40678c0 msgr2=0x7f52f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f52f40678c0 0x7f52f4069d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f530400b930 tx=0x7f5304008040 comp rx=0 tx=0).stop 2024-04-22T01:15:15.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 msgr2=0x7f531014c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 0x7f531014c1e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f530801b070 tx=0x7f5308019760 comp rx=0 tx=0).stop 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 shutdown_connections 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f52f40678c0 0x7f52f4069d80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f531015e6e0 0x7f531014c1e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53100ceda0 0x7f531014e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 --2- 172.21.15.70:0/2045864513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53100ce920 0x7f5310146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 >> 172.21.15.70:0/2045864513 conn(0x7f53100b1d40 msgr2=0x7f5310153e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 shutdown_connections 2024-04-22T01:15:15.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.682+0000 7f5316f3a640 1 -- 172.21.15.70:0/2045864513 wait complete. 2024-04-22T01:15:15.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:15.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.814+0000 7f5a6df2e640 1 Processor -- start 2024-04-22T01:15:15.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.814+0000 7f5a6df2e640 1 -- start start 2024-04-22T01:15:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5a6815dfb0 0x7f5a680b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a680b6550 con 0x7f5a680cea80 2024-04-22T01:15:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a6815e390 con 0x7f5a680d3120 2024-04-22T01:15:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5a68161e00 con 0x7f5a6815dfb0 2024-04-22T01:15:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a677fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5a6815dfb0 0x7f5a680b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a66ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5a680cea80 0x7f5a680b5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:40878/0 (socket says 172.21.15.70:40878) 2024-04-22T01:15:15.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a66ffd640 1 -- 172.21.15.70:0/3846552905 learned_addr learned my addr 172.21.15.70:0/3846552905 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:15.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a677fe640 1 -- 172.21.15.70:0/3846552905 >> v1:172.21.15.98:6789/0 conn(0x7f5a680d3120 legacy=0x7f5a680b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:15.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a677fe640 1 -- 172.21.15.70:0/3846552905 >> v1:172.21.15.70:6789/0 conn(0x7f5a680cea80 legacy=0x7f5a680b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a667fc640 1 -- 172.21.15.70:0/3846552905 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916253591 0 0) 0x7f5a680b6550 con 0x7f5a680cea80 2024-04-22T01:15:15.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a677fe640 1 -- 172.21.15.70:0/3846552905 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a681620e0 con 0x7f5a6815dfb0 2024-04-22T01:15:15.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a677fe640 1 --2- 172.21.15.70:0/3846552905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6815dfb0 0x7f5a680b95d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097c0 tx=0x7f5a5801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a786a901556b1651 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a667fc640 1 -- 172.21.15.70:0/3846552905 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580b0070 con 0x7f5a6815dfb0 2024-04-22T01:15:15.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a667fc640 1 -- 172.21.15.70:0/3846552905 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a580ab600 con 0x7f5a6815dfb0 2024-04-22T01:15:15.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a667fc640 1 -- 172.21.15.70:0/3846552905 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580ab9d0 con 0x7f5a6815dfb0 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3846552905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6815dfb0 msgr2=0x7f5a680b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3846552905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6815dfb0 0x7f5a680b95d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097c0 tx=0x7f5a5801c460 comp rx=0 tx=0).stop 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3846552905 shutdown_connections 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3846552905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a6815dfb0 0x7f5a680b95d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3846552905 >> 172.21.15.70:0/3846552905 conn(0x7f5a6806c9b0 msgr2=0x7f5a6806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3846552905 shutdown_connections 2024-04-22T01:15:15.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.818+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3846552905 wait complete. 2024-04-22T01:15:15.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 Processor -- start 2024-04-22T01:15:15.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 -- start start 2024-04-22T01:15:15.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 0x7f5a68159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680d3120 0x7f5a68155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a6815dfb0 0x7f5a68158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a680cf2b0 con 0x7f5a680cea80 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a68161a90 con 0x7f5a680d3120 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5a680cf430 con 0x7f5a6815dfb0 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 0x7f5a68159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 0x7f5a68159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54984/0 (socket says 172.21.15.70:54984) 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 -- 172.21.15.70:0/3460788863 learned_addr learned my addr 172.21.15.70:0/3460788863 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a66ffd640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680d3120 0x7f5a68155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a67fff640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a6815dfb0 0x7f5a68158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 -- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a6815dfb0 msgr2=0x7f5a68158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a6815dfb0 0x7f5a68158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 -- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680d3120 msgr2=0x7f5a68155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680d3120 0x7f5a68155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:15.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 -- 172.21.15.70:0/3460788863 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6815d0b0 con 0x7f5a680cea80 2024-04-22T01:15:15.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a677fe640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 0x7f5a68159ea0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a580ae040 tx=0x7f5a5801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580b0050 con 0x7f5a680cea80 2024-04-22T01:15:15.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a580bc070 con 0x7f5a680cea80 2024-04-22T01:15:15.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580b5a70 con 0x7f5a680cea80 2024-04-22T01:15:15.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6815cdd0 con 0x7f5a680cea80 2024-04-22T01:15:15.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.822+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a680c9d80 con 0x7f5a680cea80 2024-04-22T01:15:15.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.826+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a68068210 con 0x7f5a680cea80 2024-04-22T01:15:15.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.826+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a580ab4c0 con 0x7f5a680cea80 2024-04-22T01:15:15.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.826+0000 7f5a4ffff640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5a540678c0 0x7f5a54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:15.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.826+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5a58120800 con 0x7f5a680cea80 2024-04-22T01:15:15.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.826+0000 7f5a66ffd640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5a540678c0 0x7f5a54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:15.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.826+0000 7f5a66ffd640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5a540678c0 0x7f5a54069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a681607d0 tx=0x7f5a5c007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:15.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:15.830+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5a58016050 con 0x7f5a680cea80 2024-04-22T01:15:16.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.022+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a680d18f0 con 0x7f5a680cea80 2024-04-22T01:15:16.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a4ffff640 1 -- 172.21.15.70:0/3460788863 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f5a580f0ff0 con 0x7f5a680cea80 2024-04-22T01:15:16.024 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:16.024 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:16.025 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:16.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5a540678c0 msgr2=0x7f5a54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5a540678c0 0x7f5a54069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a681607d0 tx=0x7f5a5c007c10 comp rx=0 tx=0).stop 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 msgr2=0x7f5a68159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 0x7f5a68159ea0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a580ae040 tx=0x7f5a5801aca0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 shutdown_connections 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5a540678c0 0x7f5a54069d80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a6815dfb0 0x7f5a68158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680d3120 0x7f5a68155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 --2- 172.21.15.70:0/3460788863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a680cea80 0x7f5a68159ea0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 >> 172.21.15.70:0/3460788863 conn(0x7f5a6806c9b0 msgr2=0x7f5a6806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 shutdown_connections 2024-04-22T01:15:16.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.026+0000 7f5a6df2e640 1 -- 172.21.15.70:0/3460788863 wait complete. 2024-04-22T01:15:16.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:16.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.162+0000 7f999c81a640 1 Processor -- start 2024-04-22T01:15:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.162+0000 7f999c81a640 1 -- start start 2024-04-22T01:15:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999c81a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f999815bb40 0x7f9998072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999c81a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f999815bf20 con 0x7f99980b3cd0 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999c81a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99980aec30 con 0x7f999806ed40 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999c81a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f999806e8b0 con 0x7f999815bb40 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f9996d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f99980b3cd0 0x7f9998161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42564/0 (socket says 172.21.15.70:42564) 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f9996d9d640 1 -- 172.21.15.70:0/200509039 learned_addr learned my addr 172.21.15.70:0/200509039 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 23089355 0 0) 0x7f999815bf20 con 0x7f99980b3cd0 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 --2- 172.21.15.70:0/200509039 >> v2:172.21.15.98:3300/0 conn(0x7f999815bb40 0x7f9998072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9984003530 con 0x7f99980b3cd0 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2501642396 0 0) 0x7f9984003530 con 0x7f99980b3cd0 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 >> v2:172.21.15.98:3300/0 conn(0x7f999815bb40 msgr2=0x7f9998072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 --2- 172.21.15.70:0/200509039 >> v2:172.21.15.98:3300/0 conn(0x7f999815bb40 0x7f9998072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 >> v1:172.21.15.98:6789/0 conn(0x7f999806ed40 legacy=0x7f99980b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99980aef10 con 0x7f99980b3cd0 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f9990001a00 con 0x7f99980b3cd0 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 >> v1:172.21.15.70:6789/0 conn(0x7f99980b3cd0 legacy=0x7f9998161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99840039c0 0x7f9984003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9984004810 0x7f9984004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840051d0 0x7f998400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f999806e8b0 con 0x7f99840039c0 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f998400c070 con 0x7f9984004810 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f998400c270 con 0x7f99840051d0 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99840039c0 0x7f9984003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f9995d9b640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9984004810 0x7f9984004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 -- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840051d0 msgr2=0x7f998400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f9996d9d640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840051d0 0x7f998400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840051d0 0x7f998400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 -- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9984004810 msgr2=0x7f9984004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9984004810 0x7f9984004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f9996d9d640 1 -- 172.21.15.70:0/200509039 reap_dead start 2024-04-22T01:15:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f999659c640 1 -- 172.21.15.70:0/200509039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9984003530 con 0x7f99840039c0 2024-04-22T01:15:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.166+0000 7f9995d9b640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9984004810 0x7f9984004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:16.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999659c640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99840039c0 0x7f9984003de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9988009ae0 tx=0x7f998801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5683b90a6c6e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880b2070 con 0x7f99840039c0 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99880a5510 con 0x7f99840039c0 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999559a640 1 -- 172.21.15.70:0/200509039 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880096b0 con 0x7f99840039c0 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- 172.21.15.70:0/200509039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99840039c0 msgr2=0x7f9984003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99840039c0 0x7f9984003de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9988009ae0 tx=0x7f998801ca60 comp rx=0 tx=0).stop 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- 172.21.15.70:0/200509039 shutdown_connections 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 --2- 172.21.15.70:0/200509039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99840039c0 0x7f9984003de0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- 172.21.15.70:0/200509039 >> 172.21.15.70:0/200509039 conn(0x7f999806b1f0 msgr2=0x7f999806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- 172.21.15.70:0/200509039 shutdown_connections 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- 172.21.15.70:0/200509039 wait complete. 2024-04-22T01:15:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 Processor -- start 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- start start 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840039c0 0x7f9998157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999815f7e0 0x7f99980cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 0x7f99981fa750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99980ae850 con 0x7f99980ce040 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9998072940 con 0x7f999815f7e0 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.170+0000 7f999c81a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99980ae9d0 con 0x7f99840039c0 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 0x7f99981fa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 0x7f99981fa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50274/0 (socket says 172.21.15.70:50274) 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 -- 172.21.15.70:0/940891988 learned_addr learned my addr 172.21.15.70:0/940891988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:16.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9995d9b640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999815f7e0 0x7f99980cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f999659c640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840039c0 0x7f9998157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 -- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840039c0 msgr2=0x7f9998157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840039c0 0x7f9998157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 -- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999815f7e0 msgr2=0x7f99980cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999815f7e0 0x7f99980cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 -- 172.21.15.70:0/940891988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999815cb90 con 0x7f99980ce040 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f9996d9d640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 0x7f99981fa750 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9990000f90 tx=0x7f999000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990004d70 con 0x7f99980ce040 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f999000bb20 con 0x7f99980ce040 2024-04-22T01:15:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999815d660 con 0x7f99980ce040 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990007110 con 0x7f99980ce040 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999815dc20 con 0x7f99980ce040 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9998068210 con 0x7f99980ce040 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f999001b020 con 0x7f99980ce040 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f998effd640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99780678f0 0x7f9978069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9990086120 con 0x7f99980ce040 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f999659c640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99780678f0 0x7f9978069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.174+0000 7f999659c640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99780678f0 0x7f9978069db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f99880af000 tx=0x7f9988008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.178+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9990056920 con 0x7f99980ce040 2024-04-22T01:15:16.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.338+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99980b2cb0 con 0x7f99980ce040 2024-04-22T01:15:16.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f998effd640 1 -- 172.21.15.70:0/940891988 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7f9990056220 con 0x7f99980ce040 2024-04-22T01:15:16.341 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:16.341 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:16.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99780678f0 msgr2=0x7f9978069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99780678f0 0x7f9978069db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f99880af000 tx=0x7f9988008040 comp rx=0 tx=0).stop 2024-04-22T01:15:16.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 msgr2=0x7f99981fa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 0x7f99981fa750 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9990000f90 tx=0x7f999000b4a0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 shutdown_connections 2024-04-22T01:15:16.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99780678f0 0x7f9978069db0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99980ce040 0x7f99981fa750 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999815f7e0 0x7f99980cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 --2- 172.21.15.70:0/940891988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99840039c0 0x7f9998157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 >> 172.21.15.70:0/940891988 conn(0x7f999806b1f0 msgr2=0x7f99980b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 shutdown_connections 2024-04-22T01:15:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.342+0000 7f999c81a640 1 -- 172.21.15.70:0/940891988 wait complete. 2024-04-22T01:15:16.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:16.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 Processor -- start 2024-04-22T01:15:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 -- start start 2024-04-22T01:15:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc15e6e0 0x7fd7fc15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7fc0ce920 0x7fd7fc154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc15f000 con 0x7fd7fc0ceda0 2024-04-22T01:15:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd7fc0b3800 con 0x7fd7fc0ce920 2024-04-22T01:15:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd801466640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd7fc0b3ae0 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7fc0ce920 0x7fd7fc154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7fc0ce920 0x7fd7fc154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50278/0 (socket says 172.21.15.70:50278) 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 -- 172.21.15.70:0/4278827783 learned_addr learned my addr 172.21.15.70:0/4278827783 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7faffd640 1 --2- 172.21.15.70:0/4278827783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc15e6e0 0x7fd7fc15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 -- 172.21.15.70:0/4278827783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc15e6e0 msgr2=0x7fd7fc15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 --2- 172.21.15.70:0/4278827783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc15e6e0 0x7fd7fc15eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 -- 172.21.15.70:0/4278827783 >> v1:172.21.15.98:6789/0 conn(0x7fd7fc0ceda0 legacy=0x7fd7fc06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.486+0000 7fd7fa7fc640 1 -- 172.21.15.70:0/4278827783 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc0cde40 con 0x7fd7fc0ce920 2024-04-22T01:15:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fa7fc640 1 --2- 172.21.15.70:0/4278827783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc154c60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009870 tx=0x7fd7e401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de757df46d88e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7f9ffb640 1 -- 172.21.15.70:0/4278827783 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40aa610 con 0x7fd7fc0ce920 2024-04-22T01:15:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7f9ffb640 1 -- 172.21.15.70:0/4278827783 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40af070 con 0x7fd7fc0ce920 2024-04-22T01:15:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7f9ffb640 1 -- 172.21.15.70:0/4278827783 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40aa9d0 con 0x7fd7fc0ce920 2024-04-22T01:15:16.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- 172.21.15.70:0/4278827783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc0ce920 msgr2=0x7fd7fc154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 --2- 172.21.15.70:0/4278827783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc154c60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009870 tx=0x7fd7e401c4f0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- 172.21.15.70:0/4278827783 shutdown_connections 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 --2- 172.21.15.70:0/4278827783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc154c60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 --2- 172.21.15.70:0/4278827783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc15e6e0 0x7fd7fc15eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- 172.21.15.70:0/4278827783 >> 172.21.15.70:0/4278827783 conn(0x7fd7fc0b1d40 msgr2=0x7fd7fc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- 172.21.15.70:0/4278827783 shutdown_connections 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- 172.21.15.70:0/4278827783 wait complete. 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 Processor -- start 2024-04-22T01:15:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- start start 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc0ceda0 0x7fd7fc158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 0x7fd7fc14f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc0b32c0 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1555b0 con 0x7fd7fc0ce920 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd801466640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd7fc0b3440 con 0x7fd7fc0ceda0 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 0x7fd7fc14f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 0x7fd7fc14f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50284/0 (socket says 172.21.15.70:50284) 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/599397428 learned_addr learned my addr 172.21.15.70:0/599397428 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7faffd640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fa7fc640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc0ceda0 0x7fd7fc158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc0ceda0 msgr2=0x7fd7fc158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc0ceda0 0x7fd7fc158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7fc0ce920 msgr2=0x7fd7fc0afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc0afaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/599397428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc159e60 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.490+0000 7fd7fb7fe640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 0x7fd7fc14f060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b970 tx=0x7fd7ec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec019070 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec00cb40 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7fc15a140 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec014440 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7fc14c4a0 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7fc068210 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd7ec01c070 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7f37fe640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7dc0678c0 0x7fd7dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7ec085c00 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7faffd640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7dc0678c0 0x7fd7dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.494+0000 7fd7faffd640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7dc0678c0 0x7fd7dc069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc14ea70 tx=0x7fd7f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.498+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd7ec08c080 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.662+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd7fc0d08e0 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.662+0000 7fd7f37fe640 1 -- 172.21.15.70:0/599397428 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fd7ec056400 con 0x7fd7fc15e6e0 2024-04-22T01:15:16.663 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7dc0678c0 msgr2=0x7fd7dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7dc0678c0 0x7fd7dc069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc14ea70 tx=0x7fd7f4008040 comp rx=0 tx=0).stop 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 msgr2=0x7fd7fc14f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 0x7fd7fc14f060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b970 tx=0x7fd7ec00be30 comp rx=0 tx=0).stop 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 shutdown_connections 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7dc0678c0 0x7fd7dc069d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7fc15e6e0 0x7fd7fc14f060 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7fc0ceda0 0x7fd7fc158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 --2- 172.21.15.70:0/599397428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7fc0ce920 0x7fd7fc0afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 >> 172.21.15.70:0/599397428 conn(0x7fd7fc0b1d40 msgr2=0x7fd7fc06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 shutdown_connections 2024-04-22T01:15:16.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.666+0000 7fd801466640 1 -- 172.21.15.70:0/599397428 wait complete. 2024-04-22T01:15:16.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:16.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 Processor -- start 2024-04-22T01:15:16.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 -- start start 2024-04-22T01:15:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77440ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77440cd050 0x7f77440b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774415c290 con 0x7f77440b4450 2024-04-22T01:15:16.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f774415c570 con 0x7f77440cd050 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f774a8aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f774415c850 con 0x7f7744072280 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7743fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f77440b4450 0x7f7744071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42566/0 (socket says 172.21.15.70:42566) 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7743fff640 1 -- 172.21.15.70:0/3462601240 learned_addr learned my addr 172.21.15.70:0/3462601240 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841604698 0 0) 0x7f774415c290 con 0x7f77440b4450 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f77437fe640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77440ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7730003530 con 0x7f77440b4450 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 --2- 172.21.15.70:0/3462601240 >> v2:172.21.15.98:3300/0 conn(0x7f77440cd050 0x7f77440b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3145081994 0 0) 0x7f7730003530 con 0x7f77440b4450 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 msgr2=0x7f77440ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77440ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 >> v2:172.21.15.98:3300/0 conn(0x7f77440cd050 msgr2=0x7f77440b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 --2- 172.21.15.70:0/3462601240 >> v2:172.21.15.98:3300/0 conn(0x7f77440cd050 0x7f77440b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774415cb30 con 0x7f77440b4450 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f773c001a00 con 0x7f77440b4450 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f77437fe640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77440ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 >> v1:172.21.15.70:6789/0 conn(0x7f77440b4450 legacy=0x7f7744071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77300039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730003f00 0x7f7730004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7730004d50 0x7f77300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f774415c570 con 0x7f7730004d50 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f774415c850 con 0x7f7730003f00 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f773000ba00 con 0x7f7744072280 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7730004d50 0x7f77300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7743fff640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77300039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 -- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 msgr2=0x7f77300039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7744072280 0x7f77300039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 -- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730003f00 msgr2=0x7f7730004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730003f00 0x7f7730004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.806+0000 7f7748e47640 1 -- 172.21.15.70:0/3462601240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7730003530 con 0x7f7730004d50 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7748e47640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7730004d50 0x7f77300051d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7734009ae0 tx=0x7f773401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb74967ab8ee501 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77340b2070 con 0x7f7730004d50 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77340a56a0 con 0x7f7730004d50 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7742ffd640 1 -- 172.21.15.70:0/3462601240 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77340096b0 con 0x7f7730004d50 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- 172.21.15.70:0/3462601240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7730004d50 msgr2=0x7f77300051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 --2- 172.21.15.70:0/3462601240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7730004d50 0x7f77300051d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7734009ae0 tx=0x7f773401cab0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7748e47640 1 -- 172.21.15.70:0/3462601240 reap_dead start 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- 172.21.15.70:0/3462601240 shutdown_connections 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- 172.21.15.70:0/3462601240 >> 172.21.15.70:0/3462601240 conn(0x7f77440b1d40 msgr2=0x7f77440cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- 172.21.15.70:0/3462601240 shutdown_connections 2024-04-22T01:15:16.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- 172.21.15.70:0/3462601240 wait complete. 2024-04-22T01:15:16.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 Processor -- start 2024-04-22T01:15:16.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- start start 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730004d50 0x7f774414b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77440afc50 0x7f77440ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 0x7f77440c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77440b4830 con 0x7f77440cad40 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f77440cd4a0 con 0x7f7730004d50 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f774a8aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f77440b49b0 con 0x7f77440afc50 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7748e47640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 0x7f77440c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7748e47640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 0x7f77440c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50306/0 (socket says 172.21.15.70:50306) 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7748e47640 1 -- 172.21.15.70:0/534813081 learned_addr learned my addr 172.21.15.70:0/534813081 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.810+0000 7f7743fff640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730004d50 0x7f774414b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7748e47640 1 -- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77440afc50 msgr2=0x7f77440ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f77437fe640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77440afc50 0x7f77440ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7748e47640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77440afc50 0x7f77440ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7748e47640 1 -- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730004d50 msgr2=0x7f774414b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7748e47640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730004d50 0x7f774414b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7748e47640 1 -- 172.21.15.70:0/534813081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774406e4a0 con 0x7f77440cad40 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7748e47640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 0x7f77440c8dc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7734009ae0 tx=0x7f7730005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77340b2040 con 0x7f77440cad40 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7734004020 con 0x7f77440cad40 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774406e780 con 0x7f77440cad40 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734004430 con 0x7f77440cad40 2024-04-22T01:15:16.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774406ed40 con 0x7f77440cad40 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7744068210 con 0x7f77440cad40 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f77340bf060 con 0x7f77440cad40 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7740ff9640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77240678c0 0x7f7724069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7743fff640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77240678c0 0x7f7724069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f773411fd80 con 0x7f77440cad40 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.814+0000 7f7743fff640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77240678c0 0x7f7724069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f773c002240 tx=0x7f773c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:16.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.818+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f77340167c0 con 0x7f77440cad40 2024-04-22T01:15:16.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.982+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7744071ef0 con 0x7f77440cad40 2024-04-22T01:15:16.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.982+0000 7f7740ff9640 1 -- 172.21.15.70:0/534813081 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f77340f04f0 con 0x7f77440cad40 2024-04-22T01:15:16.982 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77240678c0 msgr2=0x7f7724069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77240678c0 0x7f7724069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f773c002240 tx=0x7f773c009040 comp rx=0 tx=0).stop 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 msgr2=0x7f77440c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 0x7f77440c8dc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7734009ae0 tx=0x7f7730005840 comp rx=0 tx=0).stop 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 shutdown_connections 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77240678c0 0x7f7724069d80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440cad40 0x7f77440c8dc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77440afc50 0x7f77440ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 --2- 172.21.15.70:0/534813081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7730004d50 0x7f774414b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 >> 172.21.15.70:0/534813081 conn(0x7f77440b1d40 msgr2=0x7f77441559b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 shutdown_connections 2024-04-22T01:15:16.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:16.986+0000 7f774a8aa640 1 -- 172.21.15.70:0/534813081 wait complete. 2024-04-22T01:15:17.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:17.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 Processor -- start 2024-04-22T01:15:17.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 -- start start 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f14a8072280 0x7f14a80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80cd050 0x7f14a80b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14a815c290 con 0x7f14a80b4450 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f14a815c570 con 0x7f14a8072280 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14aec32640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f14a815c850 con 0x7f14a80cd050 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14ac9ce640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f14a80b4450 0x7f14a8071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42570/0 (socket says 172.21.15.70:42570) 2024-04-22T01:15:17.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14a7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f14a8072280 0x7f14a80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14ac9ce640 1 -- 172.21.15.70:0/1548622645 learned_addr learned my addr 172.21.15.70:0/1548622645 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14ad1cf640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80cd050 0x7f14a80b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14a7fff640 1 -- 172.21.15.70:0/1548622645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80cd050 msgr2=0x7f14a80b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14a7fff640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80cd050 0x7f14a80b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14a7fff640 1 -- 172.21.15.70:0/1548622645 >> v1:172.21.15.70:6789/0 conn(0x7f14a80b4450 legacy=0x7f14a8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14a7fff640 1 -- 172.21.15.70:0/1548622645 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a815cb30 con 0x7f14a8072280 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14a77fe640 1 -- 172.21.15.70:0/1548622645 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3084634601 0 0) 0x7f14a815c290 con 0x7f14a80b4450 2024-04-22T01:15:17.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.122+0000 7f14ad1cf640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80cd050 0x7f14a80b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:17.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a7fff640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a80ccb10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f14940097c0 tx=0x7f149401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344e03aa8a6e07bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a77fe640 1 -- 172.21.15.70:0/1548622645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14940b0070 con 0x7f14a8072280 2024-04-22T01:15:17.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a77fe640 1 -- 172.21.15.70:0/1548622645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14940ab600 con 0x7f14a8072280 2024-04-22T01:15:17.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a77fe640 1 -- 172.21.15.70:0/1548622645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14940ab9d0 con 0x7f14a8072280 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/1548622645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 msgr2=0x7f14a80ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a80ccb10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f14940097c0 tx=0x7f149401c410 comp rx=0 tx=0).stop 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/1548622645 shutdown_connections 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80cd050 0x7f14a80b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 --2- 172.21.15.70:0/1548622645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a80ccb10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/1548622645 >> 172.21.15.70:0/1548622645 conn(0x7f14a80b1d40 msgr2=0x7f14a80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:17.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/1548622645 shutdown_connections 2024-04-22T01:15:17.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/1548622645 wait complete. 2024-04-22T01:15:17.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 Processor -- start 2024-04-22T01:15:17.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- start start 2024-04-22T01:15:17.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80b4450 0x7f14a814dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14a80cd050 0x7f14a81f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14a80cd980 con 0x7f14a8072280 2024-04-22T01:15:17.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14a80cd800 con 0x7f14a80cd050 2024-04-22T01:15:17.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f14a80b3790 con 0x7f14a80b4450 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50322/0 (socket says 172.21.15.70:50322) 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 -- 172.21.15.70:0/2362484628 learned_addr learned my addr 172.21.15.70:0/2362484628 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 -- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80b4450 msgr2=0x7f14a814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ad1cf640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14a80cd050 0x7f14a81f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80b4450 0x7f14a814dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 -- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14a80cd050 msgr2=0x7f14a81f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14a80cd050 0x7f14a81f7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 -- 172.21.15.70:0/2362484628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a8155b40 con 0x7f14a8072280 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14ac9ce640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a81535a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f149c00c520 tx=0x7f149c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c018070 con 0x7f14a8072280 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f149c0074e0 con 0x7f14a8072280 2024-04-22T01:15:17.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a8146040 con 0x7f14a8072280 2024-04-22T01:15:17.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c0049f0 con 0x7f14a8072280 2024-04-22T01:15:17.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.126+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a8155e20 con 0x7f14a8072280 2024-04-22T01:15:17.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.130+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14a8068210 con 0x7f14a8072280 2024-04-22T01:15:17.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.130+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f149c01b070 con 0x7f14a8072280 2024-04-22T01:15:17.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.130+0000 7f14a57fa640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f148c0678c0 0x7f148c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.130+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f149c085410 con 0x7f14a8072280 2024-04-22T01:15:17.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.130+0000 7f14a7fff640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f148c0678c0 0x7f148c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.130+0000 7f14a7fff640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f148c0678c0 0x7f148c069d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f14a806fce0 tx=0x7f1494008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.134+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f149c055c10 con 0x7f14a8072280 2024-04-22T01:15:17.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.278+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f14a80d01e0 con 0x7f14a8072280 2024-04-22T01:15:17.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14a57fa640 1 -- 172.21.15.70:0/2362484628 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f149c055510 con 0x7f14a8072280 2024-04-22T01:15:17.280 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:17.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f148c0678c0 msgr2=0x7f148c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f148c0678c0 0x7f148c069d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f14a806fce0 tx=0x7f1494008040 comp rx=0 tx=0).stop 2024-04-22T01:15:17.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 msgr2=0x7f14a81535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a81535a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f149c00c520 tx=0x7f149c00c9e0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 shutdown_connections 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f148c0678c0 0x7f148c069d80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14a80cd050 0x7f14a81f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14a80b4450 0x7f14a814dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 --2- 172.21.15.70:0/2362484628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14a8072280 0x7f14a81535a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 >> 172.21.15.70:0/2362484628 conn(0x7f14a80b1d40 msgr2=0x7f14a815d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 shutdown_connections 2024-04-22T01:15:17.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.282+0000 7f14aec32640 1 -- 172.21.15.70:0/2362484628 wait complete. 2024-04-22T01:15:17.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:17.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 Processor -- start 2024-04-22T01:15:17.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 -- start start 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f8806b1f0 0x7f2f8806b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2f88161970 0x7f2f88161e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f880d1a70 con 0x7f2f8806efb0 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2f880cd570 con 0x7f2f88161970 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8fb5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2f8806fd20 con 0x7f2f8806b1f0 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2f88161970 0x7f2f88161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2f88161970 0x7f2f88161e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50338/0 (socket says 172.21.15.70:50338) 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8d0f7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2f8806efb0 0x7f2f8806f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42582/0 (socket says 172.21.15.70:42582) 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 -- 172.21.15.70:0/3854870227 learned_addr learned my addr 172.21.15.70:0/3854870227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8d8f8640 1 --2- 172.21.15.70:0/3854870227 >> v2:172.21.15.98:3300/0 conn(0x7f2f8806b1f0 0x7f2f8806b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 -- 172.21.15.70:0/3854870227 >> v2:172.21.15.98:3300/0 conn(0x7f2f8806b1f0 msgr2=0x7f2f8806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 --2- 172.21.15.70:0/3854870227 >> v2:172.21.15.98:3300/0 conn(0x7f2f8806b1f0 0x7f2f8806b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 -- 172.21.15.70:0/3854870227 >> v1:172.21.15.70:6789/0 conn(0x7f2f8806efb0 legacy=0x7f2f8806f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8c8f6640 1 -- 172.21.15.70:0/3854870227 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300066165 0 0) 0x7f2f880d1a70 con 0x7f2f8806efb0 2024-04-22T01:15:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8e0f9640 1 -- 172.21.15.70:0/3854870227 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8806bb10 con 0x7f2f88161970 2024-04-22T01:15:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.410+0000 7f2f8d8f8640 1 --2- 172.21.15.70:0/3854870227 >> v2:172.21.15.98:3300/0 conn(0x7f2f8806b1f0 0x7f2f8806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:17.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8e0f9640 1 --2- 172.21.15.70:0/3854870227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88161970 0x7f2f88161e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f8809a420 tx=0x7f2f8001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afed31173acbdf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8c8f6640 1 -- 172.21.15.70:0/3854870227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f800b0070 con 0x7f2f88161970 2024-04-22T01:15:17.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8c8f6640 1 -- 172.21.15.70:0/3854870227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f800ab600 con 0x7f2f88161970 2024-04-22T01:15:17.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8c8f6640 1 -- 172.21.15.70:0/3854870227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f800ab9d0 con 0x7f2f88161970 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3854870227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88161970 msgr2=0x7f2f88161e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3854870227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88161970 0x7f2f88161e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f8809a420 tx=0x7f2f8001c520 comp rx=0 tx=0).stop 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3854870227 shutdown_connections 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3854870227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88161970 0x7f2f88161e00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3854870227 >> v2:172.21.15.98:3300/0 conn(0x7f2f8806b1f0 0x7f2f8806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3854870227 >> 172.21.15.70:0/3854870227 conn(0x7f2f880733e0 msgr2=0x7f2f88099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3854870227 shutdown_connections 2024-04-22T01:15:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3854870227 wait complete. 2024-04-22T01:15:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 Processor -- start 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- start start 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8806b1f0 0x7f2f881590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 0x7f2f880af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f88161970 0x7f2f880aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8815e230 con 0x7f2f8806efb0 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8815e0b0 con 0x7f2f8806b1f0 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8fb5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2f88162750 con 0x7f2f88161970 2024-04-22T01:15:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 0x7f2f880af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 0x7f2f880af850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50350/0 (socket says 172.21.15.70:50350) 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 -- 172.21.15.70:0/3442713977 learned_addr learned my addr 172.21.15.70:0/3442713977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 -- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f88161970 msgr2=0x7f2f880aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f88161970 0x7f2f880aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d8f8640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8806b1f0 0x7f2f881590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 -- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8806b1f0 msgr2=0x7f2f881590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8806b1f0 0x7f2f881590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 -- 172.21.15.70:0/3442713977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8814fff0 con 0x7f2f8806efb0 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f8d0f7640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 0x7f2f880af850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c001570 tx=0x7f2f7c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.414+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c018070 con 0x7f2f8806efb0 2024-04-22T01:15:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7c00a640 con 0x7f2f8806efb0 2024-04-22T01:15:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f880afd90 con 0x7f2f8806efb0 2024-04-22T01:15:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c00aa30 con 0x7f2f8806efb0 2024-04-22T01:15:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f88158600 con 0x7f2f8806efb0 2024-04-22T01:15:17.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f88068920 con 0x7f2f8806efb0 2024-04-22T01:15:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2f7c01b070 con 0x7f2f8806efb0 2024-04-22T01:15:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f867fc640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f740678c0 0x7f2f74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2f7c0856a0 con 0x7f2f8806efb0 2024-04-22T01:15:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f8d8f8640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f740678c0 0x7f2f74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.418+0000 7f2f8d8f8640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f740678c0 0x7f2f74069d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2f78006fd0 tx=0x7f2f78008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.422+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2f8814fff0 con 0x7f2f8806efb0 2024-04-22T01:15:17.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.574+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2f8815a8b0 con 0x7f2f8806efb0 2024-04-22T01:15:17.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.574+0000 7f2f867fc640 1 -- 172.21.15.70:0/3442713977 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f2f7c055ea0 con 0x7f2f8806efb0 2024-04-22T01:15:17.576 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f740678c0 msgr2=0x7f2f74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f740678c0 0x7f2f74069d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2f78006fd0 tx=0x7f2f78008040 comp rx=0 tx=0).stop 2024-04-22T01:15:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 msgr2=0x7f2f880af850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 0x7f2f880af850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c001570 tx=0x7f2f7c00cd20 comp rx=0 tx=0).stop 2024-04-22T01:15:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 shutdown_connections 2024-04-22T01:15:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f740678c0 0x7f2f74069d80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f88161970 0x7f2f880aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f8806efb0 0x7f2f880af850 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 --2- 172.21.15.70:0/3442713977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8806b1f0 0x7f2f881590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 >> 172.21.15.70:0/3442713977 conn(0x7f2f880733e0 msgr2=0x7f2f88159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 shutdown_connections 2024-04-22T01:15:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.578+0000 7f2f8fb5c640 1 -- 172.21.15.70:0/3442713977 wait complete. 2024-04-22T01:15:17.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:17.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 Processor -- start 2024-04-22T01:15:17.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 -- start start 2024-04-22T01:15:17.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 0x7f497c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f497c15f7b0 con 0x7f497c09a200 2024-04-22T01:15:17.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f497c0d0ba0 con 0x7f497c0cf6b0 2024-04-22T01:15:17.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f49839d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f497c0cd190 con 0x7f497c15f3d0 2024-04-22T01:15:17.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f4981f6d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 0x7f497c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c099cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:54204/0 (socket says 172.21.15.70:54204) 2024-04-22T01:15:17.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f4981f6d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 0x7f497c0cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:59132/0 (socket says 172.21.15.70:59132) 2024-04-22T01:15:17.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 -- 172.21.15.70:0/2992534432 learned_addr learned my addr 172.21.15.70:0/2992534432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:17.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f4980f6b640 1 --1- 172.21.15.70:0/2992534432 >> v1:172.21.15.98:6789/0 conn(0x7f497c09a200 0x7f497c09a660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:38342/0 (socket says 172.21.15.70:38342) 2024-04-22T01:15:17.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 -- 172.21.15.70:0/2992534432 >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 msgr2=0x7f497c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:17.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 --2- 172.21.15.70:0/2992534432 >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 0x7f497c0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 -- 172.21.15.70:0/2992534432 >> v1:172.21.15.98:6789/0 conn(0x7f497c09a200 legacy=0x7f497c09a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f498176c640 1 -- 172.21.15.70:0/2992534432 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c0cd3e0 con 0x7f497c15f3d0 2024-04-22T01:15:17.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.702+0000 7f4981f6d640 1 --2- 172.21.15.70:0/2992534432 >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 0x7f497c0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:17.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f498176c640 1 --2- 172.21.15.70:0/2992534432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c099cc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4978009870 tx=0x7f497801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdc744f253e5dc0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f4973fff640 1 -- 172.21.15.70:0/2992534432 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780ab650 con 0x7f497c15f3d0 2024-04-22T01:15:17.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f4973fff640 1 -- 172.21.15.70:0/2992534432 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49780b0070 con 0x7f497c15f3d0 2024-04-22T01:15:17.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f4973fff640 1 -- 172.21.15.70:0/2992534432 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780aba10 con 0x7f497c15f3d0 2024-04-22T01:15:17.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 -- 172.21.15.70:0/2992534432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 msgr2=0x7f497c099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 --2- 172.21.15.70:0/2992534432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c099cc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4978009870 tx=0x7f497801c560 comp rx=0 tx=0).stop 2024-04-22T01:15:17.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 -- 172.21.15.70:0/2992534432 shutdown_connections 2024-04-22T01:15:17.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 --2- 172.21.15.70:0/2992534432 >> v2:172.21.15.98:3300/0 conn(0x7f497c0cf6b0 0x7f497c0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 --2- 172.21.15.70:0/2992534432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c099cc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 -- 172.21.15.70:0/2992534432 >> 172.21.15.70:0/2992534432 conn(0x7f497c0b35d0 msgr2=0x7f497c0b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 -- 172.21.15.70:0/2992534432 shutdown_connections 2024-04-22T01:15:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 -- 172.21.15.70:0/2992534432 wait complete. 2024-04-22T01:15:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 Processor -- start 2024-04-22T01:15:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.706+0000 7f49839d0640 1 -- start start 2024-04-22T01:15:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c09a200 0x7f497c158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 0x7f497c14e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c1549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f497c0c6640 con 0x7f497c0cf6b0 2024-04-22T01:15:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f497c0b2b50 con 0x7f497c09a200 2024-04-22T01:15:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f497c0c67c0 con 0x7f497c15f3d0 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 0x7f497c14e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 0x7f497c14e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50360/0 (socket says 172.21.15.70:50360) 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 -- 172.21.15.70:0/3444192023 learned_addr learned my addr 172.21.15.70:0/3444192023 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f498176c640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c09a200 0x7f497c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4981f6d640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c1549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 -- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 msgr2=0x7f497c1549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c1549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 -- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c09a200 msgr2=0x7f497c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c09a200 0x7f497c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 -- 172.21.15.70:0/3444192023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c159080 con 0x7f497c0cf6b0 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f498176c640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c09a200 0x7f497c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4980f6b640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 0x7f497c14e0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f496c00c420 tx=0x7f496c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c0074e0 con 0x7f497c0cf6b0 2024-04-22T01:15:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496c007680 con 0x7f497c0cf6b0 2024-04-22T01:15:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c159360 con 0x7f497c0cf6b0 2024-04-22T01:15:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c004a30 con 0x7f497c0cf6b0 2024-04-22T01:15:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c159920 con 0x7f497c0cf6b0 2024-04-22T01:15:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c068920 con 0x7f497c0cf6b0 2024-04-22T01:15:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f496c01a020 con 0x7f497c0cf6b0 2024-04-22T01:15:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4971ffb640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49600678c0 0x7f4960069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f496c085760 con 0x7f497c0cf6b0 2024-04-22T01:15:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f498176c640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49600678c0 0x7f4960069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.710+0000 7f498176c640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49600678c0 0x7f4960069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49780af000 tx=0x7f49780a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:17.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.714+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f496c055ff0 con 0x7f497c0cf6b0 2024-04-22T01:15:17.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.910+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f497c15eb40 con 0x7f497c0cf6b0 2024-04-22T01:15:17.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.910+0000 7f4971ffb640 1 -- 172.21.15.70:0/3444192023 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f496c0558f0 con 0x7f497c0cf6b0 2024-04-22T01:15:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:17.912 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:17.912 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:17.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49600678c0 msgr2=0x7f4960069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49600678c0 0x7f4960069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49780af000 tx=0x7f49780a7000 comp rx=0 tx=0).stop 2024-04-22T01:15:17.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 msgr2=0x7f497c14e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 0x7f497c14e0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f496c00c420 tx=0x7f496c00c8e0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 shutdown_connections 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49600678c0 0x7f4960069d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f497c15f3d0 0x7f497c1549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f497c0cf6b0 0x7f497c14e0c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 --2- 172.21.15.70:0/3444192023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f497c09a200 0x7f497c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 >> 172.21.15.70:0/3444192023 conn(0x7f497c0b35d0 msgr2=0x7f497c0c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 shutdown_connections 2024-04-22T01:15:17.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:17.914+0000 7f49839d0640 1 -- 172.21.15.70:0/3444192023 wait complete. 2024-04-22T01:15:17.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:18.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.046+0000 7fa0de52e640 1 Processor -- start 2024-04-22T01:15:18.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.046+0000 7fa0de52e640 1 -- start start 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0de52e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa0d815f3d0 0x7fa0d8099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0de52e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0d80d06c0 0x7fa0d809a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0de52e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d815f7b0 con 0x7fa0d809a200 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0de52e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d80d0ba0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0de52e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d80cd190 con 0x7fa0d80d06c0 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa0d815f3d0 0x7fa0d8099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa0d815f3d0 0x7fa0d8099cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50376/0 (socket says 172.21.15.70:50376) 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 -- 172.21.15.70:0/4073229822 learned_addr learned my addr 172.21.15.70:0/4073229822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:18.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0dcacb640 1 --2- 172.21.15.70:0/4073229822 >> v2:172.21.15.98:3300/0 conn(0x7fa0d80d06c0 0x7fa0d809a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 -- 172.21.15.70:0/4073229822 >> v2:172.21.15.98:3300/0 conn(0x7fa0d80d06c0 msgr2=0x7fa0d809a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 --2- 172.21.15.70:0/4073229822 >> v2:172.21.15.98:3300/0 conn(0x7fa0d80d06c0 0x7fa0d809a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 -- 172.21.15.70:0/4073229822 >> v1:172.21.15.70:6789/0 conn(0x7fa0d809a200 legacy=0x7fa0d80c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 -- 172.21.15.70:0/4073229822 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d80cd3e0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d77fe640 1 -- 172.21.15.70:0/4073229822 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148148733 0 0) 0x7fa0d815f7b0 con 0x7fa0d809a200 2024-04-22T01:15:18.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0dcacb640 1 --2- 172.21.15.70:0/4073229822 >> v2:172.21.15.98:3300/0 conn(0x7fa0d80d06c0 0x7fa0d809a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:18.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d7fff640 1 --2- 172.21.15.70:0/4073229822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d8099cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8073980 tx=0x7fa0c001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953ba817aa4501e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d77fe640 1 -- 172.21.15.70:0/4073229822 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00b0070 con 0x7fa0d815f3d0 2024-04-22T01:15:18.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d77fe640 1 -- 172.21.15.70:0/4073229822 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c00ab600 con 0x7fa0d815f3d0 2024-04-22T01:15:18.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.050+0000 7fa0d77fe640 1 -- 172.21.15.70:0/4073229822 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00ab9d0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/4073229822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 msgr2=0x7fa0d8099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 --2- 172.21.15.70:0/4073229822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d8099cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8073980 tx=0x7fa0c001c520 comp rx=0 tx=0).stop 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/4073229822 shutdown_connections 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 --2- 172.21.15.70:0/4073229822 >> v2:172.21.15.98:3300/0 conn(0x7fa0d80d06c0 0x7fa0d809a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 --2- 172.21.15.70:0/4073229822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d8099cc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/4073229822 >> 172.21.15.70:0/4073229822 conn(0x7fa0d80b35d0 msgr2=0x7fa0d80b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/4073229822 shutdown_connections 2024-04-22T01:15:18.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/4073229822 wait complete. 2024-04-22T01:15:18.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 Processor -- start 2024-04-22T01:15:18.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- start start 2024-04-22T01:15:18.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d809a200 0x7fa0d80ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d80d06c0 0x7fa0d80cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d809b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d80c6550 con 0x7fa0d815f3d0 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d80b2b50 con 0x7fa0d809a200 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d80c66d0 con 0x7fa0d80d06c0 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d809b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d809b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50380/0 (socket says 172.21.15.70:50380) 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 -- 172.21.15.70:0/2156731909 learned_addr learned my addr 172.21.15.70:0/2156731909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0d7fff640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d809a200 0x7fa0d80ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0cf7fe640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d80d06c0 0x7fa0d80cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 -- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d80d06c0 msgr2=0x7fa0d80cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d80d06c0 0x7fa0d80cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 -- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d809a200 msgr2=0x7fa0d80ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d809a200 0x7fa0d80ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 -- 172.21.15.70:0/2156731909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d8154fc0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0dcacb640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d809b820 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000cef0 tx=0x7fa0d000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d001a070 con 0x7fa0d815f3d0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d000d860 con 0x7fa0d815f3d0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d81554b0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d000dc50 con 0x7fa0d815f3d0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d809cbe0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.054+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d8068920 con 0x7fa0d815f3d0 2024-04-22T01:15:18.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.058+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa0d001d070 con 0x7fa0d815f3d0 2024-04-22T01:15:18.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.058+0000 7fa0d57fa640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa0b80678c0 0x7fa0b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.058+0000 7fa0d7fff640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa0b80678c0 0x7fa0b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.058+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa0d00868d0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.058+0000 7fa0d7fff640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa0b80678c0 0x7fa0b8069d80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00ad000 tx=0x7fa0c00a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.062+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa0d0057030 con 0x7fa0d815f3d0 2024-04-22T01:15:18.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.210+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0d80c72f0 con 0x7fa0d815f3d0 2024-04-22T01:15:18.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.210+0000 7fa0d57fa640 1 -- 172.21.15.70:0/2156731909 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18478 (secure 0 0 0) 0x7fa0d0056930 con 0x7fa0d815f3d0 2024-04-22T01:15:18.212 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:18.212 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":24,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:14:51.117723+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa0b80678c0 msgr2=0x7fa0b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa0b80678c0 0x7fa0b8069d80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00ad000 tx=0x7fa0c00a6000 comp rx=0 tx=0).stop 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 msgr2=0x7fa0d809b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d809b820 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000cef0 tx=0x7fa0d000a710 comp rx=0 tx=0).stop 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 shutdown_connections 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa0b80678c0 0x7fa0b8069d80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d815f3d0 0x7fa0d809b820 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d80d06c0 0x7fa0d80cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 --2- 172.21.15.70:0/2156731909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d809a200 0x7fa0d80ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 >> 172.21.15.70:0/2156731909 conn(0x7fa0d80b35d0 msgr2=0x7fa0d8070fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 shutdown_connections 2024-04-22T01:15:18.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.214+0000 7fa0de52e640 1 -- 172.21.15.70:0/2156731909 wait complete. 2024-04-22T01:15:18.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:18.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 Processor -- start 2024-04-22T01:15:18.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 -- start start 2024-04-22T01:15:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd81415f180 0x7fd81406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd81406b730 0x7fd8140af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd81406efc0 con 0x7fd81415e690 2024-04-22T01:15:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd81406fbd0 con 0x7fd81406b730 2024-04-22T01:15:18.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd81941b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd81406f2a0 con 0x7fd81415f180 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd81406b730 0x7fd8140af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd81406b730 0x7fd8140af7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50392/0 (socket says 172.21.15.70:50392) 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 -- 172.21.15.70:0/2766017115 learned_addr learned my addr 172.21.15.70:0/2766017115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8127fc640 1 --2- 172.21.15.70:0/2766017115 >> v2:172.21.15.98:3300/0 conn(0x7fd81415f180 0x7fd81406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 -- 172.21.15.70:0/2766017115 >> v2:172.21.15.98:3300/0 conn(0x7fd81415f180 msgr2=0x7fd81406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 --2- 172.21.15.70:0/2766017115 >> v2:172.21.15.98:3300/0 conn(0x7fd81415f180 0x7fd81406b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 -- 172.21.15.70:0/2766017115 >> v1:172.21.15.98:6789/0 conn(0x7fd81415e690 legacy=0x7fd81415ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8137fe640 1 -- 172.21.15.70:0/2766017115 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81406f580 con 0x7fd81406b730 2024-04-22T01:15:18.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.338+0000 7fd8127fc640 1 --2- 172.21.15.70:0/2766017115 >> v2:172.21.15.98:3300/0 conn(0x7fd81415f180 0x7fd81406b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:18.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8137fe640 1 --2- 172.21.15.70:0/2766017115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81406b730 0x7fd8140af7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd814070e20 tx=0x7fd80c01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a0144df2af979 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd811ffb640 1 -- 172.21.15.70:0/2766017115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0a5670 con 0x7fd81406b730 2024-04-22T01:15:18.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd811ffb640 1 -- 172.21.15.70:0/2766017115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c0a5810 con 0x7fd81406b730 2024-04-22T01:15:18.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd811ffb640 1 -- 172.21.15.70:0/2766017115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0a5be0 con 0x7fd81406b730 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/2766017115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81406b730 msgr2=0x7fd8140af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 --2- 172.21.15.70:0/2766017115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81406b730 0x7fd8140af7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd814070e20 tx=0x7fd80c01c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/2766017115 shutdown_connections 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 --2- 172.21.15.70:0/2766017115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81406b730 0x7fd8140af7a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 --2- 172.21.15.70:0/2766017115 >> v2:172.21.15.98:3300/0 conn(0x7fd81415f180 0x7fd81406b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/2766017115 >> 172.21.15.70:0/2766017115 conn(0x7fd814099a60 msgr2=0x7fd814099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/2766017115 shutdown_connections 2024-04-22T01:15:18.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/2766017115 wait complete. 2024-04-22T01:15:18.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 Processor -- start 2024-04-22T01:15:18.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- start start 2024-04-22T01:15:18.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd81406b730 0x7fd81414bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 0x7fd8140c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd81415f180 0x7fd8140b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8140b0270 con 0x7fd81415e690 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8140b00f0 con 0x7fd81406b730 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd8140d0390 con 0x7fd81415f180 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 0x7fd8140c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 0x7fd8140c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50396/0 (socket says 172.21.15.70:50396) 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 -- 172.21.15.70:0/187505430 learned_addr learned my addr 172.21.15.70:0/187505430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8137fe640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd81415f180 0x7fd8140b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd812ffd640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd81406b730 0x7fd81414bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 -- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd81415f180 msgr2=0x7fd8140b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd81415f180 0x7fd8140b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 -- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd81406b730 msgr2=0x7fd81414bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd81406b730 0x7fd81414bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 -- 172.21.15.70:0/187505430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8140c4e60 con 0x7fd81415e690 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8137fe640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd81415f180 0x7fd8140b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd812ffd640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd81406b730 0x7fd81414bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd8127fc640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 0x7fd8140c4920 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c7f0 tx=0x7fd80800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808015070 con 0x7fd81415e690 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd808007820 con 0x7fd81415e690 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd814150650 con 0x7fd81415e690 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808007c10 con 0x7fd81415e690 2024-04-22T01:15:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.342+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd814150c10 con 0x7fd81415e690 2024-04-22T01:15:18.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.346+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd814068920 con 0x7fd81415e690 2024-04-22T01:15:18.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.346+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd808018070 con 0x7fd81415e690 2024-04-22T01:15:18.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.346+0000 7fd7fb7fe640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8000678c0 0x7fd800069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.346+0000 7fd812ffd640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8000678c0 0x7fd800069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.346+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd808086070 con 0x7fd81415e690 2024-04-22T01:15:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.346+0000 7fd812ffd640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8000678c0 0x7fd800069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd8140b33f0 tx=0x7fd804009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.350+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd808056900 con 0x7fd81415e690 2024-04-22T01:15:18.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8140cd600 con 0x7fd81415e690 2024-04-22T01:15:18.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd7fb7fe640 1 -- 172.21.15.70:0/187505430 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fd808056200 con 0x7fd81415e690 2024-04-22T01:15:18.529 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:18.529 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:18.530 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8000678c0 msgr2=0x7fd800069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8000678c0 0x7fd800069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd8140b33f0 tx=0x7fd804009040 comp rx=0 tx=0).stop 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 msgr2=0x7fd8140c4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 0x7fd8140c4920 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c7f0 tx=0x7fd80800ccb0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 shutdown_connections 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8000678c0 0x7fd800069d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd81415f180 0x7fd8140b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd81415e690 0x7fd8140c4920 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 --2- 172.21.15.70:0/187505430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd81406b730 0x7fd81414bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 >> 172.21.15.70:0/187505430 conn(0x7fd814099a60 msgr2=0x7fd814099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 shutdown_connections 2024-04-22T01:15:18.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:18.530+0000 7fd81941b640 1 -- 172.21.15.70:0/187505430 wait complete. 2024-04-22T01:15:18.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:14:46.030124+0000', 'modified': '2024-04-22T01:15:09.178266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4251': {'gid': 4251, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/3139539459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6835', 'nonce': 3139539459}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3139539459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4251, 'qdb_cluster': [4251]} 2024-04-22T01:15:18.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:15:18.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T01:15:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.082+0000 7fe7b7e99640 1 Processor -- start 2024-04-22T01:15:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.082+0000 7fe7b7e99640 1 -- start start 2024-04-22T01:15:19.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b7e99640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe7b0161b90 0x7fe7b0162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b7e99640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b006b450 con 0x7fe7b00cecd0 2024-04-22T01:15:19.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b7e99640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b00b77d0 con 0x7fe7b01615a0 2024-04-22T01:15:19.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b7e99640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b0162550 con 0x7fe7b0161b90 2024-04-22T01:15:19.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b5434640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe7b0161b90 0x7fe7b0162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b6436640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe7b00cecd0 0x7fe7b00d2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42596/0 (socket says 172.21.15.70:42596) 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b5434640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe7b0161b90 0x7fe7b0162010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50402/0 (socket says 172.21.15.70:50402) 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b6436640 1 -- 172.21.15.70:0/570201120 learned_addr learned my addr 172.21.15.70:0/570201120 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b4c33640 1 -- 172.21.15.70:0/570201120 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187487370 0 0) 0x7fe7b006b450 con 0x7fe7b00cecd0 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b4c33640 1 -- 172.21.15.70:0/570201120 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe798003530 con 0x7fe7b00cecd0 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b5434640 1 -- 172.21.15.70:0/570201120 >> v1:172.21.15.98:6789/0 conn(0x7fe7b01615a0 legacy=0x7fe7b00b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b5434640 1 -- 172.21.15.70:0/570201120 >> v1:172.21.15.70:6789/0 conn(0x7fe7b00cecd0 legacy=0x7fe7b00d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b5434640 1 -- 172.21.15.70:0/570201120 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b00ce830 con 0x7fe7b0161b90 2024-04-22T01:15:19.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b5434640 1 --2- 172.21.15.70:0/570201120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b0161b90 0x7fe7b0162010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40097c0 tx=0x7fe7a401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0aa0cbd95af69ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b4c33640 1 -- 172.21.15.70:0/570201120 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b0070 con 0x7fe7b0161b90 2024-04-22T01:15:19.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b4c33640 1 -- 172.21.15.70:0/570201120 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a40ab600 con 0x7fe7b0161b90 2024-04-22T01:15:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b4c33640 1 -- 172.21.15.70:0/570201120 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40ab9b0 con 0x7fe7b0161b90 2024-04-22T01:15:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b7e99640 1 -- 172.21.15.70:0/570201120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b0161b90 msgr2=0x7fe7b0162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.086+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/570201120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b0161b90 0x7fe7b0162010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40097c0 tx=0x7fe7a401c4a0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- 172.21.15.70:0/570201120 shutdown_connections 2024-04-22T01:15:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/570201120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b0161b90 0x7fe7b0162010 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- 172.21.15.70:0/570201120 >> 172.21.15.70:0/570201120 conn(0x7fe7b006c910 msgr2=0x7fe7b006ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- 172.21.15.70:0/570201120 shutdown_connections 2024-04-22T01:15:19.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- 172.21.15.70:0/570201120 wait complete. 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 Processor -- start 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- start start 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 0x7fe7b00b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b01615a0 0x7fe7b014bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b0161b90 0x7fe7b014b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b00d2130 con 0x7fe7b00cecd0 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b00d3450 con 0x7fe7b0161b90 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b00cf4f0 con 0x7fe7b01615a0 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 0x7fe7b00b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 0x7fe7b00b8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50418/0 (socket says 172.21.15.70:50418) 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 -- 172.21.15.70:0/366574644 learned_addr learned my addr 172.21.15.70:0/366574644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 -- 172.21.15.70:0/366574644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b01615a0 msgr2=0x7fe7b014bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b01615a0 0x7fe7b014bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 -- 172.21.15.70:0/366574644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b0161b90 msgr2=0x7fe7b014b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b0161b90 0x7fe7b014b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 -- 172.21.15.70:0/366574644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b00b5800 con 0x7fe7b00cecd0 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 0x7fe7b00b8c60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c500 tx=0x7fe7a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7ae7fc640 1 -- 172.21.15.70:0/366574644 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0005220 con 0x7fe7b00cecd0 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7ae7fc640 1 -- 172.21.15.70:0/366574644 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a00053c0 con 0x7fe7b00cecd0 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b00b5ae0 con 0x7fe7b00cecd0 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7ae7fc640 1 -- 172.21.15.70:0/366574644 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0007270 con 0x7fe7b00cecd0 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.090+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b00b60a0 con 0x7fe7b00cecd0 2024-04-22T01:15:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7ae7fc640 1 -- 172.21.15.70:0/366574644 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe7a0019020 con 0x7fe7b00cecd0 2024-04-22T01:15:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7ae7fc640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe79c0678f0 0x7fe79c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe79c0678f0 0x7fe79c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7ae7fc640 1 -- 172.21.15.70:0/366574644 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe7a0085280 con 0x7fe7b00cecd0 2024-04-22T01:15:19.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe79c0678f0 0x7fe79c069db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40ad000 tx=0x7fe7a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 Processor -- start 2024-04-22T01:15:19.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 -- start start 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b10252d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b0fdb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b0fdbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b0e295e0 con 0x55c1b1025810 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c1b0db0c50 con 0x55c1b1024ef0 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55c1b0e255a0 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b0fdb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b0fdb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50434/0 (socket says 172.21.15.70:50434) 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 -- 172.21.15.70:0/3366725380 learned_addr learned my addr 172.21.15.70:0/3366725380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b10252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 -- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 msgr2=0x55c1b0fdbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b6436640 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b0fdbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b0fdbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 -- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 msgr2=0x55c1b10252d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b10252d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 -- 172.21.15.70:0/3366725380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b0daed10 con 0x55c1b1025810 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7b5434640 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b0fdb680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4000c10 tx=0x7fe7a4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef53dfc91a2fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7977fe640 1 -- 172.21.15.70:0/3366725380 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4003b70 con 0x55c1b1025810 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7977fe640 1 -- 172.21.15.70:0/3366725380 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a40b1040 con 0x55c1b1025810 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.094+0000 7fe7977fe640 1 -- 172.21.15.70:0/3366725380 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b7450 con 0x55c1b1025810 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/3366725380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 msgr2=0x55c1b0fdb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b0fdb680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4000c10 tx=0x7fe7a4004740 comp rx=0 tx=0).stop 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/3366725380 shutdown_connections 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b0fdbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b0fdb680 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- 172.21.15.70:0/3366725380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b10252d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/3366725380 >> 172.21.15.70:0/3366725380 conn(0x55c1b0fd7b70 msgr2=0x55c1b0fd9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/3366725380 shutdown_connections 2024-04-22T01:15:19.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/3366725380 wait complete. 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- start start 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 Processor -- start 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b1009f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b100a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b100a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b0de3520 con 0x55c1b1025810 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c1b0e288f0 con 0x55c1b1024ef0 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55c1b0e2dc10 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 10 client.0 _ll_register_callbacks cb 0x55c1b0fc65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b100a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b100a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50436/0 (socket says 172.21.15.70:50436) 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 -- 172.21.15.70:0/1796428224 learned_addr learned my addr 172.21.15.70:0/1796428224 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b1009f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 -- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 msgr2=0x55c1b1009f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b1009f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 -- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 msgr2=0x55c1b100a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b100a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 -- 172.21.15.70:0/1796428224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b0dc17b0 con 0x55c1b1025810 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b1009f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b6436640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b100a9a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800d770 tx=0x7fe7a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446457041d897fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 1 -- 172.21.15.70:0/1796428224 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800bac0 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 1 -- 172.21.15.70:0/1796428224 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe7a800bc60 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7a40b1040 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 1 -- 172.21.15.70:0/1796428224 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7fe7a801f050 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 10 client.4779 fetch_fsmap learned FSMap version 9 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55c1b0db0ea0 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 1 -- 172.21.15.70:0/1796428224 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a40b1040 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 1 -- 172.21.15.70:0/1796428224 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7fe7a8022020 con 0x55c1b1025810 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 10 client.4779 fetch_fsmap finished waiting for FSMap version 9 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 10 client.4779 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 20 client.4779 populate_metadata read hostname 'smithi070' 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b1034f90 0x55c1b1037380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 4 client.4779 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 --> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1b0dc4ab0 con 0x55c1b1034f90 2024-04-22T01:15:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b1034f90 0x55c1b1037380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b1034f90 0x55c1b1037380 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fd1734b008b6907 server_cookie=cf1b20b53f97783f in_seq=0 out_seq=0 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 10 client.4779 ms_handle_connect on v2:172.21.15.70:6835/3139539459 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 1 -- 172.21.15.70:0/1796428224 <== mds.0 v2:172.21.15.70:6835/3139539459 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1b0dc4ab0 con 0x55c1b1034f90 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe794ff9640 10 client.4779 handle_command_reply: tid=0 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.098+0000 7fe7bb012000 1 client.4779 shutdown 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 msgr2=0x55c1b100a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b100a9a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800d770 tx=0x7fe7a800dc30 comp rx=0 tx=0).stop 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 shutdown_connections 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b1034f90 0x55c1b1037380 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1025810 0x55c1b100a9a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b100a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1796428224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b0fdbbc0 0x55c1b1009f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe794ff9640 0 client.4779 ms_handle_reset on v2:172.21.15.70:6835/3139539459 2024-04-22T01:15:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 >> 172.21.15.70:0/1796428224 conn(0x55c1b0fd7b70 msgr2=0x55c1b0fda7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 shutdown_connections 2024-04-22T01:15:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 1 -- 172.21.15.70:0/1796428224 wait complete. 2024-04-22T01:15:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.102+0000 7fe7bb012000 20 client.4779 trim_cache size 0 max 16384 2024-04-22T01:15:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 Processor -- start 2024-04-22T01:15:19.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 -- start start 2024-04-22T01:15:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b0fdbbc0 0x55c1b0fdaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 0x55c1b0fdb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1023f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b0204f90 con 0x55c1b1024ef0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b0203f70 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b0203920 con 0x55c1b1025810 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 0x55c1b0fdb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 0x55c1b0fdb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50442/0 (socket says 172.21.15.70:50442) 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 -- 172.21.15.70:0/1798431953 learned_addr learned my addr 172.21.15.70:0/1798431953 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 -- 172.21.15.70:0/1798431953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 msgr2=0x55c1b1023f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1023f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 -- 172.21.15.70:0/1798431953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b0fdbbc0 msgr2=0x55c1b0fdaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b0fdbbc0 0x55c1b0fdaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 -- 172.21.15.70:0/1798431953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b0db1e80 con 0x55c1b1024ef0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe7b5434640 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 0x55c1b0fdb4b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4000c10 tx=0x7fe7a40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80eedb777f5be6d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe796ffd640 1 -- 172.21.15.70:0/1798431953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4003b70 con 0x55c1b1024ef0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe796ffd640 1 -- 172.21.15.70:0/1798431953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a40b1050 con 0x55c1b1024ef0 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.118+0000 7fe796ffd640 1 -- 172.21.15.70:0/1798431953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b7450 con 0x55c1b1024ef0 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1798431953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 msgr2=0x55c1b0fdb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 0x55c1b0fdb4b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4000c10 tx=0x7fe7a40046d0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1798431953 shutdown_connections 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1023f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1024ef0 0x55c1b0fdb4b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1798431953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b0fdbbc0 0x55c1b0fdaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1798431953 >> 172.21.15.70:0/1798431953 conn(0x55c1b0fd7b70 msgr2=0x55c1b1023800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1798431953 shutdown_connections 2024-04-22T01:15:19.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1798431953 wait complete. 2024-04-22T01:15:19.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- start start 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 Processor -- start 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 0x55c1b1095210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b1095750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1037af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b0dc1dd0 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c1b0db1640 con 0x55c1b1024ef0 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55c1b0dadb10 con 0x55c1b1025810 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 10 client.0 _ll_register_callbacks cb 0x7fe7a80018e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 0x55c1b1095210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 0x55c1b1095210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50450/0 (socket says 172.21.15.70:50450) 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 -- 172.21.15.70:0/1629458884 learned_addr learned my addr 172.21.15.70:0/1629458884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b6436640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5434640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b1095750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 -- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 msgr2=0x55c1b1037af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1037af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 -- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 msgr2=0x55c1b1095750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b1095750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 -- 172.21.15.70:0/1629458884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b0df6760 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b6436640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1037af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5c35640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 0x55c1b1095210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0083500 tx=0x7fe7a0085210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86112c210a8df99a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 1 -- 172.21.15.70:0/1629458884 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0051a00 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 1 -- 172.21.15.70:0/1629458884 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe7a006d1b0 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7a40b1050 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 1 -- 172.21.15.70:0/1629458884 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7fe7a0051ba0 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 10 client.4785 fetch_fsmap learned FSMap version 9 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55c1b0e829e0 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 1 -- 172.21.15.70:0/1629458884 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a40b1050 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 1 -- 172.21.15.70:0/1629458884 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7fe7a00555f0 con 0x55c1b0fdbbc0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 10 client.4785 fetch_fsmap finished waiting for FSMap version 9 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 10 client.4785 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 20 client.4785 populate_metadata read hostname 'smithi070' 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b109c0e0 0x55c1b1038790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 4 client.4785 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 --> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] -- command(tid 0: {"prefix": "status"}) -- 0x55c1b0e36b20 con 0x55c1b109c0e0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5434640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b109c0e0 0x55c1b1038790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 10 client.4785 ms_handle_connect on v2:172.21.15.70:6835/3139539459 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7b5434640 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b109c0e0 0x55c1b1038790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6c5a52ff8d5205 server_cookie=ea1c6f1e39183af2 in_seq=0 out_seq=0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 1 -- 172.21.15.70:0/1629458884 <== mds.0 v2:172.21.15.70:6835/3139539459 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55c1b0e36b20 con 0x55c1b109c0e0 2024-04-22T01:15:19.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 10 client.4785 handle_command_reply: tid=0 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 client.4785 shutdown 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 msgr2=0x55c1b1095210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 0x55c1b1095210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0083500 tx=0x7fe7a0085210 comp rx=0 tx=0).stop 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 shutdown_connections 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:6835/3139539459,v1:172.21.15.70:6837/3139539459] conn(0x55c1b109c0e0 0x55c1b1038790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c1b1025810 0x55c1b1037af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe794ff9640 0 client.4785 ms_handle_reset on v2:172.21.15.70:6835/3139539459 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c1b1024ef0 0x55c1b1095750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 --2- 172.21.15.70:0/1629458884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b0fdbbc0 0x55c1b1095210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 >> 172.21.15.70:0/1629458884 conn(0x55c1b0fd7b70 msgr2=0x55c1b1024090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 shutdown_connections 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 1 -- 172.21.15.70:0/1629458884 wait complete. 2024-04-22T01:15:19.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.122+0000 7fe7bb012000 20 client.4785 trim_cache size 0 max 16384 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4251, 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 32.463373109999999, 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 9, 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 24, 2024-04-22T01:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 24, 2024-04-22T01:15:19.132 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 39.097964529999999 2024-04-22T01:15:19.132 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:15:19.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe79c0678f0 msgr2=0x7fe79c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe79c0678f0 0x7fe79c069db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40ad000 tx=0x7fe7a4008040 comp rx=0 tx=0).stop 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 msgr2=0x7fe7b00b8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 0x7fe7b00b8c60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c500 tx=0x7fe7a000c9c0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 shutdown_connections 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe79c0678f0 0x7fe79c069db0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b0161b90 0x7fe7b014b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b01615a0 0x7fe7b014bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 --2- 172.21.15.70:0/366574644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7b00cecd0 0x7fe7b00b8c60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 >> 172.21.15.70:0/366574644 conn(0x7fe7b006c910 msgr2=0x7fe7b00cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 shutdown_connections 2024-04-22T01:15:19.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.126+0000 7fe7b7e99640 1 -- 172.21.15.70:0/366574644 wait complete. 2024-04-22T01:15:19.140 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T01:15:19.150 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T01:15:19.150 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T01:15:19.150 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T01:15:19.150 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-22T01:15:19.167 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-22T01:15:19.167 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-22T01:15:19.167 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-22T01:15:19.167 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:15:19.167 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:19.167 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-22T01:15:19.168 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:15:19.168 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:19.168 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-22T01:15:19.168 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:15:19.168 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:19.168 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-22T01:15:19.168 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:15:19.168 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:19.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.168 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:15:19.174 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-22T01:15:19.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.174 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:15:19.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-04-22T01:15:19.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.394 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:15:19.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-22T01:15:19.462 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-04-22T01:15:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:15:19.463 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-22T01:15:19.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:15:19.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 Processor -- start 2024-04-22T01:15:19.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 -- start start 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f85b415be20 0x7f85b40cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85b40ce080 con 0x7f85b40d0570 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f85b415c200 con 0x7f85b415be20 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85bb18f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f85b40ccaa0 con 0x7f85b409a360 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f85b415be20 0x7f85b40cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f85b415be20 0x7f85b40cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50452/0 (socket says 172.21.15.70:50452) 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 -- 172.21.15.70:0/341333969 learned_addr learned my addr 172.21.15.70:0/341333969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 -- 172.21.15.70:0/341333969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 msgr2=0x7f85b40c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 --2- 172.21.15.70:0/341333969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 -- 172.21.15.70:0/341333969 >> v1:172.21.15.70:6789/0 conn(0x7f85b40d0570 legacy=0x7f85b4099c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b8f2b640 1 -- 172.21.15.70:0/341333969 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b40ccd80 con 0x7f85b415be20 2024-04-22T01:15:19.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.598+0000 7f85b37fe640 1 -- 172.21.15.70:0/341333969 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360619313 0 0) 0x7f85b40ce080 con 0x7f85b40d0570 2024-04-22T01:15:19.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b8f2b640 1 --2- 172.21.15.70:0/341333969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b40cdb40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009870 tx=0x7f85a801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9646df97b4e9cfa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b37fe640 1 -- 172.21.15.70:0/341333969 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a80aa650 con 0x7f85b415be20 2024-04-22T01:15:19.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b37fe640 1 -- 172.21.15.70:0/341333969 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a80af070 con 0x7f85b415be20 2024-04-22T01:15:19.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b37fe640 1 -- 172.21.15.70:0/341333969 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a80aaa10 con 0x7f85b415be20 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/341333969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 msgr2=0x7f85b40cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 --2- 172.21.15.70:0/341333969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b40cdb40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009870 tx=0x7f85a801c530 comp rx=0 tx=0).stop 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/341333969 shutdown_connections 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 --2- 172.21.15.70:0/341333969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 --2- 172.21.15.70:0/341333969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b40cdb40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/341333969 >> 172.21.15.70:0/341333969 conn(0x7f85b40b35a0 msgr2=0x7f85b40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/341333969 shutdown_connections 2024-04-22T01:15:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/341333969 wait complete. 2024-04-22T01:15:19.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 Processor -- start 2024-04-22T01:15:19.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- start start 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85b40d0570 0x7f85b4154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b415add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85b40cf9c0 con 0x7f85b415be20 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85b40cf840 con 0x7f85b40d0570 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f85b40d0990 con 0x7f85b409a360 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b415add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b415add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50464/0 (socket says 172.21.15.70:50464) 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 -- 172.21.15.70:0/3960663953 learned_addr learned my addr 172.21.15.70:0/3960663953 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:19.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 -- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 msgr2=0x7f85b40b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b8f2b640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b3fff640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85b40d0570 0x7f85b4154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 -- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85b40d0570 msgr2=0x7f85b4154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85b40d0570 0x7f85b4154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 -- 172.21.15.70:0/3960663953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b4155cb0 con 0x7f85b415be20 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b972c640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b415add0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00ba10 tx=0x7f85ac00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac019070 con 0x7f85b415be20 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ac00cb60 con 0x7f85b415be20 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b4154990 con 0x7f85b415be20 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac014440 con 0x7f85b415be20 2024-04-22T01:15:19.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.602+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b41551d0 con 0x7f85b415be20 2024-04-22T01:15:19.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.606+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b4068210 con 0x7f85b415be20 2024-04-22T01:15:19.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.606+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85ac01c070 con 0x7f85b415be20 2024-04-22T01:15:19.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.606+0000 7f85b17fa640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f85a00678c0 0x7f85a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.606+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f85ac085c60 con 0x7f85b415be20 2024-04-22T01:15:19.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.606+0000 7f85b8f2b640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f85a00678c0 0x7f85a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.606+0000 7f85b8f2b640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f85a00678c0 0x7f85a0069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f85a80ae000 tx=0x7f85a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.610+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f85b4154990 con 0x7f85b415be20 2024-04-22T01:15:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.810+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f85b40b7930 con 0x7f85b415be20 2024-04-22T01:15:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.810+0000 7f85b17fa640 1 -- 172.21.15.70:0/3960663953 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f85b4155cb0 con 0x7f85b415be20 2024-04-22T01:15:19.812 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:15:19.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f85a00678c0 msgr2=0x7f85a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f85a00678c0 0x7f85a0069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f85a80ae000 tx=0x7f85a8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:19.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 msgr2=0x7f85b415add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b415add0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00ba10 tx=0x7f85ac00bed0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 shutdown_connections 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f85a00678c0 0x7f85a0069d80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85b415be20 0x7f85b415add0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85b40d0570 0x7f85b4154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 --2- 172.21.15.70:0/3960663953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f85b409a360 0x7f85b40b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 >> 172.21.15.70:0/3960663953 conn(0x7f85b40b35a0 msgr2=0x7f85b4070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 shutdown_connections 2024-04-22T01:15:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19.814+0000 7f85bb18f640 1 -- 172.21.15.70:0/3960663953 wait complete. 2024-04-22T01:15:19.831 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:15:19.831 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:15:19.831 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:15:19.831 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:15:19.832 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:19.832 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:15:19.832 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:15:19.832 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:15:19.832 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:15:19.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.832 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42502sec preferred_lft 42502sec 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:15:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:19.840 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T01:15:19.840 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:19.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:19.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.968 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:15:19.977 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:15:19.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:19.978 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:15:20.027 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:20.028 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:15:20.029 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:20.029 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:20.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:15:20.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:15:20.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:15:20.029 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:20.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:20.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:20.151 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:15:20.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:20.156 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-22T01:15:20.204 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:20.205 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:20.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:20.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T01:15:20.205 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:20.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:20.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:20.318 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T01:15:20.318 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T01:15:20.319 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:20.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:20.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:20.456 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:20.456 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:20.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:15:20.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T01:15:20.457 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:20.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:20.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:20.560 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:15:20.560 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:20.560 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:20.608 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:15:20.608 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:20.608 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:20.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:15:20.718 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:15:20.767 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:20.768 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:20.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:20.769 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:15:20.816 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:15:20.817 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:15:20.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:15:20.877 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:20.928 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:15:20.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:15:20.938+0000 7f4158f46580 -1 init, newargv = 0x564290837b40 newargc=15 2024-04-22T01:15:20.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[35109]: starting ceph client 2024-04-22T01:15:20.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[35109]: starting fuse 2024-04-22T01:15:22.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:15:22.059 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:15:22.106 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:15:22.107 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:15:22.108 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:15:22.108 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:15:22.108 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:15:22.108 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:22.108 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:22.108 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:22.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:22.109 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:15:22.157 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:15:22.157 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:15:22.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:15:22.157 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:15:22.157 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:15:22.157 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:15:22.157 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:15:22.157 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:15:22.158 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:15:22.159 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:15:22.281 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.35109.asok 2024-04-22T01:15:22.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:22.443 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.35109.asok 2024-04-22T01:15:22.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:22.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.35109.asok status 2024-04-22T01:15:22.617 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:15:22.617 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:15:22.617 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:15:22.617 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:15:22.617 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:15:22.617 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "35109", 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4800, 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 4800 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:15:22.618 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3524882269 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3524882269 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.4800 192.168.0.1:0/3524882269", 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3524882269", 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 9, 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 24, 2024-04-22T01:15:22.619 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:15:22.620 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:15:22.620 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:15:22.620 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:15:22.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:15:22.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.758+0000 7fc6c7a6b640 1 Processor -- start 2024-04-22T01:15:22.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.758+0000 7fc6c7a6b640 1 -- start start 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c7a6b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc6c01455b0 0x7fc6c00cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c7a6b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6c0072230 0x7fc6c006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c7a6b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6c00d0250 con 0x7fc6c00cc150 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c7a6b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc6c00cd7b0 con 0x7fc6c01455b0 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c7a6b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc6c00cc710 con 0x7fc6c0072230 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc6c01455b0 0x7fc6c00cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc6c01455b0 0x7fc6c00cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50472/0 (socket says 172.21.15.70:50472) 2024-04-22T01:15:22.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 -- 172.21.15.70:0/647576544 learned_addr learned my addr 172.21.15.70:0/647576544 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:22.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c6008640 1 --2- 172.21.15.70:0/647576544 >> v2:172.21.15.98:3300/0 conn(0x7fc6c0072230 0x7fc6c006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 -- 172.21.15.70:0/647576544 >> v2:172.21.15.98:3300/0 conn(0x7fc6c0072230 msgr2=0x7fc6c006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 --2- 172.21.15.70:0/647576544 >> v2:172.21.15.98:3300/0 conn(0x7fc6c0072230 0x7fc6c006e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 -- 172.21.15.70:0/647576544 >> v1:172.21.15.98:6789/0 conn(0x7fc6c00cc150 legacy=0x7fc6c0071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 -- 172.21.15.70:0/647576544 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c00cc9f0 con 0x7fc6c01455b0 2024-04-22T01:15:22.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c5807640 1 --2- 172.21.15.70:0/647576544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c01455b0 0x7fc6c00cfd10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0145990 tx=0x7fc6bc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6540947750db181 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c4805640 1 -- 172.21.15.70:0/647576544 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0b0070 con 0x7fc6c01455b0 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c4805640 1 -- 172.21.15.70:0/647576544 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6bc0a4e50 con 0x7fc6c01455b0 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.762+0000 7fc6c4805640 1 -- 172.21.15.70:0/647576544 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0ab820 con 0x7fc6c01455b0 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/647576544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c01455b0 msgr2=0x7fc6c00cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/647576544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c01455b0 0x7fc6c00cfd10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0145990 tx=0x7fc6bc01c3c0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/647576544 shutdown_connections 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/647576544 >> v2:172.21.15.98:3300/0 conn(0x7fc6c0072230 0x7fc6c006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/647576544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c01455b0 0x7fc6c00cfd10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/647576544 >> 172.21.15.70:0/647576544 conn(0x7fc6c006b1f0 msgr2=0x7fc6c006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/647576544 shutdown_connections 2024-04-22T01:15:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/647576544 wait complete. 2024-04-22T01:15:22.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 Processor -- start 2024-04-22T01:15:22.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- start start 2024-04-22T01:15:22.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6c0072230 0x7fc6c00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 0x7fc6c012ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6c01455b0 0x7fc6c012b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c006f320 con 0x7fc6c00cc150 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c006f1a0 con 0x7fc6c0072230 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c0168b40 con 0x7fc6c01455b0 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 0x7fc6c012ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 0x7fc6c012ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50474/0 (socket says 172.21.15.70:50474) 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 -- 172.21.15.70:0/631930510 learned_addr learned my addr 172.21.15.70:0/631930510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5807640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6c0072230 0x7fc6c00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c6008640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6c01455b0 0x7fc6c012b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 -- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6c01455b0 msgr2=0x7fc6c012b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6c01455b0 0x7fc6c012b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 -- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6c0072230 msgr2=0x7fc6c00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6c0072230 0x7fc6c00b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 -- 172.21.15.70:0/631930510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c012ba00 con 0x7fc6c00cc150 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c5006640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 0x7fc6c012ad40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000b3e0 tx=0x7fc6b000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0015020 con 0x7fc6c00cc150 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b0011850 con 0x7fc6c00cc150 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c01f8b30 con 0x7fc6c00cc150 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0011c40 con 0x7fc6c00cc150 2024-04-22T01:15:22.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.766+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c00b0980 con 0x7fc6c00cc150 2024-04-22T01:15:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.770+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c0068210 con 0x7fc6c00cc150 2024-04-22T01:15:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.770+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6b0018020 con 0x7fc6c00cc150 2024-04-22T01:15:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.770+0000 7fc6b67fc640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6a40678f0 0x7fc6a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.770+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc6b0085ee0 con 0x7fc6c00cc150 2024-04-22T01:15:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.770+0000 7fc6c5807640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6a40678f0 0x7fc6a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.770+0000 7fc6c5807640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6a40678f0 0x7fc6a4069db0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0ad000 tx=0x7fc6bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.774+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc6b001f020 con 0x7fc6c00cc150 2024-04-22T01:15:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.974+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc6c00cbb00 con 0x7fc6c00cc150 2024-04-22T01:15:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6b67fc640 1 -- 172.21.15.70:0/631930510 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fc6b00172c0 con 0x7fc6c00cc150 2024-04-22T01:15:22.976 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6a40678f0 msgr2=0x7fc6a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6a40678f0 0x7fc6a4069db0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0ad000 tx=0x7fc6bc008040 comp rx=0 tx=0).stop 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 msgr2=0x7fc6c012ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 0x7fc6c012ad40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000b3e0 tx=0x7fc6b000b8a0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 shutdown_connections 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6a40678f0 0x7fc6a4069db0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6c01455b0 0x7fc6c012b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6c00cc150 0x7fc6c012ad40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 --2- 172.21.15.70:0/631930510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6c0072230 0x7fc6c00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 >> 172.21.15.70:0/631930510 conn(0x7fc6c006b1f0 msgr2=0x7fc6c0071b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 shutdown_connections 2024-04-22T01:15:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:22.978+0000 7fc6c7a6b640 1 -- 172.21.15.70:0/631930510 wait complete. 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:15:22.995 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:15:22.996 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-22T01:15:22.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:22.996 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:23.002 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42499sec preferred_lft 42499sec 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:15:23.003 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:15:23.004 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.005 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:15:23.053 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:15:23.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.055 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:23.109 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:23.110 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:23.110 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:15:23.110 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:23.110 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:23.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:15:23.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:15:23.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:15:23.110 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:23.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:23.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:23.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.234 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:15:23.239 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:15:23.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.240 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-22T01:15:23.289 INFO:teuthology.orchestra.run.smithi070.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-22T01:15:23.289 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:23.290 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:23.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:23.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-22T01:15:23.290 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:23.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:23.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:23.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-22T01:15:23.425 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether b2:52:ea:e6:24:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::b052:eaff:fee6:24f2/64 scope link 2024-04-22T01:15:23.426 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.427 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether b2:52:ea:e6:24:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-22T01:15:23.479 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-22T01:15:23.480 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.480 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::b052:eaff:fee6:24f2/64 scope link 2024-04-22T01:15:23.480 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:23.481 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-22T01:15:23.481 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:23.481 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:23.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-22T01:15:23.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-22T01:15:23.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-22T01:15:23.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-22T01:15:23.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-22T01:15:23.482 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:23.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:23.625 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:23.626 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:15:23.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.1 up 2024-04-22T01:15:23.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.1 master ceph-brx 2024-04-22T01:15:23.626 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:15:23.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:23.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:23.684 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-22T01:15:23.685 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:23.685 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:23.732 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-22T01:15:23.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:23.733 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:23.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:15:23.836 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:15:23.886 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:15:23.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:23.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:23.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:23.889 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:15:23.937 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:15:23.937 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2024-04-22T01:15:23.937 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-22T01:15:23.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:15:23.995 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:24.040 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:15:24.040 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:15:24.041 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:24.042 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:24.043 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:24.043 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:24.043 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:24.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:24.044 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:15:24.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:2024-04-22T01:15:24.058+0000 7f90a6175580 -1 init, newargv = 0x5599eabda8d0 newargc=15 2024-04-22T01:15:24.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[35254]: starting ceph client 2024-04-22T01:15:24.061 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:15:24.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[35254]: starting fuse 2024-04-22T01:15:25.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:15:25.179 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:25.227 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:15:25.227 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:15:25.227 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:15:25.227 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:25.228 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:15:25.229 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:25.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:25.230 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:15:25.277 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:15:25.277 INFO:teuthology.orchestra.run.smithi070.stdout:46 2024-04-22T01:15:25.277 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2024-04-22T01:15:25.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:15:25.277 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:15:25.277 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:15:25.277 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:15:25.278 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.2")) 2024-04-22T01:15:25.279 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:15:25.399 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.2.35254.asok 2024-04-22T01:15:25.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:25.411 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.35254.asok 2024-04-22T01:15:25.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:25.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.35254.asok status 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "2", 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "35254", 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:15:25.567 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4812, 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 4812 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.2:0", 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 637051188 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.2:0", 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 637051188 2024-04-22T01:15:25.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.4812 192.168.0.2:0/637051188", 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.2:0/637051188", 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 9, 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 24, 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:15:25.569 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:15:25.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:15:25.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.702+0000 7fd38b6d5640 1 Processor -- start 2024-04-22T01:15:25.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.702+0000 7fd38b6d5640 1 -- start start 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3841455b0 0x7fd3840cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3840cc150 0x7fd384071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd384072060 0x7fd38415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd3840724b0 con 0x7fd3840cc150 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd3840d0250 con 0x7fd3841455b0 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3840cd7b0 con 0x7fd384072060 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3840cc150 0x7fd384071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3840cc150 0x7fd384071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50482/0 (socket says 172.21.15.70:50482) 2024-04-22T01:15:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 -- 172.21.15.70:0/2447590363 learned_addr learned my addr 172.21.15.70:0/2447590363 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd389471640 1 --2- 172.21.15.70:0/2447590363 >> v2:172.21.15.98:3300/0 conn(0x7fd3841455b0 0x7fd3840cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd389c72640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd384072060 0x7fd38415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 -- 172.21.15.70:0/2447590363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd384072060 msgr2=0x7fd38415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd384072060 0x7fd38415f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 -- 172.21.15.70:0/2447590363 >> v2:172.21.15.98:3300/0 conn(0x7fd3841455b0 msgr2=0x7fd3840cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 --2- 172.21.15.70:0/2447590363 >> v2:172.21.15.98:3300/0 conn(0x7fd3841455b0 0x7fd3840cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 -- 172.21.15.70:0/2447590363 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3840cc710 con 0x7fd3840cc150 2024-04-22T01:15:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd389c72640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd384072060 0x7fd38415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:25.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd388c70640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840cc150 0x7fd384071b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3840d0700 tx=0x7fd37401c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d5f41d9fc46599 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:25.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd37bfff640 1 -- 172.21.15.70:0/2447590363 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740b1070 con 0x7fd3840cc150 2024-04-22T01:15:25.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd37bfff640 1 -- 172.21.15.70:0/2447590363 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3740ac440 con 0x7fd3840cc150 2024-04-22T01:15:25.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd37bfff640 1 -- 172.21.15.70:0/2447590363 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740ac810 con 0x7fd3840cc150 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- 172.21.15.70:0/2447590363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840cc150 msgr2=0x7fd384071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840cc150 0x7fd384071b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3840d0700 tx=0x7fd37401c140 comp rx=0 tx=0).stop 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- 172.21.15.70:0/2447590363 shutdown_connections 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd384072060 0x7fd38415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/2447590363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840cc150 0x7fd384071b20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/2447590363 >> v2:172.21.15.98:3300/0 conn(0x7fd3841455b0 0x7fd3840cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- 172.21.15.70:0/2447590363 >> 172.21.15.70:0/2447590363 conn(0x7fd38406b1f0 msgr2=0x7fd38406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- 172.21.15.70:0/2447590363 shutdown_connections 2024-04-22T01:15:25.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.706+0000 7fd38b6d5640 1 -- 172.21.15.70:0/2447590363 wait complete. 2024-04-22T01:15:25.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 Processor -- start 2024-04-22T01:15:25.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- start start 2024-04-22T01:15:25.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 0x7fd384161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840cc150 0x7fd384162470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3841455b0 0x7fd38413f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd38406e380 con 0x7fd384072060 2024-04-22T01:15:25.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd38406e200 con 0x7fd3840cc150 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd38406e500 con 0x7fd3841455b0 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 0x7fd384161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 0x7fd384161270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50498/0 (socket says 172.21.15.70:50498) 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 -- 172.21.15.70:0/3321805188 learned_addr learned my addr 172.21.15.70:0/3321805188 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 -- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3841455b0 msgr2=0x7fd38413f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd388c70640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840cc150 0x7fd384162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389c72640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3841455b0 0x7fd38413f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3841455b0 0x7fd38413f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 -- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840cc150 msgr2=0x7fd384162470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840cc150 0x7fd384162470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 -- 172.21.15.70:0/3321805188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38415a8a0 con 0x7fd384072060 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd389471640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 0x7fd384161270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd380007e70 tx=0x7fd38000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380014620 con 0x7fd384072060 2024-04-22T01:15:25.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd380019070 con 0x7fd384072060 2024-04-22T01:15:25.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38415ab80 con 0x7fd384072060 2024-04-22T01:15:25.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380014a00 con 0x7fd384072060 2024-04-22T01:15:25.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3841617b0 con 0x7fd384072060 2024-04-22T01:15:25.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.710+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd384068210 con 0x7fd384072060 2024-04-22T01:15:25.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.714+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd380014ba0 con 0x7fd384072060 2024-04-22T01:15:25.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.714+0000 7fd379ffb640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:25.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.714+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd380086840 con 0x7fd384072060 2024-04-22T01:15:25.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.714+0000 7fd388c70640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:25.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.714+0000 7fd388c70640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd3740ae000 tx=0x7fd3740a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:25.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.718+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd3800570d0 con 0x7fd384072060 2024-04-22T01:15:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.910+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd38415f510 con 0x7fd384072060 2024-04-22T01:15:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.910+0000 7fd379ffb640 1 -- 172.21.15.70:0/3321805188 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fd3800569d0 con 0x7fd384072060 2024-04-22T01:15:25.910 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 msgr2=0x7fd368069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd3740ae000 tx=0x7fd3740a6070 comp rx=0 tx=0).stop 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 msgr2=0x7fd384161270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 0x7fd384161270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd380007e70 tx=0x7fd38000e4b0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 shutdown_connections 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3841455b0 0x7fd38413f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840cc150 0x7fd384162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 --2- 172.21.15.70:0/3321805188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd384072060 0x7fd384161270 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 >> 172.21.15.70:0/3321805188 conn(0x7fd38406b1f0 msgr2=0x7fd3840b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 shutdown_connections 2024-04-22T01:15:25.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:25.914+0000 7fd38b6d5640 1 -- 172.21.15.70:0/3321805188 wait complete. 2024-04-22T01:15:25.927 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:15:25.927 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:15:25.927 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:15:25.927 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:15:25.927 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:25.928 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:15:25.928 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:15:25.928 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:15:25.928 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:15:25.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:25.928 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:25.935 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42500sec preferred_lft 42500sec 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:15:25.936 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:25.937 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T01:15:25.937 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:26.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:26.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:26.066 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:15:26.076 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:15:26.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:26.078 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:15:26.144 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:15:26.144 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:15:26.144 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:15:26.144 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:15:26.144 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:15:26.145 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:15:26.145 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:15:26.145 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:26.145 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:26.145 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:26.145 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:15:26.145 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:26.145 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:26.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:15:26.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:15:26.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:15:26.146 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:26.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:26.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:26.263 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:15:26.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:26.268 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-04-22T01:15:26.318 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:26.318 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:26.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:26.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-22T01:15:26.318 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:26.439 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-22T01:15:26.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-22T01:15:26.440 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:26.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:26.537 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:26.537 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:26.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:15:26.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T01:15:26.537 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:26.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:26.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:26.642 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:15:26.642 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:26.642 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:26.689 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:15:26.689 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:26.689 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:26.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:15:26.794 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:26.842 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:15:26.842 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:15:26.843 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:15:26.844 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:26.845 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:26.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:26.846 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:15:26.894 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:15:26.894 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:15:26.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:15:26.956 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:27.003 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:15:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:15:27.005 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:15:27.006 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:15:27.006 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:15:27.006 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:15:27.006 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:15:27.006 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:27.006 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:27.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:27.007 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:15:27.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:15:27.017+0000 7ffa3b3d0580 -1 init, newargv = 0x558b7b44c950 newargc=15 2024-04-22T01:15:27.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[33626]: starting ceph client 2024-04-22T01:15:27.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[33626]: starting fuse 2024-04-22T01:15:28.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:15:28.144 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:28.194 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:15:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:15:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:28.197 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:28.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:28.198 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:15:28.249 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:15:28.250 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:15:28.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:15:28.250 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:15:28.250 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:15:28.251 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:15:28.251 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:15:28.251 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:28.251 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:15:28.251 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:15:28.252 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:15:28.252 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:15:28.252 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:28.252 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:15:28.252 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:15:28.252 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:15:28.253 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:15:28.253 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:28.253 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:15:28.253 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:15:28.253 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:15:28.253 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:15:28.254 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:15:28.254 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:15:28.255 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:15:28.255 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:15:28.255 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:28.255 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:15:28.255 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:15:28.371 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.33626.asok 2024-04-22T01:15:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:28.382 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33626.asok 2024-04-22T01:15:28.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:28.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33626.asok status 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:15:28.539 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "33626", 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4597, 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4597 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:15:28.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 368279612 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 368279612 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4597 192.168.0.1:0/368279612", 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/368279612", 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 9, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 25, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:15:28.541 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:15:28.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:15:28.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 Processor -- start 2024-04-22T01:15:28.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 -- start start 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2df00aee40 0x7f2df00b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2df015bb50 0x7f2df00b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2df006dbd0 con 0x7f2df015aa30 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2df006de70 con 0x7f2df015bb50 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2df5a4c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2df006e150 con 0x7f2df00aee40 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2df015bb50 0x7f2df00b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2df015bb50 0x7f2df00b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42512/0 (socket says 172.21.15.70:42512) 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 -- 172.21.15.70:0/3632923015 learned_addr learned my addr 172.21.15.70:0/3632923015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2dee7fc640 1 --2- 172.21.15.70:0/3632923015 >> v2:172.21.15.98:3300/0 conn(0x7f2df00aee40 0x7f2df00b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 -- 172.21.15.70:0/3632923015 >> v2:172.21.15.98:3300/0 conn(0x7f2df00aee40 msgr2=0x7f2df00b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 --2- 172.21.15.70:0/3632923015 >> v2:172.21.15.98:3300/0 conn(0x7f2df00aee40 0x7f2df00b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 -- 172.21.15.70:0/3632923015 >> v1:172.21.15.98:6789/0 conn(0x7f2df015aa30 legacy=0x7f2df00ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:28.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.690+0000 7f2def7fe640 1 -- 172.21.15.70:0/3632923015 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df006e430 con 0x7f2df015bb50 2024-04-22T01:15:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2def7fe640 1 --2- 172.21.15.70:0/3632923015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015bb50 0x7f2df00b28f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2df00d07b0 tx=0x7f2de801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc096cc9ed684295 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2dedffb640 1 -- 172.21.15.70:0/3632923015 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de80a5850 con 0x7f2df015bb50 2024-04-22T01:15:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2dedffb640 1 -- 172.21.15.70:0/3632923015 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de80a59f0 con 0x7f2df015bb50 2024-04-22T01:15:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2dedffb640 1 -- 172.21.15.70:0/3632923015 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de80a5dc0 con 0x7f2df015bb50 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 -- 172.21.15.70:0/3632923015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015bb50 msgr2=0x7f2df00b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/3632923015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015bb50 0x7f2df00b28f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2df00d07b0 tx=0x7f2de801c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 -- 172.21.15.70:0/3632923015 shutdown_connections 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/3632923015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015bb50 0x7f2df00b28f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/3632923015 >> v2:172.21.15.98:3300/0 conn(0x7f2df00aee40 0x7f2df00b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 -- 172.21.15.70:0/3632923015 >> 172.21.15.70:0/3632923015 conn(0x7f2df006b1f0 msgr2=0x7f2df006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 -- 172.21.15.70:0/3632923015 shutdown_connections 2024-04-22T01:15:28.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.694+0000 7f2df5a4c640 1 -- 172.21.15.70:0/3632923015 wait complete. 2024-04-22T01:15:28.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 Processor -- start 2024-04-22T01:15:28.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- start start 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df00aee40 0x7f2df0149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 0x7f2df00b2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2df015bb50 0x7f2df00a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2df006ead0 con 0x7f2df015aa30 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2df006e950 con 0x7f2df00aee40 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2df0167a90 con 0x7f2df015bb50 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 0x7f2df00b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 0x7f2df00b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42522/0 (socket says 172.21.15.70:42522) 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 -- 172.21.15.70:0/2510711994 learned_addr learned my addr 172.21.15.70:0/2510711994 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2def7fe640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2df015bb50 0x7f2df00a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2deeffd640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df00aee40 0x7f2df0149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 -- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2df015bb50 msgr2=0x7f2df00a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2df015bb50 0x7f2df00a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 -- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df00aee40 msgr2=0x7f2df0149300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df00aee40 0x7f2df0149300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 -- 172.21.15.70:0/2510711994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df0149ce0 con 0x7f2df015aa30 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dee7fc640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 0x7f2df00b2560 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2de400d7d0 tx=0x7f2de400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de400b6c0 con 0x7f2df015aa30 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de4019070 con 0x7f2df015aa30 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df00b11e0 con 0x7f2df015aa30 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de400baa0 con 0x7f2df015aa30 2024-04-22T01:15:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df00b17a0 con 0x7f2df015aa30 2024-04-22T01:15:28.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df0068210 con 0x7f2df015aa30 2024-04-22T01:15:28.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.698+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2de400bc40 con 0x7f2df015aa30 2024-04-22T01:15:28.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.702+0000 7f2dd7fff640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ddc0678c0 0x7f2ddc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:28.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.702+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2de40864c0 con 0x7f2df015aa30 2024-04-22T01:15:28.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.702+0000 7f2deeffd640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ddc0678c0 0x7f2ddc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:28.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.702+0000 7f2deeffd640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ddc0678c0 0x7f2ddc069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2de0004e30 tx=0x7f2de0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:28.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.706+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2de4056c90 con 0x7f2df015aa30 2024-04-22T01:15:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.902+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2df00aec00 con 0x7f2df015aa30 2024-04-22T01:15:28.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2dd7fff640 1 -- 172.21.15.70:0/2510711994 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f2de4056590 con 0x7f2df015aa30 2024-04-22T01:15:28.905 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ddc0678c0 msgr2=0x7f2ddc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ddc0678c0 0x7f2ddc069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2de0004e30 tx=0x7f2de0009040 comp rx=0 tx=0).stop 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 msgr2=0x7f2df00b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 0x7f2df00b2560 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2de400d7d0 tx=0x7f2de400dc90 comp rx=0 tx=0).stop 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 shutdown_connections 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ddc0678c0 0x7f2ddc069d80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2df015bb50 0x7f2df00a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2df015aa30 0x7f2df00b2560 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 --2- 172.21.15.70:0/2510711994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df00aee40 0x7f2df0149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 >> 172.21.15.70:0/2510711994 conn(0x7f2df006b1f0 msgr2=0x7f2df015c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 shutdown_connections 2024-04-22T01:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:28.906+0000 7f2df5a4c640 1 -- 172.21.15.70:0/2510711994 wait complete. 2024-04-22T01:15:28.924 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:15:28.925 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-22T01:15:28.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:28.925 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.932 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42497sec preferred_lft 42497sec 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:15:28.933 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:28.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:28.935 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:15:28.987 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:15:28.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:28.989 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:15:29.038 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:15:29.038 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:15:29.038 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:15:29.038 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:15:29.038 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:15:29.039 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:15:29.039 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:15:29.039 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:29.039 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:29.039 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:15:29.039 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:15:29.039 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:29.039 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:29.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:15:29.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:15:29.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:15:29.039 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:29.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:29.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:29.163 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:15:29.168 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:15:29.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:29.168 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-04-22T01:15:29.214 INFO:teuthology.orchestra.run.smithi098.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-22T01:15:29.215 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:29.215 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:29.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:29.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-22T01:15:29.215 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:29.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:29.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:29.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:29.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.347 INFO:teuthology.orchestra.run.smithi098.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:15:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 7a:95:3c:38:1c:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-22T01:15:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-22T01:15:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::7895:3cff:fe38:1c32/64 scope link 2024-04-22T01:15:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:29.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-22T01:15:29.399 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 7a:95:3c:38:1c:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::7895:3cff:fe38:1c32/64 scope link 2024-04-22T01:15:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:15:29.401 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-22T01:15:29.401 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:29.401 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:29.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-22T01:15:29.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-22T01:15:29.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-22T01:15:29.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-22T01:15:29.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-22T01:15:29.402 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:29.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:29.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:29.542 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:15:29.543 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:15:29.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.1 up 2024-04-22T01:15:29.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.1 master ceph-brx 2024-04-22T01:15:29.543 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:15:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:15:29.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:29.602 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-22T01:15:29.602 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:29.602 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:29.649 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-22T01:15:29.649 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:15:29.650 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:29.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:15:29.755 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:15:29.803 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:15:29.804 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:29.805 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:29.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:29.806 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:15:29.854 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:15:29.854 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2024-04-22T01:15:29.854 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-22T01:15:29.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:15:29.913 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:15:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:15:29.964 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:29.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:29.965 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:15:29.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi098.stderr:2024-04-22T01:15:29.977+0000 7f45bf997580 -1 init, newargv = 0x55dab42dfa40 newargc=15 2024-04-22T01:15:29.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi098.stderr:ceph-fuse[33746]: starting ceph client 2024-04-22T01:15:29.982 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:15:29.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi098.stderr:ceph-fuse[33746]: starting fuse 2024-04-22T01:15:31.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:15:31.099 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:15:31.147 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:15:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:15:31.149 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:31.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.150 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:15:31.198 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:15:31.198 INFO:teuthology.orchestra.run.smithi098.stdout:46 2024-04-22T01:15:31.198 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2024-04-22T01:15:31.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:15:31.199 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.3")) 2024-04-22T01:15:31.200 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:15:31.323 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.3.33746.asok 2024-04-22T01:15:31.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:15:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:15:31.335 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.33746.asok 2024-04-22T01:15:31.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.33746.asok status 2024-04-22T01:15:31.486 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:15:31.486 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:15:31.486 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:15:31.486 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:15:31.486 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "3", 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "33746", 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4609, 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4609 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:15:31.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.2:0", 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 16951491 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.2:0", 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 16951491 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4609 192.168.0.2:0/16951491", 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.2:0/16951491", 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 9, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 26, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:15:31.488 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:15:31.489 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:15:31.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.500 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:15:31.500 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:31.508 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:15:31.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:31.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:31.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.562 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:15:31.562 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:31.610 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:15:31.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:31.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:31.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.664 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:15:31.664 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:31.672 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:15:31.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:31.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:31.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.727 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:15:31.727 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:31.775 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:15:31.775 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:31.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:31.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:31.830 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-22T01:15:31.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:31.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.977+0000 7f17a0847640 1 Processor -- start 2024-04-22T01:15:31.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.977+0000 7f17a0847640 1 -- start start 2024-04-22T01:15:31.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f17a0847640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 0x7f179c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f17a0847640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f179c0cd7b0 con 0x7f179c072060 2024-04-22T01:15:31.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f17a0847640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f179c0cc710 con 0x7f179c1455b0 2024-04-22T01:15:31.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f17a0847640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f179c0cc9f0 con 0x7f179c0cc150 2024-04-22T01:15:31.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f179c072060 0x7f179c06e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58136/0 (socket says 172.21.15.70:58136) 2024-04-22T01:15:31.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 -- 172.21.15.70:0/3022009187 learned_addr learned my addr 172.21.15.70:0/3022009187 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:31.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f1799d9b640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 0x7f179c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933799171 0 0) 0x7f179c0cd7b0 con 0x7f179c072060 2024-04-22T01:15:31.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1784003530 con 0x7f179c072060 2024-04-22T01:15:31.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 970026156 0 0) 0x7f1784003530 con 0x7f179c072060 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 msgr2=0x7f179c071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 0x7f179c071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 >> v1:172.21.15.98:6789/0 conn(0x7f179c1455b0 legacy=0x7f179c0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c0cccd0 con 0x7f179c072060 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1794001a00 con 0x7f179c072060 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 >> v1:172.21.15.70:6789/0 conn(0x7f179c072060 legacy=0x7f179c06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 0x7f17840039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1784003f30 0x7f1784004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f17840051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f179c0cc9f0 con 0x7f1784004d50 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f178400bb90 con 0x7f1784003f30 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f178400bd90 con 0x7f179c0cc150 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f17840051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179a59c640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 0x7f17840039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f1799d9b640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1784003f30 0x7f1784004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 -- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 msgr2=0x7f17840039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:31.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c0cc150 0x7f17840039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:31.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 -- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1784003f30 msgr2=0x7f1784004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:31.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1784003f30 0x7f1784004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:31.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.981+0000 7f179ad9d640 1 -- 172.21.15.70:0/3022009187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1784003530 con 0x7f1784004d50 2024-04-22T01:15:31.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179ad9d640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f17840051d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f179400a920 tx=0x7f179401d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbec913be0b783e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:31.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179400a4c0 con 0x7f1784004d50 2024-04-22T01:15:31.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17940ad650 con 0x7f1784004d50 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179959a640 1 -- 172.21.15.70:0/3022009187 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940ada80 con 0x7f1784004d50 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3022009187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 msgr2=0x7f17840051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 --2- 172.21.15.70:0/3022009187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f17840051d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f179400a920 tx=0x7f179401d420 comp rx=0 tx=0).stop 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179ad9d640 1 -- 172.21.15.70:0/3022009187 reap_dead start 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3022009187 shutdown_connections 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3022009187 >> 172.21.15.70:0/3022009187 conn(0x7f179c06b1f0 msgr2=0x7f179c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3022009187 shutdown_connections 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3022009187 wait complete. 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 Processor -- start 2024-04-22T01:15:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- start start 2024-04-22T01:15:31.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f179c15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f179c15cc00 0x7f179c13c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c13d0f0 0x7f179c13d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f179c071ca0 con 0x7f1784004d50 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f179c071b20 con 0x7f179c15cc00 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f179c071e20 con 0x7f179c13d0f0 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f179c15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f179c15b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42536/0 (socket says 172.21.15.70:42536) 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 -- 172.21.15.70:0/3953540834 learned_addr learned my addr 172.21.15.70:0/3953540834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f1799d9b640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f179c15cc00 0x7f179c13c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179ad9d640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c13d0f0 0x7f179c13d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 -- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c13d0f0 msgr2=0x7f179c13d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c13d0f0 0x7f179c13d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 -- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f179c15cc00 msgr2=0x7f179c13c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f179c15cc00 0x7f179c13c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 -- 172.21.15.70:0/3953540834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c15b9f0 con 0x7f1784004d50 2024-04-22T01:15:31.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f179a59c640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f179c15b4b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f178c010fa0 tx=0x7f178c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c018020 con 0x7f1784004d50 2024-04-22T01:15:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178c005070 con 0x7f1784004d50 2024-04-22T01:15:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c13c920 con 0x7f1784004d50 2024-04-22T01:15:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c005460 con 0x7f1784004d50 2024-04-22T01:15:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.985+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c15d4a0 con 0x7f1784004d50 2024-04-22T01:15:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.989+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f179c068210 con 0x7f1784004d50 2024-04-22T01:15:31.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.989+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f178c01b020 con 0x7f1784004d50 2024-04-22T01:15:31.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.989+0000 7f178affd640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f177c0678c0 0x7f177c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:31.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.989+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f178c089d00 con 0x7f1784004d50 2024-04-22T01:15:31.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.989+0000 7f1799d9b640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f177c0678c0 0x7f177c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:31.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.989+0000 7f1799d9b640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f177c0678c0 0x7f177c069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f179c0cbd20 tx=0x7f1790008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:31.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:31.993+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f178c022020 con 0x7f1784004d50 2024-04-22T01:15:32.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.157+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f179c15bc10 con 0x7f1784004d50 2024-04-22T01:15:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.157+0000 7f178affd640 1 -- 172.21.15.70:0/3953540834 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f178c05a5b0 con 0x7f1784004d50 2024-04-22T01:15:32.161 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:32.161 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f177c0678c0 msgr2=0x7f177c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f177c0678c0 0x7f177c069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f179c0cbd20 tx=0x7f1790008040 comp rx=0 tx=0).stop 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 msgr2=0x7f179c15b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f179c15b4b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f178c010fa0 tx=0x7f178c00e530 comp rx=0 tx=0).stop 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 shutdown_connections 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f177c0678c0 0x7f177c069d80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f179c13d0f0 0x7f179c13d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f179c15cc00 0x7f179c13c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 --2- 172.21.15.70:0/3953540834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1784004d50 0x7f179c15b4b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 >> 172.21.15.70:0/3953540834 conn(0x7f179c06b1f0 msgr2=0x7f179c0746a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 shutdown_connections 2024-04-22T01:15:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.161+0000 7f17a0847640 1 -- 172.21.15.70:0/3953540834 wait complete. 2024-04-22T01:15:32.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:32.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:32.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.301+0000 7fcba2580640 1 Processor -- start 2024-04-22T01:15:32.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba2580640 1 -- start start 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba2580640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcb9c0d08b0 0x7fcb9c0cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba2580640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb9c0cede0 0x7fcb9c0b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba2580640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb9c15c3a0 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba2580640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcb9c0ccbd0 con 0x7fcb9c0d08b0 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba2580640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcb9c0cceb0 con 0x7fcb9c0cede0 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcb9c0d08b0 0x7fcb9c0cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcb9c0d08b0 0x7fcb9c0cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42544/0 (socket says 172.21.15.70:42544) 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 -- 172.21.15.70:0/1073721293 learned_addr learned my addr 172.21.15.70:0/1073721293 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcba0b1d640 1 --2- 172.21.15.70:0/1073721293 >> v2:172.21.15.98:3300/0 conn(0x7fcb9c0cede0 0x7fcb9c0b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 -- 172.21.15.70:0/1073721293 >> v2:172.21.15.98:3300/0 conn(0x7fcb9c0cede0 msgr2=0x7fcb9c0b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 --2- 172.21.15.70:0/1073721293 >> v2:172.21.15.98:3300/0 conn(0x7fcb9c0cede0 0x7fcb9c0b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 -- 172.21.15.70:0/1073721293 >> v1:172.21.15.98:6789/0 conn(0x7fcb9c15bfc0 legacy=0x7fcb9c0d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 -- 172.21.15.70:0/1073721293 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9c0cd190 con 0x7fcb9c0d08b0 2024-04-22T01:15:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9b7fe640 1 --2- 172.21.15.70:0/1073721293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c0cf9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0cfee0 tx=0x7fcb9001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594dd2a5a34b3830 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9affd640 1 -- 172.21.15.70:0/1073721293 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb900aa650 con 0x7fcb9c0d08b0 2024-04-22T01:15:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9affd640 1 -- 172.21.15.70:0/1073721293 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb900af070 con 0x7fcb9c0d08b0 2024-04-22T01:15:32.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.305+0000 7fcb9affd640 1 -- 172.21.15.70:0/1073721293 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb900aaa10 con 0x7fcb9c0d08b0 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/1073721293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c0d08b0 msgr2=0x7fcb9c0cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 --2- 172.21.15.70:0/1073721293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c0cf9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0cfee0 tx=0x7fcb9001c4f0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/1073721293 shutdown_connections 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 --2- 172.21.15.70:0/1073721293 >> v2:172.21.15.98:3300/0 conn(0x7fcb9c0cede0 0x7fcb9c0b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 --2- 172.21.15.70:0/1073721293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c0cf9a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/1073721293 >> 172.21.15.70:0/1073721293 conn(0x7fcb9c0d1bc0 msgr2=0x7fcb9c0d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/1073721293 shutdown_connections 2024-04-22T01:15:32.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/1073721293 wait complete. 2024-04-22T01:15:32.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 Processor -- start 2024-04-22T01:15:32.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- start start 2024-04-22T01:15:32.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb9c0cede0 0x7fcb9c0b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 0x7fcb9c0c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c0b2870 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c0cc9b0 con 0x7fcb9c0d08b0 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb9c0b29f0 con 0x7fcb9c0cede0 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 0x7fcb9c0c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 0x7fcb9c0c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42558/0 (socket says 172.21.15.70:42558) 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 -- 172.21.15.70:0/4222936760 learned_addr learned my addr 172.21.15.70:0/4222936760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcb9b7fe640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 -- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb9c0cede0 msgr2=0x7fcb9c0b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcb9bfff640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb9c0cede0 0x7fcb9c0b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb9c0cede0 0x7fcb9c0b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 -- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9c0d08b0 msgr2=0x7fcb9c15e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c15e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 -- 172.21.15.70:0/4222936760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9c15f090 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcb9b7fe640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c15e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba0b1d640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 0x7fcb9c0c7280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400cd70 tx=0x7fcb9400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94003ba0 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb94003d40 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9c15f370 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94004240 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.309+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9c154320 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.313+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb9c068210 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.313+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcb940043e0 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.313+0000 7fcb98ff9640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb880678c0 0x7fcb88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.313+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcb94086b00 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.313+0000 7fcb9bfff640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb880678c0 0x7fcb88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.313+0000 7fcb9bfff640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb880678c0 0x7fcb88069d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c007fd0 tx=0x7fcb8c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.317+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcb940572d0 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcb9c0cff80 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcb98ff9640 1 -- 172.21.15.70:0/4222936760 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fcb94056bd0 con 0x7fcb9c15bfc0 2024-04-22T01:15:32.478 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb880678c0 msgr2=0x7fcb88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb880678c0 0x7fcb88069d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c007fd0 tx=0x7fcb8c009040 comp rx=0 tx=0).stop 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 msgr2=0x7fcb9c0c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 0x7fcb9c0c7280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400cd70 tx=0x7fcb9400a660 comp rx=0 tx=0).stop 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 shutdown_connections 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb880678c0 0x7fcb88069d80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb9c15bfc0 0x7fcb9c0c7280 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9c0d08b0 0x7fcb9c15e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 --2- 172.21.15.70:0/4222936760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb9c0cede0 0x7fcb9c0b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 >> 172.21.15.70:0/4222936760 conn(0x7fcb9c0d1bc0 msgr2=0x7fcb9c0d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:32.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 shutdown_connections 2024-04-22T01:15:32.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.477+0000 7fcba2580640 1 -- 172.21.15.70:0/4222936760 wait complete. 2024-04-22T01:15:32.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:32.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.617+0000 7f360124f640 1 Processor -- start 2024-04-22T01:15:32.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.617+0000 7f360124f640 1 -- start start 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35fc072280 0x7f35fc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f35fc0cd050 0x7f35fc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35fc15c290 con 0x7f35fc0b4450 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f35fc15c570 con 0x7f35fc0cd050 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f35fc15c850 con 0x7f35fc072280 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f35fc0cd050 0x7f35fc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f35fc0cd050 0x7f35fc0b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42574/0 (socket says 172.21.15.70:42574) 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f35fc0b4450 0x7f35fc071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58142/0 (socket says 172.21.15.70:58142) 2024-04-22T01:15:32.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 -- 172.21.15.70:0/2968120987 learned_addr learned my addr 172.21.15.70:0/2968120987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fa59c640 1 --2- 172.21.15.70:0/2968120987 >> v2:172.21.15.98:3300/0 conn(0x7f35fc072280 0x7f35fc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 -- 172.21.15.70:0/2968120987 >> v2:172.21.15.98:3300/0 conn(0x7f35fc072280 msgr2=0x7f35fc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 --2- 172.21.15.70:0/2968120987 >> v2:172.21.15.98:3300/0 conn(0x7f35fc072280 0x7f35fc0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 -- 172.21.15.70:0/2968120987 >> v1:172.21.15.70:6789/0 conn(0x7f35fc0b4450 legacy=0x7f35fc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 -- 172.21.15.70:0/2968120987 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc15cb30 con 0x7f35fc0cd050 2024-04-22T01:15:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35f9d9b640 1 -- 172.21.15.70:0/2968120987 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348953601 0 0) 0x7f35fc15c290 con 0x7f35fc0b4450 2024-04-22T01:15:32.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35fb59e640 1 --2- 172.21.15.70:0/2968120987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc0b2a40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f35fc0cf9a0 tx=0x7f35f401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4348c923b3ef7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35f9d9b640 1 -- 172.21.15.70:0/2968120987 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40b1070 con 0x7f35fc0cd050 2024-04-22T01:15:32.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35f9d9b640 1 -- 172.21.15.70:0/2968120987 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f40ac620 con 0x7f35fc0cd050 2024-04-22T01:15:32.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f35f9d9b640 1 -- 172.21.15.70:0/2968120987 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40ac9f0 con 0x7f35fc0cd050 2024-04-22T01:15:32.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- 172.21.15.70:0/2968120987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 msgr2=0x7f35fc0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 --2- 172.21.15.70:0/2968120987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc0b2a40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f35fc0cf9a0 tx=0x7f35f401c490 comp rx=0 tx=0).stop 2024-04-22T01:15:32.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- 172.21.15.70:0/2968120987 shutdown_connections 2024-04-22T01:15:32.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 --2- 172.21.15.70:0/2968120987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc0b2a40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 --2- 172.21.15.70:0/2968120987 >> v2:172.21.15.98:3300/0 conn(0x7f35fc072280 0x7f35fc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- 172.21.15.70:0/2968120987 >> 172.21.15.70:0/2968120987 conn(0x7f35fc0b1d40 msgr2=0x7f35fc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- 172.21.15.70:0/2968120987 shutdown_connections 2024-04-22T01:15:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.621+0000 7f360124f640 1 -- 172.21.15.70:0/2968120987 wait complete. 2024-04-22T01:15:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 Processor -- start 2024-04-22T01:15:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- start start 2024-04-22T01:15:32.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35fc072280 0x7f35fc06f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc0b4450 0x7f35fc151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc0cd980 con 0x7f35fc0cd050 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc0cd800 con 0x7f35fc0b4450 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f35fc0b3790 con 0x7f35fc072280 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42580/0 (socket says 172.21.15.70:42580) 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 -- 172.21.15.70:0/1301997604 learned_addr learned my addr 172.21.15.70:0/1301997604 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fa59c640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc0b4450 0x7f35fc151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 -- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35fc072280 msgr2=0x7f35fc06f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35fc072280 0x7f35fc06f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 -- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc0b4450 msgr2=0x7f35fc151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc0b4450 0x7f35fc151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 -- 172.21.15.70:0/1301997604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc06f8b0 con 0x7f35fc0cd050 2024-04-22T01:15:32.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fa59c640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc0b4450 0x7f35fc151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35fb59e640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc153c90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35f40190a0 tx=0x7f35f401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40b1040 con 0x7f35fc0cd050 2024-04-22T01:15:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f4004030 con 0x7f35fc0cd050 2024-04-22T01:15:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc1f9ac0 con 0x7f35fc0cd050 2024-04-22T01:15:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40043e0 con 0x7f35fc0cd050 2024-04-22T01:15:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35fc1fa080 con 0x7f35fc0cd050 2024-04-22T01:15:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.625+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35fc068210 con 0x7f35fc0cd050 2024-04-22T01:15:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.629+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f35f40b4070 con 0x7f35fc0cd050 2024-04-22T01:15:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.629+0000 7f35f37fe640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35dc0678c0 0x7f35dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.629+0000 7f35fad9d640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35dc0678c0 0x7f35dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.629+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f35f41204e0 con 0x7f35fc0cd050 2024-04-22T01:15:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.629+0000 7f35fad9d640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35dc0678c0 0x7f35dc069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f35ec001570 tx=0x7f35ec007cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.633+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f35f40bb020 con 0x7f35fc0cd050 2024-04-22T01:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35fc0d01e0 con 0x7f35fc0cd050 2024-04-22T01:15:32.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f35f37fe640 1 -- 172.21.15.70:0/1301997604 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f35f4016110 con 0x7f35fc0cd050 2024-04-22T01:15:32.794 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:32.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35dc0678c0 msgr2=0x7f35dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35dc0678c0 0x7f35dc069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f35ec001570 tx=0x7f35ec007cb0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 msgr2=0x7f35fc153c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc153c90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35f40190a0 tx=0x7f35f401acd0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 shutdown_connections 2024-04-22T01:15:32.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35dc0678c0 0x7f35dc069d80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35fc0cd050 0x7f35fc153c90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc0b4450 0x7f35fc151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 --2- 172.21.15.70:0/1301997604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35fc072280 0x7f35fc06f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 >> 172.21.15.70:0/1301997604 conn(0x7f35fc0b1d40 msgr2=0x7f35fc0cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:32.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 shutdown_connections 2024-04-22T01:15:32.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.793+0000 7f360124f640 1 -- 172.21.15.70:0/1301997604 wait complete. 2024-04-22T01:15:32.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:32.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.921+0000 7f915540e640 1 Processor -- start 2024-04-22T01:15:32.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.921+0000 7f915540e640 1 -- start start 2024-04-22T01:15:32.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f915540e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f915015de20 0x7f91500b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f915540e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f91500cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f915540e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91500cf3c0 con 0x7f91500b6db0 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f915540e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f91500b75e0 con 0x7f915015de20 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f915540e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f91500b78c0 con 0x7f91500cea10 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f915015de20 0x7f91500b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f915015de20 0x7f91500b98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42594/0 (socket says 172.21.15.70:42594) 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 -- 172.21.15.70:0/3723752537 learned_addr learned my addr 172.21.15.70:0/3723752537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 -- 172.21.15.70:0/3723752537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 msgr2=0x7f91500cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 --2- 172.21.15.70:0/3723752537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f91500cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 -- 172.21.15.70:0/3723752537 >> v1:172.21.15.98:6789/0 conn(0x7f91500b6db0 legacy=0x7f91500d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:32.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 -- 172.21.15.70:0/3723752537 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91500b7ba0 con 0x7f915015de20 2024-04-22T01:15:32.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914effd640 1 --2- 172.21.15.70:0/3723752537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91500b98a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9140009e50 tx=0x7f914001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3628b6c57d4ca071 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914dffb640 1 -- 172.21.15.70:0/3723752537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91400b4070 con 0x7f915015de20 2024-04-22T01:15:32.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914dffb640 1 -- 172.21.15.70:0/3723752537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91400af400 con 0x7f915015de20 2024-04-22T01:15:32.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.925+0000 7f914dffb640 1 -- 172.21.15.70:0/3723752537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91400af7d0 con 0x7f915015de20 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/3723752537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 msgr2=0x7f91500b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 --2- 172.21.15.70:0/3723752537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91500b98a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9140009e50 tx=0x7f914001f1e0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/3723752537 shutdown_connections 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 --2- 172.21.15.70:0/3723752537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f91500cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 --2- 172.21.15.70:0/3723752537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91500b98a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/3723752537 >> 172.21.15.70:0/3723752537 conn(0x7f9150072d20 msgr2=0x7f9150073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/3723752537 shutdown_connections 2024-04-22T01:15:32.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/3723752537 wait complete. 2024-04-22T01:15:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 Processor -- start 2024-04-22T01:15:32.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- start start 2024-04-22T01:15:32.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91500b6db0 0x7f9150153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f915015fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91501602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f915015e200 con 0x7f915015de20 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91500b8160 con 0x7f91500b6db0 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f91500b71f0 con 0x7f91500cea10 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91501602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91501602e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42598/0 (socket says 172.21.15.70:42598) 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 -- 172.21.15.70:0/901450623 learned_addr learned my addr 172.21.15.70:0/901450623 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914effd640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91500b6db0 0x7f9150153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914e7fc640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f915015fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 -- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 msgr2=0x7f915015fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f915015fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 -- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91500b6db0 msgr2=0x7f9150153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91500b6db0 0x7f9150153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 -- 172.21.15.70:0/901450623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915015efe0 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914e7fc640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f915015fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f914f7fe640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91501602e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f914800b460 tx=0x7f914800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148015020 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9148011890 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915015f2c0 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148011c80 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.929+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915015f880 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.933+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9150068210 con 0x7f915015de20 2024-04-22T01:15:32.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.933+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9148018020 con 0x7f915015de20 2024-04-22T01:15:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.933+0000 7f91377fe640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f913c0678c0 0x7f913c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.933+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f91480861e0 con 0x7f915015de20 2024-04-22T01:15:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.933+0000 7f914effd640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f913c0678c0 0x7f913c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:32.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.933+0000 7f914effd640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f913c0678c0 0x7f913c069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91400b1000 tx=0x7f91400aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:32.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:32.937+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f914801f020 con 0x7f915015de20 2024-04-22T01:15:33.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.097+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f915015fa50 con 0x7f915015de20 2024-04-22T01:15:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.097+0000 7f91377fe640 1 -- 172.21.15.70:0/901450623 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f914800d040 con 0x7f915015de20 2024-04-22T01:15:33.099 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f913c0678c0 msgr2=0x7f913c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f913c0678c0 0x7f913c069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91400b1000 tx=0x7f91400aa000 comp rx=0 tx=0).stop 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 msgr2=0x7f91501602e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91501602e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f914800b460 tx=0x7f914800b920 comp rx=0 tx=0).stop 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 shutdown_connections 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f913c0678c0 0x7f913c069d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f915015de20 0x7f91501602e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91500cea10 0x7f915015fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 --2- 172.21.15.70:0/901450623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91500b6db0 0x7f9150153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 >> 172.21.15.70:0/901450623 conn(0x7f9150072d20 msgr2=0x7f91500b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 shutdown_connections 2024-04-22T01:15:33.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.101+0000 7f915540e640 1 -- 172.21.15.70:0/901450623 wait complete. 2024-04-22T01:15:33.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:33.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 Processor -- start 2024-04-22T01:15:33.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 -- start start 2024-04-22T01:15:33.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f060815de20 0x7f06080b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f06080cea10 0x7f06080cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06080b75e0 con 0x7f06080b6db0 2024-04-22T01:15:33.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f06080b78c0 con 0x7f06080cea10 2024-04-22T01:15:33.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f060da4b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f06080b7ba0 con 0x7f060815de20 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06067fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f06080b6db0 0x7f06080d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58154/0 (socket says 172.21.15.70:58154) 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06067fc640 1 -- 172.21.15.70:0/2663614458 learned_addr learned my addr 172.21.15.70:0/2663614458 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06077fe640 1 --2- 172.21.15.70:0/2663614458 >> v2:172.21.15.70:3300/0 conn(0x7f06080cea10 0x7f06080cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f0606ffd640 1 --2- 172.21.15.70:0/2663614458 >> v2:172.21.15.98:3300/0 conn(0x7f060815de20 0x7f06080b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f0605ffb640 1 -- 172.21.15.70:0/2663614458 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242974642 0 0) 0x7f06080b75e0 con 0x7f06080b6db0 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f0605ffb640 1 -- 172.21.15.70:0/2663614458 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05f4003530 con 0x7f06080b6db0 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06077fe640 1 -- 172.21.15.70:0/2663614458 >> v2:172.21.15.98:3300/0 conn(0x7f060815de20 msgr2=0x7f06080b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06077fe640 1 --2- 172.21.15.70:0/2663614458 >> v2:172.21.15.98:3300/0 conn(0x7f060815de20 0x7f06080b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06077fe640 1 -- 172.21.15.70:0/2663614458 >> v1:172.21.15.70:6789/0 conn(0x7f06080b6db0 legacy=0x7f06080d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.241+0000 7f06077fe640 1 -- 172.21.15.70:0/2663614458 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06080b7e80 con 0x7f06080cea10 2024-04-22T01:15:33.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f06077fe640 1 --2- 172.21.15.70:0/2663614458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080cea10 0x7f06080cee60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f06080d1ac0 tx=0x7f060001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389f3450d5cb4fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f0605ffb640 1 -- 172.21.15.70:0/2663614458 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000b2070 con 0x7f06080cea10 2024-04-22T01:15:33.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f0605ffb640 1 -- 172.21.15.70:0/2663614458 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06000a59a0 con 0x7f06080cea10 2024-04-22T01:15:33.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f0605ffb640 1 -- 172.21.15.70:0/2663614458 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000a5d50 con 0x7f06080cea10 2024-04-22T01:15:33.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- 172.21.15.70:0/2663614458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080cea10 msgr2=0x7f06080cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 --2- 172.21.15.70:0/2663614458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080cea10 0x7f06080cee60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f06080d1ac0 tx=0x7f060001c900 comp rx=0 tx=0).stop 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- 172.21.15.70:0/2663614458 shutdown_connections 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 --2- 172.21.15.70:0/2663614458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080cea10 0x7f06080cee60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 --2- 172.21.15.70:0/2663614458 >> v2:172.21.15.98:3300/0 conn(0x7f060815de20 0x7f06080b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- 172.21.15.70:0/2663614458 >> 172.21.15.70:0/2663614458 conn(0x7f0608072d20 msgr2=0x7f0608073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- 172.21.15.70:0/2663614458 shutdown_connections 2024-04-22T01:15:33.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- 172.21.15.70:0/2663614458 wait complete. 2024-04-22T01:15:33.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 Processor -- start 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- start start 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 0x7f06080c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06080cea10 0x7f06081fa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f060815de20 0x7f060815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06080b7310 con 0x7f06080b6db0 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06080b7190 con 0x7f060815de20 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f060da4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f06080b8e90 con 0x7f06080cea10 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f0606ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 0x7f06080c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f0606ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 0x7f06080c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42614/0 (socket says 172.21.15.70:42614) 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.245+0000 7f0606ffd640 1 -- 172.21.15.70:0/2591598014 learned_addr learned my addr 172.21.15.70:0/2591598014 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:33.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f06077fe640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f060815de20 0x7f060815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f0606ffd640 1 -- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06080cea10 msgr2=0x7f06081fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f0606ffd640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06080cea10 0x7f06081fa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f0606ffd640 1 -- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f060815de20 msgr2=0x7f060815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f0606ffd640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f060815de20 0x7f060815c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f0606ffd640 1 -- 172.21.15.70:0/2591598014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06080c7db0 con 0x7f06080b6db0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f0606ffd640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 0x7f06080c76f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d620 tx=0x7f05f000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f000b620 con 0x7f06080b6db0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f0019070 con 0x7f06080b6db0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06080c8090 con 0x7f06080b6db0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f000b9e0 con 0x7f06080b6db0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06080c8650 con 0x7f06080b6db0 2024-04-22T01:15:33.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0608068210 con 0x7f06080b6db0 2024-04-22T01:15:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05f000bb80 con 0x7f06080b6db0 2024-04-22T01:15:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f05fffff640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05e80678c0 0x7f05e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f05f0086ff0 con 0x7f06080b6db0 2024-04-22T01:15:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f06067fc640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05e80678c0 0x7f05e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.249+0000 7f06067fc640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05e80678c0 0x7f05e8069d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05f8007fd0 tx=0x7f05f8007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.253+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f05f0057730 con 0x7f06080b6db0 2024-04-22T01:15:33.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.397+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f060806b9b0 con 0x7f06080b6db0 2024-04-22T01:15:33.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f05fffff640 1 -- 172.21.15.70:0/2591598014 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f05f0057030 con 0x7f06080b6db0 2024-04-22T01:15:33.401 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:33.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05e80678c0 msgr2=0x7f05e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05e80678c0 0x7f05e8069d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05f8007fd0 tx=0x7f05f8007900 comp rx=0 tx=0).stop 2024-04-22T01:15:33.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 msgr2=0x7f06080c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 0x7f06080c76f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d620 tx=0x7f05f000dae0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 shutdown_connections 2024-04-22T01:15:33.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05e80678c0 0x7f05e8069d80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f060815de20 0x7f060815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06080cea10 0x7f06081fa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 --2- 172.21.15.70:0/2591598014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06080b6db0 0x7f06080c76f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 >> 172.21.15.70:0/2591598014 conn(0x7f0608072d20 msgr2=0x7f06080caea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 shutdown_connections 2024-04-22T01:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.401+0000 7f060da4b640 1 -- 172.21.15.70:0/2591598014 wait complete. 2024-04-22T01:15:33.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:33.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 Processor -- start 2024-04-22T01:15:33.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 -- start start 2024-04-22T01:15:33.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83681455b0 0x7f83680cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83680cc150 0x7f8368071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83680d0250 con 0x7f8368072060 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f83680724d0 con 0x7f83681455b0 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836cd15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83680cd7b0 con 0x7f83680cc150 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83681455b0 0x7f83680cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83681455b0 0x7f83680cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42626/0 (socket says 172.21.15.70:42626) 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 -- 172.21.15.70:0/4027696024 learned_addr learned my addr 172.21.15.70:0/4027696024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 -- 172.21.15.70:0/4027696024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83680cc150 msgr2=0x7f8368071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 --2- 172.21.15.70:0/4027696024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83680cc150 0x7f8368071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 -- 172.21.15.70:0/4027696024 >> v1:172.21.15.98:6789/0 conn(0x7f8368072060 legacy=0x7f836815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:33.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 -- 172.21.15.70:0/4027696024 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83680cc710 con 0x7f83681455b0 2024-04-22T01:15:33.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.525+0000 7f836659c640 1 --2- 172.21.15.70:0/4027696024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83681455b0 0x7f83680cfd10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8358009870 tx=0x7f835801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c33a9c296d04723 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f8365d9b640 1 -- 172.21.15.70:0/4027696024 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83580b0070 con 0x7f83681455b0 2024-04-22T01:15:33.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f8365d9b640 1 -- 172.21.15.70:0/4027696024 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83580ab600 con 0x7f83681455b0 2024-04-22T01:15:33.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f8365d9b640 1 -- 172.21.15.70:0/4027696024 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83580ab9d0 con 0x7f83681455b0 2024-04-22T01:15:33.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/4027696024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83681455b0 msgr2=0x7f83680cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 --2- 172.21.15.70:0/4027696024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83681455b0 0x7f83680cfd10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8358009870 tx=0x7f835801c400 comp rx=0 tx=0).stop 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/4027696024 shutdown_connections 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 --2- 172.21.15.70:0/4027696024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83680cc150 0x7f8368071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 --2- 172.21.15.70:0/4027696024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83681455b0 0x7f83680cfd10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/4027696024 >> 172.21.15.70:0/4027696024 conn(0x7f836806b1f0 msgr2=0x7f836806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/4027696024 shutdown_connections 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/4027696024 wait complete. 2024-04-22T01:15:33.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 Processor -- start 2024-04-22T01:15:33.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- start start 2024-04-22T01:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8368072060 0x7f8368139f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 0x7f836815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83681455b0 0x7f836815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8368168b40 con 0x7f83680cc150 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f836806f210 con 0x7f8368072060 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8368168cc0 con 0x7f83681455b0 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 0x7f836815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 0x7f836815dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42634/0 (socket says 172.21.15.70:42634) 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 -- 172.21.15.70:0/3449966290 learned_addr learned my addr 172.21.15.70:0/3449966290 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f8366d9d640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83681455b0 0x7f836815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 -- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83681455b0 msgr2=0x7f836815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83681455b0 0x7f836815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 -- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8368072060 msgr2=0x7f8368139f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8368072060 0x7f8368139f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 -- 172.21.15.70:0/3449966290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836813a530 con 0x7f83680cc150 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f8366d9d640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83681455b0 0x7f836815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835ffff640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 0x7f836815dc00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f835000b550 tx=0x7f835000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350014070 con 0x7f83680cc150 2024-04-22T01:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8350010890 con 0x7f83680cc150 2024-04-22T01:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350010c80 con 0x7f83680cc150 2024-04-22T01:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f836813a810 con 0x7f83680cc150 2024-04-22T01:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.529+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f836813add0 con 0x7f83680cc150 2024-04-22T01:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.533+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8368068210 con 0x7f83680cc150 2024-04-22T01:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.533+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8350017070 con 0x7f83680cc150 2024-04-22T01:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.533+0000 7f835effd640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678c0 0x7f8348069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.533+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8350085ef0 con 0x7f83680cc150 2024-04-22T01:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.533+0000 7f836659c640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678c0 0x7f8348069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.533+0000 7f836659c640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678c0 0x7f8348069d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f83580ad000 tx=0x7f8358008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.537+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8350056620 con 0x7f83680cc150 2024-04-22T01:15:33.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.689+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f836815f0c0 con 0x7f83680cc150 2024-04-22T01:15:33.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.689+0000 7f835effd640 1 -- 172.21.15.70:0/3449966290 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f8350010e20 con 0x7f83680cc150 2024-04-22T01:15:33.692 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:33.693 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:33.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678c0 msgr2=0x7f8348069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678c0 0x7f8348069d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f83580ad000 tx=0x7f8358008040 comp rx=0 tx=0).stop 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 msgr2=0x7f836815dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 0x7f836815dc00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f835000b550 tx=0x7f835000ba10 comp rx=0 tx=0).stop 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 shutdown_connections 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678c0 0x7f8348069d80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83681455b0 0x7f836815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83680cc150 0x7f836815dc00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 --2- 172.21.15.70:0/3449966290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8368072060 0x7f8368139f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 >> 172.21.15.70:0/3449966290 conn(0x7f836806b1f0 msgr2=0x7f836806e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 shutdown_connections 2024-04-22T01:15:33.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.693+0000 7f836cd15640 1 -- 172.21.15.70:0/3449966290 wait complete. 2024-04-22T01:15:33.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:33.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:33.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 Processor -- start 2024-04-22T01:15:33.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 -- start start 2024-04-22T01:15:33.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92f4072280 0x7f92f40ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92f40cd050 0x7f92f40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f415c290 con 0x7f92f40b4450 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f92f415c570 con 0x7f92f4072280 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92fa949640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f92f415c850 con 0x7f92f40cd050 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92f4072280 0x7f92f40ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f92f40b4450 0x7f92f4071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58164/0 (socket says 172.21.15.70:58164) 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92f4072280 0x7f92f40ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42650/0 (socket says 172.21.15.70:42650) 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f3fff640 1 -- 172.21.15.70:0/2495862310 learned_addr learned my addr 172.21.15.70:0/2495862310 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f8ee6640 1 --2- 172.21.15.70:0/2495862310 >> v2:172.21.15.98:3300/0 conn(0x7f92f40cd050 0x7f92f40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f37fe640 1 -- 172.21.15.70:0/2495862310 >> v2:172.21.15.98:3300/0 conn(0x7f92f40cd050 msgr2=0x7f92f40b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f37fe640 1 --2- 172.21.15.70:0/2495862310 >> v2:172.21.15.98:3300/0 conn(0x7f92f40cd050 0x7f92f40b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f37fe640 1 -- 172.21.15.70:0/2495862310 >> v1:172.21.15.70:6789/0 conn(0x7f92f40b4450 legacy=0x7f92f4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.821+0000 7f92f37fe640 1 -- 172.21.15.70:0/2495862310 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f415cb30 con 0x7f92f4072280 2024-04-22T01:15:33.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f37fe640 1 --2- 172.21.15.70:0/2495862310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f40ccb10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92f40b27c0 tx=0x7f92e801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2429bb93cf4063a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f2ffd640 1 -- 172.21.15.70:0/2495862310 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80b0070 con 0x7f92f4072280 2024-04-22T01:15:33.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f2ffd640 1 -- 172.21.15.70:0/2495862310 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e80ab600 con 0x7f92f4072280 2024-04-22T01:15:33.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f2ffd640 1 -- 172.21.15.70:0/2495862310 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80ab9d0 con 0x7f92f4072280 2024-04-22T01:15:33.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/2495862310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 msgr2=0x7f92f40ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 --2- 172.21.15.70:0/2495862310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f40ccb10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92f40b27c0 tx=0x7f92e801c3f0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/2495862310 shutdown_connections 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 --2- 172.21.15.70:0/2495862310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f40ccb10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 --2- 172.21.15.70:0/2495862310 >> v2:172.21.15.98:3300/0 conn(0x7f92f40cd050 0x7f92f40b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/2495862310 >> 172.21.15.70:0/2495862310 conn(0x7f92f40b1d40 msgr2=0x7f92f40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/2495862310 shutdown_connections 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/2495862310 wait complete. 2024-04-22T01:15:33.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 Processor -- start 2024-04-22T01:15:33.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- start start 2024-04-22T01:15:33.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f406f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92f40b4450 0x7f92f41412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f40cd050 0x7f92f415aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40cd980 con 0x7f92f4072280 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40cd800 con 0x7f92f40cd050 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f92f40b3790 con 0x7f92f40b4450 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f406f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f406f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42664/0 (socket says 172.21.15.70:42664) 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 -- 172.21.15.70:0/3891894817 learned_addr learned my addr 172.21.15.70:0/3891894817 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 -- 172.21.15.70:0/3891894817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92f40b4450 msgr2=0x7f92f41412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92f40b4450 0x7f92f41412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 -- 172.21.15.70:0/3891894817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f40cd050 msgr2=0x7f92f415aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f40cd050 0x7f92f415aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 -- 172.21.15.70:0/3891894817 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f406e550 con 0x7f92f4072280 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f3fff640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f406f310 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c420 tx=0x7f92e400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4017070 con 0x7f92f4072280 2024-04-22T01:15:33.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e4005cf0 con 0x7f92f4072280 2024-04-22T01:15:33.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f406e830 con 0x7f92f4072280 2024-04-22T01:15:33.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e40072f0 con 0x7f92f4072280 2024-04-22T01:15:33.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.825+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f406edf0 con 0x7f92f4072280 2024-04-22T01:15:33.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.829+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f4068210 con 0x7f92f4072280 2024-04-22T01:15:33.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.829+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f92e401a070 con 0x7f92f4072280 2024-04-22T01:15:33.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.829+0000 7f92f0ff9640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92e00678c0 0x7f92e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:33.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.829+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f92e40853d0 con 0x7f92f4072280 2024-04-22T01:15:33.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.829+0000 7f92f37fe640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92e00678c0 0x7f92e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:33.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.829+0000 7f92f37fe640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92e00678c0 0x7f92e0069d80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f92e80ad000 tx=0x7f92e8004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:33.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.833+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f92e4055ba0 con 0x7f92f4072280 2024-04-22T01:15:33.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.981+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f92f40d01e0 con 0x7f92f4072280 2024-04-22T01:15:33.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.981+0000 7f92f0ff9640 1 -- 172.21.15.70:0/3891894817 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f92e40554a0 con 0x7f92f4072280 2024-04-22T01:15:33.984 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92e00678c0 msgr2=0x7f92e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92e00678c0 0x7f92e0069d80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f92e80ad000 tx=0x7f92e8004740 comp rx=0 tx=0).stop 2024-04-22T01:15:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 msgr2=0x7f92f406f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f406f310 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c420 tx=0x7f92e400c8e0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 shutdown_connections 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92e00678c0 0x7f92e0069d80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f40cd050 0x7f92f415aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92f40b4450 0x7f92f41412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 --2- 172.21.15.70:0/3891894817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92f4072280 0x7f92f406f310 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 >> 172.21.15.70:0/3891894817 conn(0x7f92f40b1d40 msgr2=0x7f92f40cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 shutdown_connections 2024-04-22T01:15:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:33.985+0000 7f92fa949640 1 -- 172.21.15.70:0/3891894817 wait complete. 2024-04-22T01:15:34.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:34.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 Processor -- start 2024-04-22T01:15:34.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 -- start start 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0b4450 0x7fb44c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb44c0720b0 0x7fb44c0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb44c0cd050 0x7fb44c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb44c072510 con 0x7fb44c0cd050 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb44c15c290 con 0x7fb44c0720b0 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb4529ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb44c15c570 con 0x7fb44c0b4450 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb44c0cd050 0x7fb44c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb44c0cd050 0x7fb44c15f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42674/0 (socket says 172.21.15.70:42674) 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 -- 172.21.15.70:0/610574991 learned_addr learned my addr 172.21.15.70:0/610574991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb44b7fe640 1 --2- 172.21.15.70:0/610574991 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0720b0 0x7fb44c0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb44bfff640 1 --2- 172.21.15.70:0/610574991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0b4450 0x7fb44c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 -- 172.21.15.70:0/610574991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0b4450 msgr2=0x7fb44c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 --2- 172.21.15.70:0/610574991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0b4450 0x7fb44c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 -- 172.21.15.70:0/610574991 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0720b0 msgr2=0x7fb44c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 --2- 172.21.15.70:0/610574991 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0720b0 0x7fb44c0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.121+0000 7fb450f49640 1 -- 172.21.15.70:0/610574991 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c15c850 con 0x7fb44c0cd050 2024-04-22T01:15:34.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb450f49640 1 --2- 172.21.15.70:0/610574991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0cd050 0x7fb44c15f770 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb44c06fe00 tx=0x7fb44401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af474b10a607e540 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb44affd640 1 -- 172.21.15.70:0/610574991 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4440a5510 con 0x7fb44c0cd050 2024-04-22T01:15:34.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb44affd640 1 -- 172.21.15.70:0/610574991 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4440a56b0 con 0x7fb44c0cd050 2024-04-22T01:15:34.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb44affd640 1 -- 172.21.15.70:0/610574991 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4440a5a80 con 0x7fb44c0cd050 2024-04-22T01:15:34.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 -- 172.21.15.70:0/610574991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0cd050 msgr2=0x7fb44c15f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 --2- 172.21.15.70:0/610574991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0cd050 0x7fb44c15f770 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb44c06fe00 tx=0x7fb44401c700 comp rx=0 tx=0).stop 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 -- 172.21.15.70:0/610574991 shutdown_connections 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 --2- 172.21.15.70:0/610574991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0cd050 0x7fb44c15f770 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 --2- 172.21.15.70:0/610574991 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0720b0 0x7fb44c0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 --2- 172.21.15.70:0/610574991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0b4450 0x7fb44c071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 -- 172.21.15.70:0/610574991 >> 172.21.15.70:0/610574991 conn(0x7fb44c0b1d40 msgr2=0x7fb44c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 -- 172.21.15.70:0/610574991 shutdown_connections 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 -- 172.21.15.70:0/610574991 wait complete. 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 Processor -- start 2024-04-22T01:15:34.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.125+0000 7fb4529ac640 1 -- start start 2024-04-22T01:15:34.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0720b0 0x7fb44c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 0x7fb44c0b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0cd050 0x7fb44c0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c0b49b0 con 0x7fb44c0b4450 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c0b4830 con 0x7fb44c0cd050 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb44c0cd800 con 0x7fb44c0720b0 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 0x7fb44c0b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 0x7fb44c0b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42682/0 (socket says 172.21.15.70:42682) 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 -- 172.21.15.70:0/2095762202 learned_addr learned my addr 172.21.15.70:0/2095762202 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb450f49640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0cd050 0x7fb44c0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44bfff640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0720b0 0x7fb44c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 -- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0720b0 msgr2=0x7fb44c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0720b0 0x7fb44c14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 -- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0cd050 msgr2=0x7fb44c0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0cd050 0x7fb44c0b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 -- 172.21.15.70:0/2095762202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c1f58d0 con 0x7fb44c0b4450 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb450f49640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0cd050 0x7fb44c0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44b7fe640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 0x7fb44c0b2e10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb44000e8a0 tx=0x7fb44000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440017070 con 0x7fb44c0b4450 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb44000ee90 con 0x7fb44c0b4450 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb44c147d70 con 0x7fb44c0b4450 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44000cb80 con 0x7fb44c0b4450 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44c148330 con 0x7fb44c0b4450 2024-04-22T01:15:34.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb44c068210 con 0x7fb44c0b4450 2024-04-22T01:15:34.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb44001a070 con 0x7fb44c0b4450 2024-04-22T01:15:34.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb448ff9640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb4380678c0 0x7fb438069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb44bfff640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb4380678c0 0x7fb438069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.129+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb440085e80 con 0x7fb44c0b4450 2024-04-22T01:15:34.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.133+0000 7fb44bfff640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb4380678c0 0x7fb438069d80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb44c14efe0 tx=0x7fb43c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.133+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb440056650 con 0x7fb44c0b4450 2024-04-22T01:15:34.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.289+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb44c0cce30 con 0x7fb44c0b4450 2024-04-22T01:15:34.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb448ff9640 1 -- 172.21.15.70:0/2095762202 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fb440055f50 con 0x7fb44c0b4450 2024-04-22T01:15:34.293 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb4380678c0 msgr2=0x7fb438069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb4380678c0 0x7fb438069d80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb44c14efe0 tx=0x7fb43c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 msgr2=0x7fb44c0b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 0x7fb44c0b2e10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb44000e8a0 tx=0x7fb44000ed60 comp rx=0 tx=0).stop 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 shutdown_connections 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb4380678c0 0x7fb438069d80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0cd050 0x7fb44c0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb44c0b4450 0x7fb44c0b2e10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 --2- 172.21.15.70:0/2095762202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c0720b0 0x7fb44c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 >> 172.21.15.70:0/2095762202 conn(0x7fb44c0b1d40 msgr2=0x7fb44c15d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 shutdown_connections 2024-04-22T01:15:34.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.293+0000 7fb4529ac640 1 -- 172.21.15.70:0/2095762202 wait complete. 2024-04-22T01:15:34.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:34.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.429+0000 7f6a16be7640 1 Processor -- start 2024-04-22T01:15:34.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.429+0000 7f6a16be7640 1 -- start start 2024-04-22T01:15:34.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a1015de20 0x7f6a100b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a100b6db0 0x7f6a100d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a100b75e0 con 0x7f6a100cea10 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6a100b78c0 con 0x7f6a1015de20 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6a100b7ba0 con 0x7f6a100b6db0 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a14983640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a1015de20 0x7f6a100b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a15184640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6a100cea10 0x7f6a100ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58176/0 (socket says 172.21.15.70:58176) 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a15184640 1 -- 172.21.15.70:0/1199822668 learned_addr learned my addr 172.21.15.70:0/1199822668 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a0ffff640 1 --2- 172.21.15.70:0/1199822668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a100b6db0 0x7f6a100d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a14983640 1 -- 172.21.15.70:0/1199822668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a100b6db0 msgr2=0x7f6a100d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a14983640 1 --2- 172.21.15.70:0/1199822668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a100b6db0 0x7f6a100d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a14983640 1 -- 172.21.15.70:0/1199822668 >> v1:172.21.15.70:6789/0 conn(0x7f6a100cea10 legacy=0x7f6a100ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a14983640 1 -- 172.21.15.70:0/1199822668 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a100b7e80 con 0x7f6a1015de20 2024-04-22T01:15:34.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a14983640 1 --2- 172.21.15.70:0/1199822668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a1015de20 0x7f6a100b98a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a040097f0 tx=0x7f6a0401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faff45d32eaeadf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a0f7fe640 1 -- 172.21.15.70:0/1199822668 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a040b5070 con 0x7f6a1015de20 2024-04-22T01:15:34.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a0f7fe640 1 -- 172.21.15.70:0/1199822668 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a040a8990 con 0x7f6a1015de20 2024-04-22T01:15:34.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a0f7fe640 1 -- 172.21.15.70:0/1199822668 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a040a8d60 con 0x7f6a1015de20 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- 172.21.15.70:0/1199822668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a1015de20 msgr2=0x7f6a100b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 --2- 172.21.15.70:0/1199822668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a1015de20 0x7f6a100b98a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a040097f0 tx=0x7f6a0401ec90 comp rx=0 tx=0).stop 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- 172.21.15.70:0/1199822668 shutdown_connections 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 --2- 172.21.15.70:0/1199822668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a100b6db0 0x7f6a100d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 --2- 172.21.15.70:0/1199822668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a1015de20 0x7f6a100b98a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- 172.21.15.70:0/1199822668 >> 172.21.15.70:0/1199822668 conn(0x7f6a10072d20 msgr2=0x7f6a10073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- 172.21.15.70:0/1199822668 shutdown_connections 2024-04-22T01:15:34.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.433+0000 7f6a16be7640 1 -- 172.21.15.70:0/1199822668 wait complete. 2024-04-22T01:15:34.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 Processor -- start 2024-04-22T01:15:34.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- start start 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a100b6db0 0x7f6a101515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 0x7f6a100c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a1015de20 0x7f6a100b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a100d22c0 con 0x7f6a100cea10 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1015e200 con 0x7f6a100b6db0 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6a100b71f0 con 0x7f6a1015de20 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 0x7f6a100c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 0x7f6a100c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42694/0 (socket says 172.21.15.70:42694) 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 -- 172.21.15.70:0/257341007 learned_addr learned my addr 172.21.15.70:0/257341007 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a15184640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a1015de20 0x7f6a100b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 -- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a1015de20 msgr2=0x7f6a100b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a1015de20 0x7f6a100b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 -- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a100b6db0 msgr2=0x7f6a101515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a14983640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a100b6db0 0x7f6a101515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a100b6db0 0x7f6a101515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 -- 172.21.15.70:0/257341007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a10151b00 con 0x7f6a100cea10 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0ffff640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 0x7f6a100c7fd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c820 tx=0x7f69fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc015070 con 0x7f6a100cea10 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69fc004240 con 0x7f6a100cea10 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc00ce10 con 0x7f6a100cea10 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a100caf40 con 0x7f6a100cea10 2024-04-22T01:15:34.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a100cb500 con 0x7f6a100cea10 2024-04-22T01:15:34.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a10068210 con 0x7f6a100cea10 2024-04-22T01:15:34.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.437+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69fc007830 con 0x7f6a100cea10 2024-04-22T01:15:34.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.441+0000 7f6a0d7fa640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69f40678c0 0x7f69f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.441+0000 7f6a14983640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69f40678c0 0x7f69f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.441+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f69fc085e80 con 0x7f6a100cea10 2024-04-22T01:15:34.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.441+0000 7f6a14983640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69f40678c0 0x7f69f4069d80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a040b2000 tx=0x7f6a040aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.445+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f69fc056650 con 0x7f6a100cea10 2024-04-22T01:15:34.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.609+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6a100b3c00 con 0x7f6a100cea10 2024-04-22T01:15:34.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a0d7fa640 1 -- 172.21.15.70:0/257341007 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f69fc055f50 con 0x7f6a100cea10 2024-04-22T01:15:34.613 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:34.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69f40678c0 msgr2=0x7f69f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69f40678c0 0x7f69f4069d80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a040b2000 tx=0x7f6a040aa070 comp rx=0 tx=0).stop 2024-04-22T01:15:34.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 msgr2=0x7f6a100c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 0x7f6a100c7fd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c820 tx=0x7f69fc00cce0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 shutdown_connections 2024-04-22T01:15:34.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69f40678c0 0x7f69f4069d80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a1015de20 0x7f6a100b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100cea10 0x7f6a100c7fd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 --2- 172.21.15.70:0/257341007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a100b6db0 0x7f6a101515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 >> 172.21.15.70:0/257341007 conn(0x7f6a10072d20 msgr2=0x7f6a100744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 shutdown_connections 2024-04-22T01:15:34.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.613+0000 7f6a16be7640 1 -- 172.21.15.70:0/257341007 wait complete. 2024-04-22T01:15:34.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:34.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 Processor -- start 2024-04-22T01:15:34.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 -- start start 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb740b98a0 0x7fbb740d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb740cea10 0x7fbb740cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb740b9ce0 con 0x7fbb7415de20 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbb740b75e0 con 0x7fbb740b98a0 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.757+0000 7fbb7c488640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbb740b78c0 con 0x7fbb740cea10 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb740b98a0 0x7fbb740d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb740b98a0 0x7fbb740d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42702/0 (socket says 172.21.15.70:42702) 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 -- 172.21.15.70:0/4214823084 learned_addr learned my addr 172.21.15.70:0/4214823084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 -- 172.21.15.70:0/4214823084 >> v2:172.21.15.98:3300/0 conn(0x7fbb740cea10 msgr2=0x7fbb740cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 --2- 172.21.15.70:0/4214823084 >> v2:172.21.15.98:3300/0 conn(0x7fbb740cea10 0x7fbb740cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 -- 172.21.15.70:0/4214823084 >> v1:172.21.15.98:6789/0 conn(0x7fbb7415de20 legacy=0x7fbb740b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 -- 172.21.15.70:0/4214823084 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb740b7ba0 con 0x7fbb740b98a0 2024-04-22T01:15:34.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79a23640 1 --2- 172.21.15.70:0/4214823084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740b98a0 0x7fbb740d1d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb740b6d00 tx=0x7fbb6401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6502ec79b44be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79222640 1 -- 172.21.15.70:0/4214823084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb640aa610 con 0x7fbb740b98a0 2024-04-22T01:15:34.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79222640 1 -- 172.21.15.70:0/4214823084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb640af070 con 0x7fbb740b98a0 2024-04-22T01:15:34.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb79222640 1 -- 172.21.15.70:0/4214823084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb640aa9d0 con 0x7fbb740b98a0 2024-04-22T01:15:34.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 -- 172.21.15.70:0/4214823084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740b98a0 msgr2=0x7fbb740d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 --2- 172.21.15.70:0/4214823084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740b98a0 0x7fbb740d1d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb740b6d00 tx=0x7fbb6401c4b0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 -- 172.21.15.70:0/4214823084 shutdown_connections 2024-04-22T01:15:34.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 --2- 172.21.15.70:0/4214823084 >> v2:172.21.15.98:3300/0 conn(0x7fbb740cea10 0x7fbb740cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 --2- 172.21.15.70:0/4214823084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740b98a0 0x7fbb740d1d80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 -- 172.21.15.70:0/4214823084 >> 172.21.15.70:0/4214823084 conn(0x7fbb74072d20 msgr2=0x7fbb74073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 -- 172.21.15.70:0/4214823084 shutdown_connections 2024-04-22T01:15:34.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.761+0000 7fbb7c488640 1 -- 172.21.15.70:0/4214823084 wait complete. 2024-04-22T01:15:34.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 Processor -- start 2024-04-22T01:15:34.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- start start 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb740b98a0 0x7fbb74159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 0x7fbb7415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb7415de20 0x7fbb7415d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb740d22c0 con 0x7fbb740cea10 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7415e200 con 0x7fbb7415de20 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbb740b8e90 con 0x7fbb740b98a0 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 0x7fbb7415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 0x7fbb7415cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42708/0 (socket says 172.21.15.70:42708) 2024-04-22T01:15:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 -- 172.21.15.70:0/1400611811 learned_addr learned my addr 172.21.15.70:0/1400611811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7aa25640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb7415de20 0x7fbb7415d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 -- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb740b98a0 msgr2=0x7fbb74159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb740b98a0 0x7fbb74159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 -- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb7415de20 msgr2=0x7fbb7415d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb7415de20 0x7fbb7415d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 -- 172.21.15.70:0/1400611811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb740cd6c0 con 0x7fbb740cea10 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7aa25640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb7415de20 0x7fbb7415d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb79a23640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 0x7fbb7415cb00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb6401ab60 tx=0x7fbb6401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb640b3800 con 0x7fbb740cea10 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb640b39c0 con 0x7fbb740cea10 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb740cb570 con 0x7fbb740cea10 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb640b3db0 con 0x7fbb740cea10 2024-04-22T01:15:34.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb740cb290 con 0x7fbb740cea10 2024-04-22T01:15:34.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb74068210 con 0x7fbb740cea10 2024-04-22T01:15:34.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.765+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb640c63c0 con 0x7fbb740cea10 2024-04-22T01:15:34.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.769+0000 7fbb6affd640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb580678c0 0x7fbb58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.769+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fbb64120a10 con 0x7fbb740cea10 2024-04-22T01:15:34.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.769+0000 7fbb7a224640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb580678c0 0x7fbb58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.769+0000 7fbb7a224640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb580678c0 0x7fbb58069d80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009650 tx=0x7fbb70006bb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.769+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbb640c3060 con 0x7fbb740cea10 2024-04-22T01:15:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbb740cf180 con 0x7fbb740cea10 2024-04-22T01:15:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb6affd640 1 -- 172.21.15.70:0/1400611811 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fbb640a7070 con 0x7fbb740cea10 2024-04-22T01:15:34.934 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb580678c0 msgr2=0x7fbb58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb580678c0 0x7fbb58069d80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009650 tx=0x7fbb70006bb0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 msgr2=0x7fbb7415cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 0x7fbb7415cb00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb6401ab60 tx=0x7fbb6401acf0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 shutdown_connections 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb580678c0 0x7fbb58069d80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb7415de20 0x7fbb7415d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb740cea10 0x7fbb7415cb00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 --2- 172.21.15.70:0/1400611811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb740b98a0 0x7fbb74159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 >> 172.21.15.70:0/1400611811 conn(0x7fbb74072d20 msgr2=0x7fbb740b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 shutdown_connections 2024-04-22T01:15:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:34.933+0000 7fbb7c488640 1 -- 172.21.15.70:0/1400611811 wait complete. 2024-04-22T01:15:34.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:35.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.069+0000 7f006c3e9640 1 Processor -- start 2024-04-22T01:15:35.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.069+0000 7f006c3e9640 1 -- start start 2024-04-22T01:15:35.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006c3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 0x7f00640cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006c3e9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00640d0250 con 0x7f0064072230 2024-04-22T01:15:35.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006c3e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00640cd7b0 con 0x7f00640cc150 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006c3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f00640cc710 con 0x7f00641455b0 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a986640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0064072230 0x7f006406e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58186/0 (socket says 172.21.15.70:58186) 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a986640 1 -- 172.21.15.70:0/116019192 learned_addr learned my addr 172.21.15.70:0/116019192 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491135389 0 0) 0x7f00640d0250 con 0x7f0064072230 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0050003530 con 0x7f0064072230 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 0x7f00640cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3752182048 0 0) 0x7f0050003530 con 0x7f0064072230 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 msgr2=0x7f00640cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 0x7f00640cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 >> v1:172.21.15.98:6789/0 conn(0x7f00640cc150 legacy=0x7f0064071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00640cc9f0 con 0x7f0064072230 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f005c001a00 con 0x7f0064072230 2024-04-22T01:15:35.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 >> v1:172.21.15.70:6789/0 conn(0x7f0064072230 legacy=0x7f006406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00500039f0 0x7f0050003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 0x7f0050004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0050004da0 0x7f0050005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00640cc710 con 0x7f00500039f0 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f005000bc50 con 0x7f0050004da0 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f005000be50 con 0x7f00641455b0 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00500039f0 0x7f0050003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 -- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 msgr2=0x7f0050004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f0069984640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 0x7f0050004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a986640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0050004da0 0x7f0050005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00641455b0 0x7f0050004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 -- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0050004da0 msgr2=0x7f0050005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0050004da0 0x7f0050005220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.073+0000 7f006a185640 1 -- 172.21.15.70:0/116019192 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0050003530 con 0x7f00500039f0 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006a185640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00500039f0 0x7f0050003e10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f006000ba00 tx=0x7f006001c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794fc0e1e0c32c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006000b600 con 0x7f00500039f0 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00600af620 con 0x7f00500039f0 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069183640 1 -- 172.21.15.70:0/116019192 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00600afa10 con 0x7f00500039f0 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/116019192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00500039f0 msgr2=0x7f0050003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 --2- 172.21.15.70:0/116019192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00500039f0 0x7f0050003e10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f006000ba00 tx=0x7f006001c640 comp rx=0 tx=0).stop 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006a986640 1 -- 172.21.15.70:0/116019192 reap_dead start 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/116019192 shutdown_connections 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/116019192 >> 172.21.15.70:0/116019192 conn(0x7f006406b1f0 msgr2=0x7f006406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/116019192 shutdown_connections 2024-04-22T01:15:35.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/116019192 wait complete. 2024-04-22T01:15:35.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 Processor -- start 2024-04-22T01:15:35.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- start start 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00500039f0 0x7f006415b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 0x7f0064163b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0064143130 0x7f00641435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0064167ed0 con 0x7f00640a8970 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0064167d50 con 0x7f00500039f0 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0064168050 con 0x7f0064143130 2024-04-22T01:15:35.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 0x7f0064163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 0x7f0064163b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42738/0 (socket says 172.21.15.70:42738) 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 -- 172.21.15.70:0/373535055 learned_addr learned my addr 172.21.15.70:0/373535055 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006a185640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00500039f0 0x7f006415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 -- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0064143130 msgr2=0x7f00641435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006a986640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0064143130 0x7f00641435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0064143130 0x7f00641435c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 -- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00500039f0 msgr2=0x7f006415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00500039f0 0x7f006415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 -- 172.21.15.70:0/373535055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0064164d70 con 0x7f00640a8970 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006a185640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00500039f0 0x7f006415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f0069984640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 0x7f0064163b60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f005400d570 tx=0x7f005400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054005480 con 0x7f00640a8970 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0054017070 con 0x7f00640a8970 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00641644d0 con 0x7f00640a8970 2024-04-22T01:15:35.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054003820 con 0x7f00640a8970 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.077+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0064164a90 con 0x7f00640a8970 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.081+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0064068210 con 0x7f00640a8970 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.081+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00540039c0 con 0x7f00640a8970 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.081+0000 7f005affd640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00480678c0 0x7f0048069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.081+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0054085e20 con 0x7f00640a8970 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.081+0000 7f006a185640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00480678c0 0x7f0048069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.081+0000 7f006a185640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00480678c0 0x7f0048069d80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f00600b0000 tx=0x7f00600046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.085+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0054019310 con 0x7f00640a8970 2024-04-22T01:15:35.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.245+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00640cbb00 con 0x7f00640a8970 2024-04-22T01:15:35.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f005affd640 1 -- 172.21.15.70:0/373535055 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f00540565f0 con 0x7f00640a8970 2024-04-22T01:15:35.250 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:35.250 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:35.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00480678c0 msgr2=0x7f0048069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00480678c0 0x7f0048069d80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f00600b0000 tx=0x7f00600046d0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 msgr2=0x7f0064163b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 0x7f0064163b60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f005400d570 tx=0x7f005400da30 comp rx=0 tx=0).stop 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 shutdown_connections 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00480678c0 0x7f0048069d80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0064143130 0x7f00641435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00640a8970 0x7f0064163b60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 --2- 172.21.15.70:0/373535055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00500039f0 0x7f006415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 >> 172.21.15.70:0/373535055 conn(0x7f006406b1f0 msgr2=0x7f00640af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 shutdown_connections 2024-04-22T01:15:35.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.249+0000 7f006c3e9640 1 -- 172.21.15.70:0/373535055 wait complete. 2024-04-22T01:15:35.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:35.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.381+0000 7fe8ce8c6640 1 Processor -- start 2024-04-22T01:15:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.381+0000 7fe8ce8c6640 1 -- start start 2024-04-22T01:15:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8ce8c6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe8c815de20 0x7fe8c80b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8ce8c6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8c80b6db0 0x7fe8c80d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8ce8c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c80cea10 0x7fe8c80cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8ce8c6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c80cf3c0 con 0x7fe8c815de20 2024-04-22T01:15:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8ce8c6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c80b75e0 con 0x7fe8c80b6db0 2024-04-22T01:15:35.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8ce8c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c80b78c0 con 0x7fe8c80cea10 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe8c815de20 0x7fe8c80b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe8c815de20 0x7fe8c80b98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42752/0 (socket says 172.21.15.70:42752) 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 -- 172.21.15.70:0/1321161696 learned_addr learned my addr 172.21.15.70:0/1321161696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8bf7fe640 1 --2- 172.21.15.70:0/1321161696 >> v2:172.21.15.98:3300/0 conn(0x7fe8c80b6db0 0x7fe8c80d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8cce63640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c80cea10 0x7fe8c80cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 -- 172.21.15.70:0/1321161696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c80cea10 msgr2=0x7fe8c80cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c80cea10 0x7fe8c80cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 -- 172.21.15.70:0/1321161696 >> v2:172.21.15.98:3300/0 conn(0x7fe8c80b6db0 msgr2=0x7fe8c80d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 --2- 172.21.15.70:0/1321161696 >> v2:172.21.15.98:3300/0 conn(0x7fe8c80b6db0 0x7fe8c80d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8c7fff640 1 -- 172.21.15.70:0/1321161696 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c80b7ba0 con 0x7fe8c815de20 2024-04-22T01:15:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.385+0000 7fe8cce63640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c80cea10 0x7fe8c80cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c7fff640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c815de20 0x7fe8c80b98a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80b77f0 tx=0x7fe8b801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b3e02673b4e0a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c77fe640 1 -- 172.21.15.70:0/1321161696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80b5070 con 0x7fe8c815de20 2024-04-22T01:15:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c77fe640 1 -- 172.21.15.70:0/1321161696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b80a89d0 con 0x7fe8c815de20 2024-04-22T01:15:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c77fe640 1 -- 172.21.15.70:0/1321161696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80a8da0 con 0x7fe8c815de20 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/1321161696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c815de20 msgr2=0x7fe8c80b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c815de20 0x7fe8c80b98a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80b77f0 tx=0x7fe8b801ed10 comp rx=0 tx=0).stop 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/1321161696 shutdown_connections 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c80cea10 0x7fe8c80cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/1321161696 >> v2:172.21.15.98:3300/0 conn(0x7fe8c80b6db0 0x7fe8c80d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/1321161696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c815de20 0x7fe8c80b98a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/1321161696 >> 172.21.15.70:0/1321161696 conn(0x7fe8c8072d20 msgr2=0x7fe8c8073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/1321161696 shutdown_connections 2024-04-22T01:15:35.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/1321161696 wait complete. 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 Processor -- start 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- start start 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8c80b6db0 0x7fe8c81f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 0x7fe8c81f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c815de20 0x7fe8c80c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c815e200 con 0x7fe8c80cea10 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c80b8160 con 0x7fe8c80b6db0 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8ce8c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c80d22c0 con 0x7fe8c815de20 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 0x7fe8c81f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 0x7fe8c81f6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42764/0 (socket says 172.21.15.70:42764) 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 -- 172.21.15.70:0/2528379907 learned_addr learned my addr 172.21.15.70:0/2528379907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 -- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c815de20 msgr2=0x7fe8c80c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c815de20 0x7fe8c80c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 -- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8c80b6db0 msgr2=0x7fe8c81f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c7fff640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8c80b6db0 0x7fe8c81f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8c80b6db0 0x7fe8c81f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 -- 172.21.15.70:0/2528379907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c815e420 con 0x7fe8c80cea10 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8bf7fe640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 0x7fe8c81f6540 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000d800 tx=0x7fe8b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0018030 con 0x7fe8c80cea10 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b00096a0 con 0x7fe8c80cea10 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.389+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0009870 con 0x7fe8c80cea10 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c815e690 con 0x7fe8c80cea10 2024-04-22T01:15:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c815ec50 con 0x7fe8c80cea10 2024-04-22T01:15:35.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c8068210 con 0x7fe8c80cea10 2024-04-22T01:15:35.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe8b001b070 con 0x7fe8c80cea10 2024-04-22T01:15:35.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8c57fa640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a80678c0 0x7fe8a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe8b00862c0 con 0x7fe8c80cea10 2024-04-22T01:15:35.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8c7fff640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a80678c0 0x7fe8a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.393+0000 7fe8c7fff640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a80678c0 0x7fe8a8069d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80b2000 tx=0x7fe8b80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.397+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe8b0056a90 con 0x7fe8c80cea10 2024-04-22T01:15:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.553+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8c8161d70 con 0x7fe8c80cea10 2024-04-22T01:15:35.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8c57fa640 1 -- 172.21.15.70:0/2528379907 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fe8b0056390 con 0x7fe8c80cea10 2024-04-22T01:15:35.557 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a80678c0 msgr2=0x7fe8a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a80678c0 0x7fe8a8069d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80b2000 tx=0x7fe8b80aa070 comp rx=0 tx=0).stop 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 msgr2=0x7fe8c81f6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 0x7fe8c81f6540 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000d800 tx=0x7fe8b000dcc0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 shutdown_connections 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe8a80678c0 0x7fe8a8069d80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8c815de20 0x7fe8c80c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c80cea10 0x7fe8c81f6540 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 --2- 172.21.15.70:0/2528379907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8c80b6db0 0x7fe8c81f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 >> 172.21.15.70:0/2528379907 conn(0x7fe8c8072d20 msgr2=0x7fe8c80b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 shutdown_connections 2024-04-22T01:15:35.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.557+0000 7fe8ce8c6640 1 -- 172.21.15.70:0/2528379907 wait complete. 2024-04-22T01:15:35.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:35.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a52fc9640 1 Processor -- start 2024-04-22T01:15:35.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a52fc9640 1 -- start start 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a52fc9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0a4c0b4450 0x7f0a4c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a52fc9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4c15c290 con 0x7f0a4c0720b0 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a52fc9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4c15c570 con 0x7f0a4c0cd220 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a52fc9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0a4c15c850 con 0x7f0a4c0b4450 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0a4c0720b0 0x7f0a4c0ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58188/0 (socket says 172.21.15.70:58188) 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4bfff640 1 -- 172.21.15.70:0/1499401837 learned_addr learned my addr 172.21.15.70:0/1499401837 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219995542 0 0) 0x7f0a4c15c290 con 0x7f0a4c0720b0 2024-04-22T01:15:35.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1499401837 >> v2:172.21.15.98:3300/0 conn(0x7f0a4c0b4450 0x7f0a4c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a34003530 con 0x7f0a4c0720b0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 597023986 0 0) 0x7f0a34003530 con 0x7f0a4c0720b0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 >> v2:172.21.15.98:3300/0 conn(0x7f0a4c0b4450 msgr2=0x7f0a4c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 --2- 172.21.15.70:0/1499401837 >> v2:172.21.15.98:3300/0 conn(0x7f0a4c0b4450 0x7f0a4c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 >> v1:172.21.15.98:6789/0 conn(0x7f0a4c0cd220 legacy=0x7f0a4c0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c15cb30 con 0x7f0a4c0720b0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f0a40001680 con 0x7f0a4c0720b0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 >> v1:172.21.15.70:6789/0 conn(0x7f0a4c0720b0 legacy=0x7f0a4c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a34003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a34004810 0x7f0a34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a340051d0 0x7f0a3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c15c850 con 0x7f0a340039c0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3400c070 con 0x7f0a340051d0 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3400c270 con 0x7f0a34004810 2024-04-22T01:15:35.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a34003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a4bfff640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a34004810 0x7f0a34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 -- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a34004810 msgr2=0x7f0a34004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a51566640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a340051d0 0x7f0a3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a34004810 0x7f0a34004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 -- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a340051d0 msgr2=0x7f0a3400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a340051d0 0x7f0a3400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a50d65640 1 -- 172.21.15.70:0/1499401837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a34003530 con 0x7f0a340039c0 2024-04-22T01:15:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.697+0000 7f0a51566640 1 -- 172.21.15.70:0/1499401837 reap_dead start 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a34003de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009ae0 tx=0x7f0a3c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd26d2984b005ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0096b0 con 0x7f0a340039c0 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3c0a6420 con 0x7f0a340039c0 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a4b7fe640 1 -- 172.21.15.70:0/1499401837 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0a6810 con 0x7f0a340039c0 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1499401837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 msgr2=0x7f0a34003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a34003de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009ae0 tx=0x7f0a3c01ca70 comp rx=0 tx=0).stop 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1499401837 shutdown_connections 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1499401837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a34003de0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1499401837 >> 172.21.15.70:0/1499401837 conn(0x7f0a4c0b1d40 msgr2=0x7f0a4c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1499401837 shutdown_connections 2024-04-22T01:15:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1499401837 wait complete. 2024-04-22T01:15:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 Processor -- start 2024-04-22T01:15:35.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.701+0000 7f0a52fc9640 1 -- start start 2024-04-22T01:15:35.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a4c13f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4c1e9500 0x7f0a4c1e9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4c1403e0 0x7f0a4c140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c0cd800 con 0x7f0a340039c0 2024-04-22T01:15:35.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c0724e0 con 0x7f0a4c1e9500 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4c0cd980 con 0x7f0a4c1403e0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a4c13f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a4c13f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42786/0 (socket says 172.21.15.70:42786) 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 -- 172.21.15.70:0/1219202022 learned_addr learned my addr 172.21.15.70:0/1219202022 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a4bfff640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4c1e9500 0x7f0a4c1e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a51566640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4c1403e0 0x7f0a4c140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 -- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4c1403e0 msgr2=0x7f0a4c140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4c1403e0 0x7f0a4c140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 -- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4c1e9500 msgr2=0x7f0a4c1e9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4c1e9500 0x7f0a4c1e9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 -- 172.21.15.70:0/1219202022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c1ea2c0 con 0x7f0a340039c0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a50d65640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a4c13f2b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009ae0 tx=0x7f0a3c019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0be070 con 0x7f0a340039c0 2024-04-22T01:15:35.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3c00f000 con 0x7f0a340039c0 2024-04-22T01:15:35.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0b9410 con 0x7f0a340039c0 2024-04-22T01:15:35.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c1ea5a0 con 0x7f0a340039c0 2024-04-22T01:15:35.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c13fec0 con 0x7f0a340039c0 2024-04-22T01:15:35.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4c1eaac0 con 0x7f0a340039c0 2024-04-22T01:15:35.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a3c0c1070 con 0x7f0a340039c0 2024-04-22T01:15:35.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a497fa640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a380678c0 0x7f0a38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0a3c11f800 con 0x7f0a340039c0 2024-04-22T01:15:35.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a4bfff640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a380678c0 0x7f0a38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.705+0000 7f0a4bfff640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a380678c0 0x7f0a38069d80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a40001ec0 tx=0x7f0a40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.709+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0a4c1eaac0 con 0x7f0a340039c0 2024-04-22T01:15:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.861+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0a4c068210 con 0x7f0a340039c0 2024-04-22T01:15:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a497fa640 1 -- 172.21.15.70:0/1219202022 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f0a3c0ac020 con 0x7f0a340039c0 2024-04-22T01:15:35.865 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a380678c0 msgr2=0x7f0a38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a380678c0 0x7f0a38069d80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a40001ec0 tx=0x7f0a40009040 comp rx=0 tx=0).stop 2024-04-22T01:15:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 msgr2=0x7f0a4c13f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a4c13f2b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009ae0 tx=0x7f0a3c019810 comp rx=0 tx=0).stop 2024-04-22T01:15:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 shutdown_connections 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a380678c0 0x7f0a38069d80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4c1403e0 0x7f0a4c140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4c1e9500 0x7f0a4c1e9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 --2- 172.21.15.70:0/1219202022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a340039c0 0x7f0a4c13f2b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 >> 172.21.15.70:0/1219202022 conn(0x7f0a4c0b1d40 msgr2=0x7f0a4c0b2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 shutdown_connections 2024-04-22T01:15:35.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.865+0000 7f0a52fc9640 1 -- 172.21.15.70:0/1219202022 wait complete. 2024-04-22T01:15:35.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:35.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.989+0000 7f9740af7640 1 Processor -- start 2024-04-22T01:15:35.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9740af7640 1 -- start start 2024-04-22T01:15:35.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9740af7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f973c15de20 0x7f973c0b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9740af7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f973c0cea10 0x7f973c0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:35.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9740af7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973c0b75e0 con 0x7f973c0b6db0 2024-04-22T01:15:35.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9740af7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f973c0b78c0 con 0x7f973c0cea10 2024-04-22T01:15:35.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9740af7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f973c0b7ba0 con 0x7f973c15de20 2024-04-22T01:15:35.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f973c0cea10 0x7f973c0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9739d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f973c0b6db0 0x7f973c0d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58200/0 (socket says 172.21.15.70:58200) 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f973c0cea10 0x7f973c0cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42792/0 (socket says 172.21.15.70:42792) 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f9739d9b640 1 -- 172.21.15.70:0/1016817199 learned_addr learned my addr 172.21.15.70:0/1016817199 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973a59c640 1 --2- 172.21.15.70:0/1016817199 >> v2:172.21.15.98:3300/0 conn(0x7f973c15de20 0x7f973c0b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 -- 172.21.15.70:0/1016817199 >> v2:172.21.15.98:3300/0 conn(0x7f973c15de20 msgr2=0x7f973c0b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 --2- 172.21.15.70:0/1016817199 >> v2:172.21.15.98:3300/0 conn(0x7f973c15de20 0x7f973c0b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 -- 172.21.15.70:0/1016817199 >> v1:172.21.15.70:6789/0 conn(0x7f973c0b6db0 legacy=0x7f973c0d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 -- 172.21.15.70:0/1016817199 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973c0b7e80 con 0x7f973c0cea10 2024-04-22T01:15:35.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973ad9d640 1 --2- 172.21.15.70:0/1016817199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c0cee60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f973c0d1ac0 tx=0x7f973401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c06f48509241207 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:35.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973959a640 1 -- 172.21.15.70:0/1016817199 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97340ab430 con 0x7f973c0cea10 2024-04-22T01:15:35.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973959a640 1 -- 172.21.15.70:0/1016817199 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97340b0070 con 0x7f973c0cea10 2024-04-22T01:15:35.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.993+0000 7f973959a640 1 -- 172.21.15.70:0/1016817199 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97340ab7f0 con 0x7f973c0cea10 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/1016817199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 msgr2=0x7f973c0cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 --2- 172.21.15.70:0/1016817199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c0cee60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f973c0d1ac0 tx=0x7f973401c450 comp rx=0 tx=0).stop 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/1016817199 shutdown_connections 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 --2- 172.21.15.70:0/1016817199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c0cee60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 --2- 172.21.15.70:0/1016817199 >> v2:172.21.15.98:3300/0 conn(0x7f973c15de20 0x7f973c0b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/1016817199 >> 172.21.15.70:0/1016817199 conn(0x7f973c072d20 msgr2=0x7f973c073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:35.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/1016817199 shutdown_connections 2024-04-22T01:15:35.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/1016817199 wait complete. 2024-04-22T01:15:35.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 Processor -- start 2024-04-22T01:15:35.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- start start 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f973c0b6db0 0x7f973c0d2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f973c15de20 0x7f973c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f973c0b7310 con 0x7f973c0cea10 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f973c0b7190 con 0x7f973c15de20 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f973c0b8e90 con 0x7f973c0b6db0 2024-04-22T01:15:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42794/0 (socket says 172.21.15.70:42794) 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 -- 172.21.15.70:0/3190773322 learned_addr learned my addr 172.21.15.70:0/3190773322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f973ad9d640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f973c15de20 0x7f973c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f973a59c640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f973c0b6db0 0x7f973c0d2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 -- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f973c0b6db0 msgr2=0x7f973c0d2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f973c0b6db0 0x7f973c0d2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 -- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f973c15de20 msgr2=0x7f973c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f973c15de20 0x7f973c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 -- 172.21.15.70:0/3190773322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973c0d2b60 con 0x7f973c0cea10 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f973a59c640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f973c0b6db0 0x7f973c0d2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9739d9b640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c156c80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f973000cb10 tx=0x7f973000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730017070 con 0x7f973c0cea10 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f973000ac70 con 0x7f973c0cea10 2024-04-22T01:15:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973000ae60 con 0x7f973c0cea10 2024-04-22T01:15:36.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973c0d2e40 con 0x7f973c0cea10 2024-04-22T01:15:36.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:35.997+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f973c0d3610 con 0x7f973c0cea10 2024-04-22T01:15:36.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.001+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f973c068210 con 0x7f973c0cea10 2024-04-22T01:15:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.001+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f973001a070 con 0x7f973c0cea10 2024-04-22T01:15:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.001+0000 7f972affd640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f971c0678c0 0x7f971c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.001+0000 7f973a59c640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f971c0678c0 0x7f971c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.001+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9730085e00 con 0x7f973c0cea10 2024-04-22T01:15:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.001+0000 7f973a59c640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f971c0678c0 0x7f971c069d80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f973c0ca3b0 tx=0x7f972c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.005+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9730056540 con 0x7f973c0cea10 2024-04-22T01:15:36.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.161+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f973c06b9b0 con 0x7f973c0cea10 2024-04-22T01:15:36.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f972affd640 1 -- 172.21.15.70:0/3190773322 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f9730055e40 con 0x7f973c0cea10 2024-04-22T01:15:36.165 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:36.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f971c0678c0 msgr2=0x7f971c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f971c0678c0 0x7f971c069d80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f973c0ca3b0 tx=0x7f972c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:36.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 msgr2=0x7f973c156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c156c80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f973000cb10 tx=0x7f973000a060 comp rx=0 tx=0).stop 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 shutdown_connections 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f971c0678c0 0x7f971c069d80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f973c15de20 0x7f973c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f973c0cea10 0x7f973c156c80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 --2- 172.21.15.70:0/3190773322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f973c0b6db0 0x7f973c0d2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 >> 172.21.15.70:0/3190773322 conn(0x7f973c072d20 msgr2=0x7f973c15a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 shutdown_connections 2024-04-22T01:15:36.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.165+0000 7f9740af7640 1 -- 172.21.15.70:0/3190773322 wait complete. 2024-04-22T01:15:36.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:36.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.301+0000 7f88ae8c8640 1 Processor -- start 2024-04-22T01:15:36.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.301+0000 7f88ae8c8640 1 -- start start 2024-04-22T01:15:36.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ae8c8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 0x7f88a815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ae8c8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88a80ce920 0x7f88a8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ae8c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a815f000 con 0x7f88a80ceda0 2024-04-22T01:15:36.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ae8c8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f88a80b3800 con 0x7f88a80ce920 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ae8c8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88a80b3ae0 con 0x7f88a815e6e0 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f88a80ceda0 0x7f88a806d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59302/0 (socket says 172.21.15.70:59302) 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 -- 172.21.15.70:0/2717714874 learned_addr learned my addr 172.21.15.70:0/2717714874 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a77fe640 1 --2- 172.21.15.70:0/2717714874 >> v2:172.21.15.98:3300/0 conn(0x7f88a80ce920 0x7f88a8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a7fff640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 0x7f88a815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900016368 0 0) 0x7f88a815f000 con 0x7f88a80ceda0 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8890003530 con 0x7f88a80ceda0 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2416426005 0 0) 0x7f8890003530 con 0x7f88a80ceda0 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 msgr2=0x7f88a815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 0x7f88a815eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 >> v2:172.21.15.98:3300/0 conn(0x7f88a80ce920 msgr2=0x7f88a8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 --2- 172.21.15.70:0/2717714874 >> v2:172.21.15.98:3300/0 conn(0x7f88a80ce920 0x7f88a8154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a80cde40 con 0x7f88a80ceda0 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f88a0001a00 con 0x7f88a80ceda0 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 >> v1:172.21.15.70:6789/0 conn(0x7f88a80ceda0 legacy=0x7f88a806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 0x7f88900039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8890003f40 0x7f8890004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88900051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88a80b3800 con 0x7f8890004d50 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88a80b3ae0 con 0x7f8890003f40 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f889000b9d0 con 0x7f88a815e6e0 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88900051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a77fe640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8890003f40 0x7f8890004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a7fff640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 0x7f88900039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 -- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 msgr2=0x7f88900039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a815e6e0 0x7f88900039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 -- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8890003f40 msgr2=0x7f8890004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8890003f40 0x7f8890004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 -- 172.21.15.70:0/2717714874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8890003530 con 0x7f8890004d50 2024-04-22T01:15:36.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88a77fe640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8890003f40 0x7f8890004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:36.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.305+0000 7f88ace65640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88900051d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f88a000a880 tx=0x7f88a001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e56679c9c00c229 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a000a4a0 con 0x7f8890004d50 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a00ad630 con 0x7f8890004d50 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a6ffd640 1 -- 172.21.15.70:0/2717714874 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a00ada20 con 0x7f8890004d50 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/2717714874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 msgr2=0x7f88900051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/2717714874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88900051d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f88a000a880 tx=0x7f88a001d340 comp rx=0 tx=0).stop 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ace65640 1 -- 172.21.15.70:0/2717714874 reap_dead start 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/2717714874 shutdown_connections 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/2717714874 >> 172.21.15.70:0/2717714874 conn(0x7f88a80b1d40 msgr2=0x7f88a80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/2717714874 shutdown_connections 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/2717714874 wait complete. 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 Processor -- start 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- start start 2024-04-22T01:15:36.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88a80cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a80ac8d0 0x7f88a814d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a814f1a0 0x7f88a814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88a806f0e0 con 0x7f8890004d50 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88a80ce400 con 0x7f88a814f1a0 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88a806f260 con 0x7f88a80ac8d0 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88a80cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88a80cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39778/0 (socket says 172.21.15.70:39778) 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 -- 172.21.15.70:0/1580952046 learned_addr learned my addr 172.21.15.70:0/1580952046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ace65640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a814f1a0 0x7f88a814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a77fe640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a80ac8d0 0x7f88a814d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 -- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a80ac8d0 msgr2=0x7f88a814d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a80ac8d0 0x7f88a814d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 -- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a814f1a0 msgr2=0x7f88a814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a814f1a0 0x7f88a814f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 -- 172.21.15.70:0/1580952046 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a80ad0e0 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a7fff640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88a80cbee0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8898011620 tx=0x7f8898011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889800f920 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889800fac0 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a80ad3c0 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898015460 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.309+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a80ad6a0 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.313+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a8068210 con 0x7f8890004d50 2024-04-22T01:15:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.313+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8898015600 con 0x7f8890004d50 2024-04-22T01:15:36.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.313+0000 7f88a4ff9640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88940678c0 0x7f8894069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.313+0000 7f88a77fe640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88940678c0 0x7f8894069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.313+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8898086a40 con 0x7f8890004d50 2024-04-22T01:15:36.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.313+0000 7f88a77fe640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88940678c0 0x7f8894069d80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f88a8073320 tx=0x7f889c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.317+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8898019030 con 0x7f8890004d50 2024-04-22T01:15:36.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.477+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f88a80b3f30 con 0x7f8890004d50 2024-04-22T01:15:36.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.477+0000 7f88a4ff9640 1 -- 172.21.15.70:0/1580952046 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f8898057210 con 0x7f8890004d50 2024-04-22T01:15:36.479 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88940678c0 msgr2=0x7f8894069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88940678c0 0x7f8894069d80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f88a8073320 tx=0x7f889c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 msgr2=0x7f88a80cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88a80cbee0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8898011620 tx=0x7f8898011ae0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 shutdown_connections 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88940678c0 0x7f8894069d80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a814f1a0 0x7f88a814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88a80ac8d0 0x7f88a814d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 --2- 172.21.15.70:0/1580952046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8890004d50 0x7f88a80cbee0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 >> 172.21.15.70:0/1580952046 conn(0x7f88a80b1d40 msgr2=0x7f88a80d0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 shutdown_connections 2024-04-22T01:15:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.481+0000 7f88ae8c8640 1 -- 172.21.15.70:0/1580952046 wait complete. 2024-04-22T01:15:36.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:36.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.601+0000 7f35760ad640 1 Processor -- start 2024-04-22T01:15:36.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.601+0000 7f35760ad640 1 -- start start 2024-04-22T01:15:36.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f35760ad640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f3570166b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f35760ad640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35700cd7b0 con 0x7f35700cc150 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f35760ad640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35700cc710 con 0x7f35701455b0 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f35760ad640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f35700cc9f0 con 0x7f3570072230 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356effd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f35700cc150 0x7f3570071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59318/0 (socket says 172.21.15.70:59318) 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356effd640 1 -- 172.21.15.70:0/897360973 learned_addr learned my addr 172.21.15.70:0/897360973 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3240715283 0 0) 0x7f35700cd7b0 con 0x7f35700cc150 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f355c003530 con 0x7f35700cc150 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356ffff640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f3570166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1789198609 0 0) 0x7f355c003530 con 0x7f35700cc150 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 msgr2=0x7f3570166b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f3570166b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 >> v1:172.21.15.98:6789/0 conn(0x7f35701455b0 legacy=0x7f35700cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35700cccd0 con 0x7f35700cc150 2024-04-22T01:15:36.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3558001a00 con 0x7f35700cc150 2024-04-22T01:15:36.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 >> v1:172.21.15.70:6789/0 conn(0x7f35700cc150 legacy=0x7f3570071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f355c0039f0 0x7f355c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f355c004810 0x7f355c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f355c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35700cc9f0 con 0x7f355c0039f0 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f355c005580 con 0x7f355c004810 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f355c00bf10 con 0x7f3570072230 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f355c0039f0 0x7f355c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356effd640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f355c004810 0x7f355c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 -- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 msgr2=0x7f355c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356ffff640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f355c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f355c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 -- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f355c004810 msgr2=0x7f355c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f355c004810 0x7f355c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 -- 172.21.15.70:0/897360973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c003530 con 0x7f355c0039f0 2024-04-22T01:15:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356effd640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f355c004810 0x7f355c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356ffff640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570072230 0x7f355c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.605+0000 7f356f7fe640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f355c0039f0 0x7f355c003e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3560001780 tx=0x7f356001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b843144f99fe28 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35600013a0 con 0x7f355c0039f0 2024-04-22T01:15:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35600052f0 con 0x7f355c0039f0 2024-04-22T01:15:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356e7fc640 1 -- 172.21.15.70:0/897360973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560003840 con 0x7f355c0039f0 2024-04-22T01:15:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/897360973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f355c0039f0 msgr2=0x7f355c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 --2- 172.21.15.70:0/897360973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f355c0039f0 0x7f355c003e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3560001780 tx=0x7f356001c420 comp rx=0 tx=0).stop 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356ffff640 1 -- 172.21.15.70:0/897360973 reap_dead start 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/897360973 shutdown_connections 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/897360973 >> 172.21.15.70:0/897360973 conn(0x7f357006b1f0 msgr2=0x7f357006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/897360973 shutdown_connections 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/897360973 wait complete. 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 Processor -- start 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- start start 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 0x7f35700b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3570141a70 0x7f3570163d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570161800 0x7f357015b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35701676d0 con 0x7f3570161420 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3570167550 con 0x7f3570141a70 2024-04-22T01:15:36.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3570167850 con 0x7f3570161800 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 0x7f35700b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 0x7f35700b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39792/0 (socket says 172.21.15.70:39792) 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 -- 172.21.15.70:0/3425841120 learned_addr learned my addr 172.21.15.70:0/3425841120 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356effd640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3570141a70 0x7f3570163d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356ffff640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570161800 0x7f357015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 -- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570161800 msgr2=0x7f357015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570161800 0x7f357015b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 -- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3570141a70 msgr2=0x7f3570163d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3570141a70 0x7f3570163d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 -- 172.21.15.70:0/3425841120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35700aca70 con 0x7f3570161420 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356effd640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3570141a70 0x7f3570163d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356ffff640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570161800 0x7f357015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f356f7fe640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 0x7f35700b2c50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3560000c10 tx=0x7f355c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560007a40 con 0x7f3570161420 2024-04-22T01:15:36.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35600ba070 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35701646c0 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560000ee0 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.609+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35700acff0 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.613+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3570068210 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.613+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3560005270 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.613+0000 7f3567fff640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3550067a60 0x7f3550069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.613+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f356011ffd0 con 0x7f3570161420 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.613+0000 7f356effd640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3550067a60 0x7f3550069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.613+0000 7f356effd640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3550067a60 0x7f3550069f20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f35701fa4e0 tx=0x7f3558009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.617+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3560016130 con 0x7f3570161420 2024-04-22T01:15:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.765+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f357015f510 con 0x7f3570161420 2024-04-22T01:15:36.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f3567fff640 1 -- 172.21.15.70:0/3425841120 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f35600b33b0 con 0x7f3570161420 2024-04-22T01:15:36.769 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:36.769 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:36.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3550067a60 msgr2=0x7f3550069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3550067a60 0x7f3550069f20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f35701fa4e0 tx=0x7f3558009040 comp rx=0 tx=0).stop 2024-04-22T01:15:36.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 msgr2=0x7f35700b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 0x7f35700b2c50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3560000c10 tx=0x7f355c005840 comp rx=0 tx=0).stop 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 shutdown_connections 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3550067a60 0x7f3550069f20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3570161800 0x7f357015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3570141a70 0x7f3570163d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 --2- 172.21.15.70:0/3425841120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570161420 0x7f35700b2c50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 >> 172.21.15.70:0/3425841120 conn(0x7f357006b1f0 msgr2=0x7f357015cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 shutdown_connections 2024-04-22T01:15:36.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.769+0000 7f35760ad640 1 -- 172.21.15.70:0/3425841120 wait complete. 2024-04-22T01:15:36.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:36.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:36.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.901+0000 7fd68ae4b640 1 Processor -- start 2024-04-22T01:15:36.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.901+0000 7fd68ae4b640 1 -- start start 2024-04-22T01:15:36.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd68ae4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6840cd280 0x7fd6840cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd68ae4b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6840c5720 con 0x7fd6840b43e0 2024-04-22T01:15:36.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd68ae4b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6840cd710 con 0x7fd68415f3e0 2024-04-22T01:15:36.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd68ae4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6840cd9f0 con 0x7fd6840cd280 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6893e8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd6840b43e0 0x7fd6840cfce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59326/0 (socket says 172.21.15.70:59326) 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6893e8640 1 -- 172.21.15.70:0/687856780 learned_addr learned my addr 172.21.15.70:0/687856780 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6840cd280 0x7fd6840cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383908968 0 0) 0x7fd6840c5720 con 0x7fd6840b43e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd66c003530 con 0x7fd6840b43e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328190889 0 0) 0x7fd6840cd710 con 0x7fd68415f3e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6840c5720 con 0x7fd68415f3e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 210311745 0 0) 0x7fd66c003530 con 0x7fd6840b43e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6840cd280 msgr2=0x7fd6840cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6840cd280 0x7fd6840cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 >> v1:172.21.15.98:6789/0 conn(0x7fd68415f3e0 legacy=0x7fd684099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6840cdcd0 con 0x7fd6840b43e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fd67c0016a0 con 0x7fd6840b43e0 2024-04-22T01:15:36.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 >> v1:172.21.15.70:6789/0 conn(0x7fd6840b43e0 legacy=0x7fd6840cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66c003b80 0x7fd66c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66c004810 0x7fd66c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c0051d0 0x7fd66c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6840cd9f0 con 0x7fd66c004810 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c00bfc0 con 0x7fd66c0051d0 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd66c00c1c0 con 0x7fd66c003b80 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6840cd280 0x7fd6840cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66c004810 0x7fd66c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd6893e8640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c0051d0 0x7fd66c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd688be7640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66c003b80 0x7fd66c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 -- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66c003b80 msgr2=0x7fd66c003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66c003b80 0x7fd66c003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 -- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c0051d0 msgr2=0x7fd66c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:36.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c0051d0 0x7fd66c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.905+0000 7fd683fff640 1 -- 172.21.15.70:0/687856780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66c003530 con 0x7fd66c004810 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd683fff640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66c004810 0x7fd66c004c90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd6840b4300 tx=0x7fd67801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b736ffc71acb8da5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6780097f0 con 0x7fd66c004810 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd678004030 con 0x7fd66c004810 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6837fe640 1 -- 172.21.15.70:0/687856780 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678004400 con 0x7fd66c004810 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- 172.21.15.70:0/687856780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66c004810 msgr2=0x7fd66c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/687856780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66c004810 0x7fd66c004c90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd6840b4300 tx=0x7fd67801cab0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6893e8640 1 -- 172.21.15.70:0/687856780 reap_dead start 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- 172.21.15.70:0/687856780 shutdown_connections 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- 172.21.15.70:0/687856780 >> 172.21.15.70:0/687856780 conn(0x7fd6840b36c0 msgr2=0x7fd6840b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- 172.21.15.70:0/687856780 shutdown_connections 2024-04-22T01:15:36.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- 172.21.15.70:0/687856780 wait complete. 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 Processor -- start 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- start start 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c004810 0x7fd68414cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd684157510 0x7fd6840b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 0x7fd6840cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd68409a440 con 0x7fd684157990 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd68415f7c0 con 0x7fd66c004810 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd68ae4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd68409a5c0 con 0x7fd684157510 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6893e8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 0x7fd6840cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6893e8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 0x7fd6840cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39808/0 (socket says 172.21.15.70:39808) 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd6893e8640 1 -- 172.21.15.70:0/210263305 learned_addr learned my addr 172.21.15.70:0/210263305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd688be7640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c004810 0x7fd68414cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.909+0000 7fd683fff640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd684157510 0x7fd6840b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6893e8640 1 -- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd684157510 msgr2=0x7fd6840b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6893e8640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd684157510 0x7fd6840b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6893e8640 1 -- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c004810 msgr2=0x7fd68414cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6893e8640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c004810 0x7fd68414cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6893e8640 1 -- 172.21.15.70:0/210263305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6840c8ca0 con 0x7fd684157990 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6893e8640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 0x7fd6840cb290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd67c001060 tx=0x7fd67c00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c019070 con 0x7fd684157990 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67c010510 con 0x7fd684157990 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6840c8f40 con 0x7fd684157990 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c010900 con 0x7fd684157990 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6840c9500 con 0x7fd684157990 2024-04-22T01:15:36.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd684068920 con 0x7fd684157990 2024-04-22T01:15:36.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd67c01c070 con 0x7fd684157990 2024-04-22T01:15:36.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6817fa640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6700678c0 0x7fd670069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:36.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd67c085470 con 0x7fd684157990 2024-04-22T01:15:36.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd688be7640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6700678c0 0x7fd670069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:36.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.913+0000 7fd688be7640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6700678c0 0x7fd670069d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6740059b0 tx=0x7fd674009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:36.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:36.917+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd67c055bb0 con 0x7fd684157990 2024-04-22T01:15:37.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd684070180 con 0x7fd684157990 2024-04-22T01:15:37.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd6817fa640 1 -- 172.21.15.70:0/210263305 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fd67c0554b0 con 0x7fd684157990 2024-04-22T01:15:37.078 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6700678c0 msgr2=0x7fd670069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6700678c0 0x7fd670069d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6740059b0 tx=0x7fd674009040 comp rx=0 tx=0).stop 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 msgr2=0x7fd6840cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 0x7fd6840cb290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd67c001060 tx=0x7fd67c00e9e0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 shutdown_connections 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6700678c0 0x7fd670069d80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd684157990 0x7fd6840cb290 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd684157510 0x7fd6840b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 --2- 172.21.15.70:0/210263305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c004810 0x7fd68414cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 >> 172.21.15.70:0/210263305 conn(0x7fd6840b36c0 msgr2=0x7fd6840c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:37.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 shutdown_connections 2024-04-22T01:15:37.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.077+0000 7fd68ae4b640 1 -- 172.21.15.70:0/210263305 wait complete. 2024-04-22T01:15:37.097 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:37.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8c1b2b640 1 Processor -- start 2024-04-22T01:15:37.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8c1b2b640 1 -- start start 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8c1b2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 0x7fc8bc154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8c1b2b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8bc0b3800 con 0x7fc8bc0ceda0 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8c1b2b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8bc0b3ae0 con 0x7fc8bc15e6e0 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8c1b2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc8bc0cde40 con 0x7fc8bc0ce920 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8bbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc8bc0ceda0 0x7fc8bc06d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59340/0 (socket says 172.21.15.70:59340) 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8bbfff640 1 -- 172.21.15.70:0/151056344 learned_addr learned my addr 172.21.15.70:0/151056344 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8baffd640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 0x7fc8bc154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132052395 0 0) 0x7fc8bc0b3800 con 0x7fc8bc0ceda0 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8a4003530 con 0x7fc8bc0ceda0 2024-04-22T01:15:37.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.205+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 944851993 0 0) 0x7fc8a4003530 con 0x7fc8bc0ceda0 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 msgr2=0x7fc8bc154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 0x7fc8bc154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 >> v1:172.21.15.98:6789/0 conn(0x7fc8bc15e6e0 legacy=0x7fc8bc15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc0ce120 con 0x7fc8bc0ceda0 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fc8b4001680 con 0x7fc8bc0ceda0 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 >> v1:172.21.15.70:6789/0 conn(0x7fc8bc0ceda0 legacy=0x7fc8bc06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8a40039f0 0x7fc8a4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a4004810 0x7fc8a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 0x7fc8a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc0cde40 con 0x7fc8a40039f0 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a400bab0 con 0x7fc8a4004810 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a400bcb0 con 0x7fc8bc0ce920 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8a40039f0 0x7fc8a4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8baffd640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a4004810 0x7fc8a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 -- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 msgr2=0x7fc8a40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:37.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc0ce920 0x7fc8a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 -- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a4004810 msgr2=0x7fc8a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:37.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a4004810 0x7fc8a4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 -- 172.21.15.70:0/151056344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a4003530 con 0x7fc8a40039f0 2024-04-22T01:15:37.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8a40039f0 0x7fc8a4003e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac001e40 tx=0x7fc8ac01c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef9a529b3ae9d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac001a90 con 0x7fc8a40039f0 2024-04-22T01:15:37.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ac003b80 con 0x7fc8a40039f0 2024-04-22T01:15:37.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8ba7fc640 1 -- 172.21.15.70:0/151056344 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac005080 con 0x7fc8a40039f0 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/151056344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8a40039f0 msgr2=0x7fc8a4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/151056344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8a40039f0 0x7fc8a4003e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac001e40 tx=0x7fc8ac01c710 comp rx=0 tx=0).stop 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8bbfff640 1 -- 172.21.15.70:0/151056344 reap_dead start 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/151056344 shutdown_connections 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/151056344 >> 172.21.15.70:0/151056344 conn(0x7fc8bc0b1d40 msgr2=0x7fc8bc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/151056344 shutdown_connections 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.209+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/151056344 wait complete. 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 Processor -- start 2024-04-22T01:15:37.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- start start 2024-04-22T01:15:37.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a40039f0 0x7fc8bc1557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc142460 0x7fc8bc141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 0x7fc8bc157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc0b2e80 con 0x7fc8bc157ab0 2024-04-22T01:15:37.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc0b2d00 con 0x7fc8a40039f0 2024-04-22T01:15:37.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc8bc0b3000 con 0x7fc8bc142460 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 0x7fc8bc157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 0x7fc8bc157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39826/0 (socket says 172.21.15.70:39826) 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 -- 172.21.15.70:0/1931123091 learned_addr learned my addr 172.21.15.70:0/1931123091 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a40039f0 0x7fc8bc1557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 -- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc142460 msgr2=0x7fc8bc141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc142460 0x7fc8bc141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 -- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a40039f0 msgr2=0x7fc8bc1557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a40039f0 0x7fc8bc1557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 -- 172.21.15.70:0/1931123091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc149880 con 0x7fc8bc157ab0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8bbfff640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 0x7fc8bc157f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc072e20 tx=0x7fc8b400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4007230 con 0x7fc8bc157ab0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b40073d0 con 0x7fc8bc157ab0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8bc155eb0 con 0x7fc8bc157ab0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b40047f0 con 0x7fc8bc157ab0 2024-04-22T01:15:37.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8bc156470 con 0x7fc8bc157ab0 2024-04-22T01:15:37.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8bc156640 con 0x7fc8bc157ab0 2024-04-22T01:15:37.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.213+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8b40049b0 con 0x7fc8bc157ab0 2024-04-22T01:15:37.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.217+0000 7fc8a3fff640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8a80678c0 0x7fc8a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.217+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc8b40855f0 con 0x7fc8bc157ab0 2024-04-22T01:15:37.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.217+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8a80678c0 0x7fc8a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.217+0000 7fc8bb7fe640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8a80678c0 0x7fc8a8069d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac005000 tx=0x7fc8ac001520 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.221+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc8b4055d30 con 0x7fc8bc157ab0 2024-04-22T01:15:37.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.365+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc8bc068210 con 0x7fc8bc157ab0 2024-04-22T01:15:37.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.365+0000 7fc8a3fff640 1 -- 172.21.15.70:0/1931123091 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fc8b4055630 con 0x7fc8bc157ab0 2024-04-22T01:15:37.367 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:37.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8a80678c0 msgr2=0x7fc8a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8a80678c0 0x7fc8a8069d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac005000 tx=0x7fc8ac001520 comp rx=0 tx=0).stop 2024-04-22T01:15:37.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 msgr2=0x7fc8bc157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 0x7fc8bc157f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc072e20 tx=0x7fc8b400cb90 comp rx=0 tx=0).stop 2024-04-22T01:15:37.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 shutdown_connections 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8a80678c0 0x7fc8a8069d80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8bc157ab0 0x7fc8bc157f60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8bc142460 0x7fc8bc141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 --2- 172.21.15.70:0/1931123091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8a40039f0 0x7fc8bc1557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 >> 172.21.15.70:0/1931123091 conn(0x7fc8bc0b1d40 msgr2=0x7fc8bc06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 shutdown_connections 2024-04-22T01:15:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.369+0000 7fc8c1b2b640 1 -- 172.21.15.70:0/1931123091 wait complete. 2024-04-22T01:15:37.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:37.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.505+0000 7f20ae199640 1 Processor -- start 2024-04-22T01:15:37.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.505+0000 7f20ae199640 1 -- start start 2024-04-22T01:15:37.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20a80af900 0x7f20a80d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20a80d0e50 0x7f20a8164ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20a80b0000 con 0x7f20a815ad00 2024-04-22T01:15:37.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f20a806fcb0 con 0x7f20a80d0e50 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f20a806ff90 con 0x7f20a80af900 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20a80d0e50 0x7f20a8164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20a80d0e50 0x7f20a8164ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39834/0 (socket says 172.21.15.70:39834) 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f20a815ad00 0x7f20a806f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59344/0 (socket says 172.21.15.70:59344) 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 -- 172.21.15.70:0/2135283369 learned_addr learned my addr 172.21.15.70:0/2135283369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a6ffd640 1 --2- 172.21.15.70:0/2135283369 >> v2:172.21.15.98:3300/0 conn(0x7f20a80af900 0x7f20a80d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 -- 172.21.15.70:0/2135283369 >> v2:172.21.15.98:3300/0 conn(0x7f20a80af900 msgr2=0x7f20a80d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 --2- 172.21.15.70:0/2135283369 >> v2:172.21.15.98:3300/0 conn(0x7f20a80af900 0x7f20a80d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 -- 172.21.15.70:0/2135283369 >> v1:172.21.15.70:6789/0 conn(0x7f20a815ad00 legacy=0x7f20a806f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a67fc640 1 -- 172.21.15.70:0/2135283369 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719083872 0 0) 0x7f20a80b0000 con 0x7f20a815ad00 2024-04-22T01:15:37.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 -- 172.21.15.70:0/2135283369 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a8070270 con 0x7f20a80d0e50 2024-04-22T01:15:37.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a7fff640 1 --2- 172.21.15.70:0/2135283369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80d0e50 0x7f20a8164ba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f20a80b3620 tx=0x7f20a001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c93d4ae48adcd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a67fc640 1 -- 172.21.15.70:0/2135283369 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a00a5880 con 0x7f20a80d0e50 2024-04-22T01:15:37.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a67fc640 1 -- 172.21.15.70:0/2135283369 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a00a5a20 con 0x7f20a80d0e50 2024-04-22T01:15:37.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20a67fc640 1 -- 172.21.15.70:0/2135283369 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a00a5df0 con 0x7f20a80d0e50 2024-04-22T01:15:37.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- 172.21.15.70:0/2135283369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80d0e50 msgr2=0x7f20a8164ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 --2- 172.21.15.70:0/2135283369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80d0e50 0x7f20a8164ba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f20a80b3620 tx=0x7f20a001c9c0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- 172.21.15.70:0/2135283369 shutdown_connections 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 --2- 172.21.15.70:0/2135283369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80d0e50 0x7f20a8164ba0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 --2- 172.21.15.70:0/2135283369 >> v2:172.21.15.98:3300/0 conn(0x7f20a80af900 0x7f20a80d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- 172.21.15.70:0/2135283369 >> 172.21.15.70:0/2135283369 conn(0x7f20a8099a60 msgr2=0x7f20a8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- 172.21.15.70:0/2135283369 shutdown_connections 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.509+0000 7f20ae199640 1 -- 172.21.15.70:0/2135283369 wait complete. 2024-04-22T01:15:37.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 Processor -- start 2024-04-22T01:15:37.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- start start 2024-04-22T01:15:37.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 0x7f20a80b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a80d0e50 0x7f20a815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a815ad00 0x7f20a81f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20a806f850 con 0x7f20a80af900 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f20a815b0e0 con 0x7f20a80d0e50 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f20a806f9d0 con 0x7f20a815ad00 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 0x7f20a80b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 0x7f20a80b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39842/0 (socket says 172.21.15.70:39842) 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 -- 172.21.15.70:0/2138846090 learned_addr learned my addr 172.21.15.70:0/2138846090 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a6ffd640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a80d0e50 0x7f20a815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a7fff640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a815ad00 0x7f20a81f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 -- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a815ad00 msgr2=0x7f20a81f5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a815ad00 0x7f20a81f5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 -- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a80d0e50 msgr2=0x7f20a815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a80d0e50 0x7f20a815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 -- 172.21.15.70:0/2138846090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a80b4a80 con 0x7f20a80af900 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20a77fe640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 0x7f20a80b5b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f20980011f0 tx=0x7f209800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098018070 con 0x7f20a80af900 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20980074e0 con 0x7f20a80af900 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a80b4d60 con 0x7f20a80af900 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098004a10 con 0x7f20a80af900 2024-04-22T01:15:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a80b5320 con 0x7f20a80af900 2024-04-22T01:15:37.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.513+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a8068920 con 0x7f20a80af900 2024-04-22T01:15:37.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.517+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f209801b070 con 0x7f20a80af900 2024-04-22T01:15:37.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.517+0000 7f208ffff640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20940678c0 0x7f2094069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.517+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f20980854b0 con 0x7f20a80af900 2024-04-22T01:15:37.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.517+0000 7f20a6ffd640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20940678c0 0x7f2094069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.517+0000 7f20a6ffd640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20940678c0 0x7f2094069d80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f20a80c3f50 tx=0x7f209c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.521+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2098055bf0 con 0x7f20a80af900 2024-04-22T01:15:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.677+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f20a80b6400 con 0x7f20a80af900 2024-04-22T01:15:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.677+0000 7f208ffff640 1 -- 172.21.15.70:0/2138846090 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f20980554f0 con 0x7f20a80af900 2024-04-22T01:15:37.680 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20940678c0 msgr2=0x7f2094069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20940678c0 0x7f2094069d80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f20a80c3f50 tx=0x7f209c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 msgr2=0x7f20a80b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 0x7f20a80b5b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f20980011f0 tx=0x7f209800c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 shutdown_connections 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20940678c0 0x7f2094069d80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a815ad00 0x7f20a81f5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a80d0e50 0x7f20a815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 --2- 172.21.15.70:0/2138846090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20a80af900 0x7f20a80b5b20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 >> 172.21.15.70:0/2138846090 conn(0x7f20a8099a60 msgr2=0x7f20a80cb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 shutdown_connections 2024-04-22T01:15:37.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.681+0000 7f20ae199640 1 -- 172.21.15.70:0/2138846090 wait complete. 2024-04-22T01:15:37.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:37.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 Processor -- start 2024-04-22T01:15:37.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 -- start start 2024-04-22T01:15:37.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa7815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faa780ce920 0x7faa78154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7815f000 con 0x7faa780ceda0 2024-04-22T01:15:37.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7faa780b3800 con 0x7faa780ce920 2024-04-22T01:15:37.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7fecf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faa780b3ae0 con 0x7faa7815e6e0 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faa780ce920 0x7faa78154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7e46c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7faa780ceda0 0x7faa7806d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59350/0 (socket says 172.21.15.70:59350) 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 --2- 172.21.15.70:0/3714036602 >> v2:172.21.15.70:3300/0 conn(0x7faa780ce920 0x7faa78154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39856/0 (socket says 172.21.15.70:39856) 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7e46c640 1 -- 172.21.15.70:0/3714036602 learned_addr learned my addr 172.21.15.70:0/3714036602 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7dc6b640 1 --2- 172.21.15.70:0/3714036602 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa7815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 -- 172.21.15.70:0/3714036602 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 msgr2=0x7faa7815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 --2- 172.21.15.70:0/3714036602 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa7815eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 -- 172.21.15.70:0/3714036602 >> v1:172.21.15.70:6789/0 conn(0x7faa780ceda0 legacy=0x7faa7806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 -- 172.21.15.70:0/3714036602 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa780cde40 con 0x7faa780ce920 2024-04-22T01:15:37.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7cc69640 1 -- 172.21.15.70:0/3714036602 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525064882 0 0) 0x7faa7815f000 con 0x7faa780ceda0 2024-04-22T01:15:37.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.821+0000 7faa7d46a640 1 --2- 172.21.15.70:0/3714036602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa78154c60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa680097c0 tx=0x7faa6801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2e22709ef6294d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7cc69640 1 -- 172.21.15.70:0/3714036602 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680a7070 con 0x7faa780ce920 2024-04-22T01:15:37.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7cc69640 1 -- 172.21.15.70:0/3714036602 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa680b0070 con 0x7faa780ce920 2024-04-22T01:15:37.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7cc69640 1 -- 172.21.15.70:0/3714036602 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680ab890 con 0x7faa780ce920 2024-04-22T01:15:37.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3714036602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 msgr2=0x7faa78154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3714036602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa78154c60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa680097c0 tx=0x7faa6801c470 comp rx=0 tx=0).stop 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3714036602 shutdown_connections 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3714036602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa78154c60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3714036602 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa7815eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3714036602 >> 172.21.15.70:0/3714036602 conn(0x7faa780b1d40 msgr2=0x7faa780cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3714036602 shutdown_connections 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3714036602 wait complete. 2024-04-22T01:15:37.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 Processor -- start 2024-04-22T01:15:37.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- start start 2024-04-22T01:15:37.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa780ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa780ceda0 0x7faa78151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa78156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa780b32c0 con 0x7faa780ce920 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa781555b0 con 0x7faa780ceda0 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faa780b3440 con 0x7faa7815e6e0 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa780ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa780ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39860/0 (socket says 172.21.15.70:39860) 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 -- 172.21.15.70:0/3693418219 learned_addr learned my addr 172.21.15.70:0/3693418219 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 -- 172.21.15.70:0/3693418219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 msgr2=0x7faa78156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa78156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 -- 172.21.15.70:0/3693418219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa780ceda0 msgr2=0x7faa78151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa780ceda0 0x7faa78151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 -- 172.21.15.70:0/3693418219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7815c380 con 0x7faa780ce920 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7dc6b640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa780ae880 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7faa7400d800 tx=0x7faa7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7400b6c0 con 0x7faa780ce920 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa74019070 con 0x7faa780ce920 2024-04-22T01:15:37.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa780af0d0 con 0x7faa780ce920 2024-04-22T01:15:37.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7400ba60 con 0x7faa780ce920 2024-04-22T01:15:37.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.825+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7815c900 con 0x7faa780ce920 2024-04-22T01:15:37.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.829+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa7815c190 con 0x7faa780ce920 2024-04-22T01:15:37.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.829+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa7400bc00 con 0x7faa780ce920 2024-04-22T01:15:37.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.829+0000 7faa6e7fc640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa5c0678c0 0x7faa5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:37.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.829+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7faa74086550 con 0x7faa780ce920 2024-04-22T01:15:37.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.829+0000 7faa7d46a640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa5c0678c0 0x7faa5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:37.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.829+0000 7faa7d46a640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa5c0678c0 0x7faa5c069d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa680af000 tx=0x7faa68004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:37.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:37.833+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7faa74056c90 con 0x7faa780ce920 2024-04-22T01:15:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faa780af3b0 con 0x7faa780ce920 2024-04-22T01:15:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa6e7fc640 1 -- 172.21.15.70:0/3693418219 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7faa74056590 con 0x7faa780ce920 2024-04-22T01:15:38.003 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa5c0678c0 msgr2=0x7faa5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa5c0678c0 0x7faa5c069d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa680af000 tx=0x7faa68004740 comp rx=0 tx=0).stop 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 msgr2=0x7faa780ae880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa780ae880 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7faa7400d800 tx=0x7faa7400dcc0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 shutdown_connections 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa5c0678c0 0x7faa5c069d80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa7815e6e0 0x7faa78156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa780ceda0 0x7faa78151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 --2- 172.21.15.70:0/3693418219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa780ce920 0x7faa780ae880 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.001+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 >> 172.21.15.70:0/3693418219 conn(0x7faa780b1d40 msgr2=0x7faa781527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.005+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 shutdown_connections 2024-04-22T01:15:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.005+0000 7faa7fecf640 1 -- 172.21.15.70:0/3693418219 wait complete. 2024-04-22T01:15:38.028 INFO:tasks.cephfs_test_runner:Starting test: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) test: 2024-04-22T01:15:38.028 INFO:tasks.cephfs_test_runner:test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2024-04-22T01:15:38.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.141+0000 7f6394435640 1 Processor -- start 2024-04-22T01:15:38.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.141+0000 7f6394435640 1 -- start start 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f6394435640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f638c15aa30 0x7f638c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f6394435640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f638c15bb50 0x7f638c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f6394435640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f638c06dbd0 con 0x7f638c0aec70 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f6394435640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f638c0af0a0 con 0x7f638c15aa30 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f6394435640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f638c06de70 con 0x7f638c15bb50 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63921d1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f638c15aa30 0x7f638c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63929d2640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f638c0aec70 0x7f638c15e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59362/0 (socket says 172.21.15.70:59362) 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63929d2640 1 -- 172.21.15.70:0/2683321028 learned_addr learned my addr 172.21.15.70:0/2683321028 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63919d0640 1 --2- 172.21.15.70:0/2683321028 >> v2:172.21.15.98:3300/0 conn(0x7f638c15bb50 0x7f638c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63921d1640 1 -- 172.21.15.70:0/2683321028 >> v2:172.21.15.98:3300/0 conn(0x7f638c15bb50 msgr2=0x7f638c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63921d1640 1 --2- 172.21.15.70:0/2683321028 >> v2:172.21.15.98:3300/0 conn(0x7f638c15bb50 0x7f638c0ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63921d1640 1 -- 172.21.15.70:0/2683321028 >> v1:172.21.15.70:6789/0 conn(0x7f638c0aec70 legacy=0x7f638c15e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63921d1640 1 -- 172.21.15.70:0/2683321028 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c06e150 con 0x7f638c15aa30 2024-04-22T01:15:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63911cf640 1 -- 172.21.15.70:0/2683321028 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3274666897 0 0) 0x7f638c06dbd0 con 0x7f638c0aec70 2024-04-22T01:15:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63921d1640 1 --2- 172.21.15.70:0/2683321028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c15aa30 0x7f638c0b4720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f638c0d0780 tx=0x7f638801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c652e4d5bf25fdd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.145+0000 7f63911cf640 1 -- 172.21.15.70:0/2683321028 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880b0070 con 0x7f638c15aa30 2024-04-22T01:15:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63911cf640 1 -- 172.21.15.70:0/2683321028 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63880ab600 con 0x7f638c15aa30 2024-04-22T01:15:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63911cf640 1 -- 172.21.15.70:0/2683321028 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880ab9d0 con 0x7f638c15aa30 2024-04-22T01:15:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/2683321028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c15aa30 msgr2=0x7f638c0b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 --2- 172.21.15.70:0/2683321028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c15aa30 0x7f638c0b4720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f638c0d0780 tx=0x7f638801c480 comp rx=0 tx=0).stop 2024-04-22T01:15:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/2683321028 shutdown_connections 2024-04-22T01:15:38.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 --2- 172.21.15.70:0/2683321028 >> v2:172.21.15.98:3300/0 conn(0x7f638c15bb50 0x7f638c0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 --2- 172.21.15.70:0/2683321028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c15aa30 0x7f638c0b4720 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/2683321028 >> 172.21.15.70:0/2683321028 conn(0x7f638c06b1f0 msgr2=0x7f638c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/2683321028 shutdown_connections 2024-04-22T01:15:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/2683321028 wait complete. 2024-04-22T01:15:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 Processor -- start 2024-04-22T01:15:38.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- start start 2024-04-22T01:15:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 0x7f638c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f638c15aa30 0x7f638c158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c15bb50 0x7f638c13e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f638c15af90 con 0x7f638c0aec70 2024-04-22T01:15:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f638c15ae10 con 0x7f638c15bb50 2024-04-22T01:15:38.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f638c06e950 con 0x7f638c15aa30 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 0x7f638c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 0x7f638c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39882/0 (socket says 172.21.15.70:39882) 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 -- 172.21.15.70:0/1294909519 learned_addr learned my addr 172.21.15.70:0/1294909519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 -- 172.21.15.70:0/1294909519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f638c15aa30 msgr2=0x7f638c158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f638c15aa30 0x7f638c158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 -- 172.21.15.70:0/1294909519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c15bb50 msgr2=0x7f638c13e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c15bb50 0x7f638c13e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 -- 172.21.15.70:0/1294909519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c0b2f50 con 0x7f638c0aec70 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f63921d1640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 0x7f638c158770 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f63880ae040 tx=0x7f638801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880b0050 con 0x7f638c0aec70 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63880bc070 con 0x7f638c0aec70 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638c0b31c0 con 0x7f638c0aec70 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880b5830 con 0x7f638c0aec70 2024-04-22T01:15:38.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.149+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638c13c230 con 0x7f638c0aec70 2024-04-22T01:15:38.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.153+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638c068210 con 0x7f638c0aec70 2024-04-22T01:15:38.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.153+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f63880b59d0 con 0x7f638c0aec70 2024-04-22T01:15:38.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.153+0000 7f6382ffd640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63700678c0 0x7f6370069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.153+0000 7f63919d0640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63700678c0 0x7f6370069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.153+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6388120830 con 0x7f638c0aec70 2024-04-22T01:15:38.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.153+0000 7f63919d0640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63700678c0 0x7f6370069d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f638c1f7e60 tx=0x7f637c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.157+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6388016070 con 0x7f638c0aec70 2024-04-22T01:15:38.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.317+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f638c06f190 con 0x7f638c0aec70 2024-04-22T01:15:38.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6382ffd640 1 -- 172.21.15.70:0/1294909519 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f63880f0f70 con 0x7f638c0aec70 2024-04-22T01:15:38.322 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:38.322 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63700678c0 msgr2=0x7f6370069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63700678c0 0x7f6370069d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f638c1f7e60 tx=0x7f637c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 msgr2=0x7f638c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 0x7f638c158770 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f63880ae040 tx=0x7f638801acc0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 shutdown_connections 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63700678c0 0x7f6370069d80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638c15bb50 0x7f638c13e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f638c15aa30 0x7f638c158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 --2- 172.21.15.70:0/1294909519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f638c0aec70 0x7f638c158770 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 >> 172.21.15.70:0/1294909519 conn(0x7f638c06b1f0 msgr2=0x7f638c15d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 shutdown_connections 2024-04-22T01:15:38.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.321+0000 7f6394435640 1 -- 172.21.15.70:0/1294909519 wait complete. 2024-04-22T01:15:38.341 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:38.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:38.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.457+0000 7f18549c1640 1 Processor -- start 2024-04-22T01:15:38.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.457+0000 7f18549c1640 1 -- start start 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f18549c1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f18500b98a0 0x7f18500d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f18549c1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18500cea10 0x7f18500cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f18549c1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18500b9ce0 con 0x7f185015de20 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f18549c1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f18500b75e0 con 0x7f18500b98a0 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f18549c1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f18500b78c0 con 0x7f18500cea10 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f18500b98a0 0x7f18500d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f18500b98a0 0x7f18500d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39898/0 (socket says 172.21.15.70:39898) 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 -- 172.21.15.70:0/3592168823 learned_addr learned my addr 172.21.15.70:0/3592168823 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 -- 172.21.15.70:0/3592168823 >> v2:172.21.15.98:3300/0 conn(0x7f18500cea10 msgr2=0x7f18500cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184ed9d640 1 --2- 172.21.15.70:0/3592168823 >> v2:172.21.15.98:3300/0 conn(0x7f18500cea10 0x7f18500cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 --2- 172.21.15.70:0/3592168823 >> v2:172.21.15.98:3300/0 conn(0x7f18500cea10 0x7f18500cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 -- 172.21.15.70:0/3592168823 >> v1:172.21.15.98:6789/0 conn(0x7f185015de20 legacy=0x7f18500b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:38.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 -- 172.21.15.70:0/3592168823 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18500b7ba0 con 0x7f18500b98a0 2024-04-22T01:15:38.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184ed9d640 1 --2- 172.21.15.70:0/3592168823 >> v2:172.21.15.98:3300/0 conn(0x7f18500cea10 0x7f18500cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:38.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184dd9b640 1 --2- 172.21.15.70:0/3592168823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500b98a0 0x7f18500d1d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f18500b6d00 tx=0x7f184401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3852ddecafbc8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184d59a640 1 -- 172.21.15.70:0/3592168823 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440aa610 con 0x7f18500b98a0 2024-04-22T01:15:38.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184d59a640 1 -- 172.21.15.70:0/3592168823 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18440af070 con 0x7f18500b98a0 2024-04-22T01:15:38.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.461+0000 7f184d59a640 1 -- 172.21.15.70:0/3592168823 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440aa9d0 con 0x7f18500b98a0 2024-04-22T01:15:38.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/3592168823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500b98a0 msgr2=0x7f18500d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 --2- 172.21.15.70:0/3592168823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500b98a0 0x7f18500d1d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f18500b6d00 tx=0x7f184401c4b0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/3592168823 shutdown_connections 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 --2- 172.21.15.70:0/3592168823 >> v2:172.21.15.98:3300/0 conn(0x7f18500cea10 0x7f18500cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 --2- 172.21.15.70:0/3592168823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500b98a0 0x7f18500d1d80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/3592168823 >> 172.21.15.70:0/3592168823 conn(0x7f1850072d20 msgr2=0x7f1850073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/3592168823 shutdown_connections 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/3592168823 wait complete. 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 Processor -- start 2024-04-22T01:15:38.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- start start 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18500b98a0 0x7f185015b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 0x7f185014c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015de20 0x7f185014c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18500d22c0 con 0x7f18500cea10 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f185015e200 con 0x7f185015de20 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f18500b8e90 con 0x7f18500b98a0 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 0x7f185014c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 0x7f185014c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39912/0 (socket says 172.21.15.70:39912) 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 -- 172.21.15.70:0/1797499027 learned_addr learned my addr 172.21.15.70:0/1797499027 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184ed9d640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015de20 0x7f185014c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184e59c640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18500b98a0 0x7f185015b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 -- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18500b98a0 msgr2=0x7f185015b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18500b98a0 0x7f185015b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 -- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015de20 msgr2=0x7f185014c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015de20 0x7f185014c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 -- 172.21.15.70:0/1797499027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185015d2f0 con 0x7f18500cea10 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f184dd9b640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 0x7f185014c0d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f184401cd50 tx=0x7f184401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440b3800 con 0x7f18500cea10 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18440b39a0 con 0x7f18500cea10 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185015d5d0 con 0x7f18500cea10 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440b3d70 con 0x7f18500cea10 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.465+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185014cd10 con 0x7f18500cea10 2024-04-22T01:15:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.469+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1850068210 con 0x7f18500cea10 2024-04-22T01:15:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.469+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f18440b2380 con 0x7f18500cea10 2024-04-22T01:15:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.469+0000 7f183effd640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f18300678c0 0x7f1830069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.469+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1844120240 con 0x7f18500cea10 2024-04-22T01:15:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.469+0000 7f184e59c640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f18300678c0 0x7f1830069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.469+0000 7f184e59c640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f18300678c0 0x7f1830069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1850154210 tx=0x7f1840008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.473+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f18440a7070 con 0x7f18500cea10 2024-04-22T01:15:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.629+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1850068210 con 0x7f18500cea10 2024-04-22T01:15:38.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.629+0000 7f183effd640 1 -- 172.21.15.70:0/1797499027 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f18440160d0 con 0x7f18500cea10 2024-04-22T01:15:38.631 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:38.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f18300678c0 msgr2=0x7f1830069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f18300678c0 0x7f1830069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1850154210 tx=0x7f1840008040 comp rx=0 tx=0).stop 2024-04-22T01:15:38.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 msgr2=0x7f185014c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 0x7f185014c0d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f184401cd50 tx=0x7f184401acf0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 shutdown_connections 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f18300678c0 0x7f1830069d80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015de20 0x7f185014c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18500cea10 0x7f185014c0d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 --2- 172.21.15.70:0/1797499027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18500b98a0 0x7f185015b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 >> 172.21.15.70:0/1797499027 conn(0x7f1850072d20 msgr2=0x7f18500b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 shutdown_connections 2024-04-22T01:15:38.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.633+0000 7f18549c1640 1 -- 172.21.15.70:0/1797499027 wait complete. 2024-04-22T01:15:38.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.757+0000 7f4270919640 1 Processor -- start 2024-04-22T01:15:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.757+0000 7f4270919640 1 -- start start 2024-04-22T01:15:38.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f4270919640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f426c0b4450 0x7f426c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f4270919640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f426c0cd220 0x7f426c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f4270919640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f426c15c290 con 0x7f426c0720b0 2024-04-22T01:15:38.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f4270919640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f426c15c570 con 0x7f426c0cd220 2024-04-22T01:15:38.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f4270919640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f426c15c850 con 0x7f426c0b4450 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f426c0cd220 0x7f426c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f426c0cd220 0x7f426c0b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39918/0 (socket says 172.21.15.70:39918) 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 -- 172.21.15.70:0/2537785147 learned_addr learned my addr 172.21.15.70:0/2537785147 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426a59c640 1 --2- 172.21.15.70:0/2537785147 >> v2:172.21.15.98:3300/0 conn(0x7f426c0b4450 0x7f426c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 -- 172.21.15.70:0/2537785147 >> v2:172.21.15.98:3300/0 conn(0x7f426c0b4450 msgr2=0x7f426c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 --2- 172.21.15.70:0/2537785147 >> v2:172.21.15.98:3300/0 conn(0x7f426c0b4450 0x7f426c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 -- 172.21.15.70:0/2537785147 >> v1:172.21.15.98:6789/0 conn(0x7f426c0720b0 legacy=0x7f426c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 -- 172.21.15.70:0/2537785147 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426c15cb30 con 0x7f426c0cd220 2024-04-22T01:15:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426ad9d640 1 --2- 172.21.15.70:0/2537785147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c0b2a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f426c06fce0 tx=0x7f426401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ae19cae787c286 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426959a640 1 -- 172.21.15.70:0/2537785147 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42640a5670 con 0x7f426c0cd220 2024-04-22T01:15:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426959a640 1 -- 172.21.15.70:0/2537785147 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42640a5810 con 0x7f426c0cd220 2024-04-22T01:15:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.761+0000 7f426959a640 1 -- 172.21.15.70:0/2537785147 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42640a5be0 con 0x7f426c0cd220 2024-04-22T01:15:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/2537785147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 msgr2=0x7f426c0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 --2- 172.21.15.70:0/2537785147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c0b2a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f426c06fce0 tx=0x7f426401c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/2537785147 shutdown_connections 2024-04-22T01:15:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 --2- 172.21.15.70:0/2537785147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c0b2a40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 --2- 172.21.15.70:0/2537785147 >> v2:172.21.15.98:3300/0 conn(0x7f426c0b4450 0x7f426c071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/2537785147 >> 172.21.15.70:0/2537785147 conn(0x7f426c0b1d40 msgr2=0x7f426c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/2537785147 shutdown_connections 2024-04-22T01:15:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/2537785147 wait complete. 2024-04-22T01:15:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 Processor -- start 2024-04-22T01:15:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- start start 2024-04-22T01:15:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f426c0720b0 0x7f426c14dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f426c0b4450 0x7f426c157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f426c0b3910 con 0x7f426c0cd220 2024-04-22T01:15:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f426c0b3790 con 0x7f426c0b4450 2024-04-22T01:15:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f426c168b10 con 0x7f426c0720b0 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39934/0 (socket says 172.21.15.70:39934) 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 -- 172.21.15.70:0/701421693 learned_addr learned my addr 172.21.15.70:0/701421693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 -- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f426c0720b0 msgr2=0x7f426c14dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f426c0720b0 0x7f426c14dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 -- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f426c0b4450 msgr2=0x7f426c157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4269d9b640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f426c0b4450 0x7f426c157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f426c0b4450 0x7f426c157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 -- 172.21.15.70:0/701421693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426c14e320 con 0x7f426c0cd220 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f426ad9d640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c151610 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f42640af040 tx=0x7f4264019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264009510 con 0x7f426c0cd220 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42640be070 con 0x7f426c0cd220 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f426c152600 con 0x7f426c0cd220 2024-04-22T01:15:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42640b88d0 con 0x7f426c0cd220 2024-04-22T01:15:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.765+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426c152bc0 con 0x7f426c0cd220 2024-04-22T01:15:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.769+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f426c068210 con 0x7f426c0cd220 2024-04-22T01:15:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.769+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f42640b8a70 con 0x7f426c0cd220 2024-04-22T01:15:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.769+0000 7f425affd640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f424c0678c0 0x7f424c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.769+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f42641210d0 con 0x7f426c0cd220 2024-04-22T01:15:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.769+0000 7f426a59c640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f424c0678c0 0x7f424c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.769+0000 7f426a59c640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f424c0678c0 0x7f424c069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f425c0098f0 tx=0x7f425c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.773+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f42640165a0 con 0x7f426c0cd220 2024-04-22T01:15:38.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.925+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f426c0d01e0 con 0x7f426c0cd220 2024-04-22T01:15:38.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.925+0000 7f425affd640 1 -- 172.21.15.70:0/701421693 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f42640f18d0 con 0x7f426c0cd220 2024-04-22T01:15:38.927 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:38.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f424c0678c0 msgr2=0x7f424c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f424c0678c0 0x7f424c069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f425c0098f0 tx=0x7f425c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:38.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 msgr2=0x7f426c151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:38.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c151610 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f42640af040 tx=0x7f4264019750 comp rx=0 tx=0).stop 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 shutdown_connections 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f424c0678c0 0x7f424c069d80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f426c0cd220 0x7f426c151610 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f426c0b4450 0x7f426c157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 --2- 172.21.15.70:0/701421693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f426c0720b0 0x7f426c14dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 >> 172.21.15.70:0/701421693 conn(0x7f426c0b1d40 msgr2=0x7f426c15dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 shutdown_connections 2024-04-22T01:15:38.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:38.929+0000 7f4270919640 1 -- 172.21.15.70:0/701421693 wait complete. 2024-04-22T01:15:38.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:39.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.053+0000 7fca7e527640 1 Processor -- start 2024-04-22T01:15:39.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.053+0000 7fca7e527640 1 -- start start 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca7e527640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fca78162350 0x7fca7806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca7e527640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca78072280 0x7fca780ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca7e527640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca7806e340 con 0x7fca7806de30 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca7e527640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fca7806e620 con 0x7fca78162350 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca7e527640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fca7806e900 con 0x7fca78072280 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca77fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fca78162350 0x7fca7806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca777fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fca7806de30 0x7fca78071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59376/0 (socket says 172.21.15.70:59376) 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca777fe640 1 -- 172.21.15.70:0/505609961 learned_addr learned my addr 172.21.15.70:0/505609961 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:39.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca7cac4640 1 --2- 172.21.15.70:0/505609961 >> v2:172.21.15.98:3300/0 conn(0x7fca78072280 0x7fca780ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca77fff640 1 -- 172.21.15.70:0/505609961 >> v2:172.21.15.98:3300/0 conn(0x7fca78072280 msgr2=0x7fca780ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:39.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca77fff640 1 --2- 172.21.15.70:0/505609961 >> v2:172.21.15.98:3300/0 conn(0x7fca78072280 0x7fca780ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca77fff640 1 -- 172.21.15.70:0/505609961 >> v1:172.21.15.70:6789/0 conn(0x7fca7806de30 legacy=0x7fca78071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca77fff640 1 -- 172.21.15.70:0/505609961 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7806ebe0 con 0x7fca78162350 2024-04-22T01:15:39.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.057+0000 7fca76ffd640 1 -- 172.21.15.70:0/505609961 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367614845 0 0) 0x7fca7806e340 con 0x7fca7806de30 2024-04-22T01:15:39.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca77fff640 1 --2- 172.21.15.70:0/505609961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78162350 0x7fca7806d8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca780d0ab0 tx=0x7fca6801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76368dcd1656a5c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca76ffd640 1 -- 172.21.15.70:0/505609961 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680a7070 con 0x7fca78162350 2024-04-22T01:15:39.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca76ffd640 1 -- 172.21.15.70:0/505609961 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca680b0070 con 0x7fca78162350 2024-04-22T01:15:39.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca76ffd640 1 -- 172.21.15.70:0/505609961 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680ab890 con 0x7fca78162350 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/505609961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78162350 msgr2=0x7fca7806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 --2- 172.21.15.70:0/505609961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78162350 0x7fca7806d8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca780d0ab0 tx=0x7fca6801c450 comp rx=0 tx=0).stop 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/505609961 shutdown_connections 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 --2- 172.21.15.70:0/505609961 >> v2:172.21.15.98:3300/0 conn(0x7fca78072280 0x7fca780ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 --2- 172.21.15.70:0/505609961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78162350 0x7fca7806d8f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/505609961 >> 172.21.15.70:0/505609961 conn(0x7fca780b1d40 msgr2=0x7fca780cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/505609961 shutdown_connections 2024-04-22T01:15:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/505609961 wait complete. 2024-04-22T01:15:39.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 Processor -- start 2024-04-22T01:15:39.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- start start 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca7806de30 0x7fca7815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 0x7fca78149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca78162350 0x7fca78149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca7816f4c0 con 0x7fca78072280 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca7816f340 con 0x7fca78162350 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fca7816f640 con 0x7fca7806de30 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 0x7fca78149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 0x7fca78149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39964/0 (socket says 172.21.15.70:39964) 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 -- 172.21.15.70:0/4241736636 learned_addr learned my addr 172.21.15.70:0/4241736636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca77fff640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca7806de30 0x7fca7815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7cac4640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca78162350 0x7fca78149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 -- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca7806de30 msgr2=0x7fca7815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca7806de30 0x7fca7815adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 -- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca78162350 msgr2=0x7fca78149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca78162350 0x7fca78149ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 -- 172.21.15.70:0/4241736636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7814a6b0 con 0x7fca78072280 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca77fff640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca7806de30 0x7fca7815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca777fe640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 0x7fca78149ab0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0011f0 tx=0x7fca6c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c018070 con 0x7fca78072280 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c0074e0 con 0x7fca78072280 2024-04-22T01:15:39.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c004a10 con 0x7fca78072280 2024-04-22T01:15:39.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca781522b0 con 0x7fca78072280 2024-04-22T01:15:39.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.061+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca78152870 con 0x7fca78072280 2024-04-22T01:15:39.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.065+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca78152f20 con 0x7fca78072280 2024-04-22T01:15:39.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.065+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fca6c01b070 con 0x7fca78072280 2024-04-22T01:15:39.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.065+0000 7fca74ff9640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca640678c0 0x7fca64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.065+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fca6c0853d0 con 0x7fca78072280 2024-04-22T01:15:39.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.065+0000 7fca77fff640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca640678c0 0x7fca64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.065+0000 7fca77fff640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca640678c0 0x7fca64069d80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fca680af000 tx=0x7fca68008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.069+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fca6c055b10 con 0x7fca78072280 2024-04-22T01:15:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca78068210 con 0x7fca78072280 2024-04-22T01:15:39.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca74ff9640 1 -- 172.21.15.70:0/4241736636 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fca6c055410 con 0x7fca78072280 2024-04-22T01:15:39.230 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:39.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca640678c0 msgr2=0x7fca64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca640678c0 0x7fca64069d80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fca680af000 tx=0x7fca68008040 comp rx=0 tx=0).stop 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 msgr2=0x7fca78149ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 0x7fca78149ab0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0011f0 tx=0x7fca6c00c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 shutdown_connections 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca640678c0 0x7fca64069d80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca78162350 0x7fca78149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca78072280 0x7fca78149ab0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 --2- 172.21.15.70:0/4241736636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca7806de30 0x7fca7815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 >> 172.21.15.70:0/4241736636 conn(0x7fca780b1d40 msgr2=0x7fca780b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 shutdown_connections 2024-04-22T01:15:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.229+0000 7fca7e527640 1 -- 172.21.15.70:0/4241736636 wait complete. 2024-04-22T01:15:39.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:39.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.357+0000 7fd930035640 1 Processor -- start 2024-04-22T01:15:39.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.357+0000 7fd930035640 1 -- start start 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd930035640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9280ce920 0x7fd928154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd930035640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9280ceda0 0x7fd9281551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd930035640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9280b3800 con 0x7fd92815e6e0 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd930035640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd9280b3ae0 con 0x7fd9280ceda0 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd930035640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd9280cde40 con 0x7fd9280ce920 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92ddd1640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd92815e6e0 0x7fd92815eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59378/0 (socket says 172.21.15.70:59378) 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92ddd1640 1 -- 172.21.15.70:0/3884562108 learned_addr learned my addr 172.21.15.70:0/3884562108 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 --2- 172.21.15.70:0/3884562108 >> v2:172.21.15.98:3300/0 conn(0x7fd9280ceda0 0x7fd9281551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3009983499 0 0) 0x7fd9280b3800 con 0x7fd92815e6e0 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd914003530 con 0x7fd92815e6e0 2024-04-22T01:15:39.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92d5d0640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9280ce920 0x7fd928154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4285724654 0 0) 0x7fd914003530 con 0x7fd92815e6e0 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9280ce920 msgr2=0x7fd928154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9280ce920 0x7fd928154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 >> v2:172.21.15.98:3300/0 conn(0x7fd9280ceda0 msgr2=0x7fd9281551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 --2- 172.21.15.70:0/3884562108 >> v2:172.21.15.98:3300/0 conn(0x7fd9280ceda0 0x7fd9281551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9280ce120 con 0x7fd92815e6e0 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fd924001680 con 0x7fd92815e6e0 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 >> v1:172.21.15.70:6789/0 conn(0x7fd92815e6e0 legacy=0x7fd92815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9140039c0 0x7fd914003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd914004810 0x7fd914004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd91400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9280b3ae0 con 0x7fd9140051d0 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9280cde40 con 0x7fd9140039c0 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd91400bf40 con 0x7fd914004810 2024-04-22T01:15:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92d5d0640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9280ce920 0x7fd928154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd91400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 -- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd914004810 msgr2=0x7fd914004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd914004810 0x7fd914004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 -- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9140039c0 msgr2=0x7fd914003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9140039c0 0x7fd914003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.361+0000 7fd92e5d2640 1 -- 172.21.15.70:0/3884562108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd914003530 con 0x7fd9140051d0 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92e5d2640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd91400b9d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd920010350 tx=0x7fd92001eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b8718be16e8c05 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92000be20 con 0x7fd9140051d0 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92000b0d0 con 0x7fd9140051d0 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92cdcf640 1 -- 172.21.15.70:0/3884562108 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920009540 con 0x7fd9140051d0 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/3884562108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 msgr2=0x7fd91400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 --2- 172.21.15.70:0/3884562108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd91400b9d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd920010350 tx=0x7fd92001eec0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92e5d2640 1 -- 172.21.15.70:0/3884562108 reap_dead start 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/3884562108 shutdown_connections 2024-04-22T01:15:39.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/3884562108 >> 172.21.15.70:0/3884562108 conn(0x7fd9280b1d40 msgr2=0x7fd9280cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:39.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/3884562108 shutdown_connections 2024-04-22T01:15:39.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/3884562108 wait complete. 2024-04-22T01:15:39.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 Processor -- start 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- start start 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd9280af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd928146bc0 0x7fd928153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd928146030 0x7fd9281464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd92806f260 con 0x7fd9140051d0 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd92806f0e0 con 0x7fd928146bc0 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd928154c60 con 0x7fd928146030 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd9280af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd9280af5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39982/0 (socket says 172.21.15.70:39982) 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 -- 172.21.15.70:0/289252303 learned_addr learned my addr 172.21.15.70:0/289252303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92d5d0640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd928146bc0 0x7fd928153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 -- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd928146030 msgr2=0x7fd9281464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd928146030 0x7fd9281464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 -- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd928146bc0 msgr2=0x7fd928153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd928146bc0 0x7fd928153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 -- 172.21.15.70:0/289252303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9281f3ca0 con 0x7fd9140051d0 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd92ddd1640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd9280af5b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd92400c7a0 tx=0x7fd92400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924018070 con 0x7fd9140051d0 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd924007500 con 0x7fd9140051d0 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9280afd20 con 0x7fd9140051d0 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924004a30 con 0x7fd9140051d0 2024-04-22T01:15:39.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.365+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9280b02e0 con 0x7fd9140051d0 2024-04-22T01:15:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.369+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd928068210 con 0x7fd9140051d0 2024-04-22T01:15:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.369+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd92401b070 con 0x7fd9140051d0 2024-04-22T01:15:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.369+0000 7fd91e7fc640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd90c0678f0 0x7fd90c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.369+0000 7fd92d5d0640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd90c0678f0 0x7fd90c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.369+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd924085420 con 0x7fd9140051d0 2024-04-22T01:15:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.369+0000 7fd92d5d0640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd90c0678f0 0x7fd90c069db0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd92814fee0 tx=0x7fd918008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.373+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd924055bf0 con 0x7fd9140051d0 2024-04-22T01:15:39.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd928152fa0 con 0x7fd9140051d0 2024-04-22T01:15:39.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd91e7fc640 1 -- 172.21.15.70:0/289252303 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fd9240554f0 con 0x7fd9140051d0 2024-04-22T01:15:39.522 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:39.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd90c0678f0 msgr2=0x7fd90c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd90c0678f0 0x7fd90c069db0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd92814fee0 tx=0x7fd918008040 comp rx=0 tx=0).stop 2024-04-22T01:15:39.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 msgr2=0x7fd9280af5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd9280af5b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd92400c7a0 tx=0x7fd92400cc60 comp rx=0 tx=0).stop 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 shutdown_connections 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd90c0678f0 0x7fd90c069db0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd928146030 0x7fd9281464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd928146bc0 0x7fd928153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 --2- 172.21.15.70:0/289252303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9140051d0 0x7fd9280af5b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 >> 172.21.15.70:0/289252303 conn(0x7fd9280b1d40 msgr2=0x7fd92815b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 shutdown_connections 2024-04-22T01:15:39.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.521+0000 7fd930035640 1 -- 172.21.15.70:0/289252303 wait complete. 2024-04-22T01:15:39.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values' 2024-04-22T01:15:39.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 Processor -- start 2024-04-22T01:15:39.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 -- start start 2024-04-22T01:15:39.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d2815bb40 0x7f3d28072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 0x7f3d28161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2815bf20 con 0x7f3d2806ed40 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2806e8b0 con 0x7f3d2815bb40 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2f64c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3d280aec30 con 0x7f3d280b3cd0 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2dbe9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 0x7f3d28161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2d3e8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d2815bb40 0x7f3d28072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2dbe9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 0x7f3d28161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:44390/0 (socket says 172.21.15.70:44390) 2024-04-22T01:15:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.653+0000 7f3d2dbe9640 1 -- 172.21.15.70:0/2788315762 learned_addr learned my addr 172.21.15.70:0/2788315762 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2dbe9640 1 -- 172.21.15.70:0/2788315762 >> v2:172.21.15.98:3300/0 conn(0x7f3d2815bb40 msgr2=0x7f3d28072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2dbe9640 1 --2- 172.21.15.70:0/2788315762 >> v2:172.21.15.98:3300/0 conn(0x7f3d2815bb40 0x7f3d28072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2dbe9640 1 -- 172.21.15.70:0/2788315762 >> v1:172.21.15.98:6789/0 conn(0x7f3d2806ed40 legacy=0x7f3d280b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2dbe9640 1 -- 172.21.15.70:0/2788315762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d280aef10 con 0x7f3d280b3cd0 2024-04-22T01:15:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2dbe9640 1 --2- 172.21.15.70:0/2788315762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 0x7f3d28161ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009e50 tx=0x7f3d2001ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ef451af58834d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d1ffff640 1 -- 172.21.15.70:0/2788315762 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200a58d0 con 0x7f3d280b3cd0 2024-04-22T01:15:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d1ffff640 1 -- 172.21.15.70:0/2788315762 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d200a5a70 con 0x7f3d280b3cd0 2024-04-22T01:15:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d1ffff640 1 -- 172.21.15.70:0/2788315762 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200a5e40 con 0x7f3d280b3cd0 2024-04-22T01:15:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 -- 172.21.15.70:0/2788315762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 msgr2=0x7f3d28161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/2788315762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 0x7f3d28161ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009e50 tx=0x7f3d2001ca00 comp rx=0 tx=0).stop 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 -- 172.21.15.70:0/2788315762 shutdown_connections 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/2788315762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280b3cd0 0x7f3d28161ac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/2788315762 >> v2:172.21.15.98:3300/0 conn(0x7f3d2815bb40 0x7f3d28072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 -- 172.21.15.70:0/2788315762 >> 172.21.15.70:0/2788315762 conn(0x7f3d2806b1f0 msgr2=0x7f3d2806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 -- 172.21.15.70:0/2788315762 shutdown_connections 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.657+0000 7f3d2f64c640 1 -- 172.21.15.70:0/2788315762 wait complete. 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 Processor -- start 2024-04-22T01:15:39.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- start start 2024-04-22T01:15:39.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2806ed40 0x7f3d281537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 0x7f3d28154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d2815bb40 0x7f3d28156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2806f740 con 0x7f3d280b3cd0 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2806f5c0 con 0x7f3d2806ed40 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2816ae60 con 0x7f3d2815bb40 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 0x7f3d28154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 0x7f3d28154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39984/0 (socket says 172.21.15.70:39984) 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 -- 172.21.15.70:0/893011846 learned_addr learned my addr 172.21.15.70:0/893011846 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2dbe9640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d2815bb40 0x7f3d28156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2d3e8640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2806ed40 0x7f3d281537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 -- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d2815bb40 msgr2=0x7f3d28156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d2815bb40 0x7f3d28156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 -- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2806ed40 msgr2=0x7f3d281537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2806ed40 0x7f3d281537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 -- 172.21.15.70:0/893011846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d28156eb0 con 0x7f3d280b3cd0 2024-04-22T01:15:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2cbe7640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 0x7f3d28154ee0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800c3a0 tx=0x7f3d1800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d180052d0 con 0x7f3d280b3cd0 2024-04-22T01:15:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d18005470 con 0x7f3d280b3cd0 2024-04-22T01:15:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18007260 con 0x7f3d280b3cd0 2024-04-22T01:15:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d28153e40 con 0x7f3d280b3cd0 2024-04-22T01:15:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d28154400 con 0x7f3d280b3cd0 2024-04-22T01:15:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.661+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d28068210 con 0x7f3d280b3cd0 2024-04-22T01:15:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.665+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3d18007400 con 0x7f3d280b3cd0 2024-04-22T01:15:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.665+0000 7f3d1dffb640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d0c0678f0 0x7f3d0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.665+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3d18085500 con 0x7f3d280b3cd0 2024-04-22T01:15:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.665+0000 7f3d2d3e8640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d0c0678f0 0x7f3d0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.665+0000 7f3d2d3e8640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d0c0678f0 0x7f3d0c069db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009af0 tx=0x7f3d24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:39.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.669+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3d18055c40 con 0x7f3d280b3cd0 2024-04-22T01:15:39.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.869+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]} v 0) -- 0x7f3d2806e040 con 0x7f3d280b3cd0 2024-04-22T01:15:39.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.981+0000 7f3d1dffb640 1 -- 172.21.15.70:0/893011846 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]}]=0 v69) ==== 163+0+0 (secure 0 0 0) 0x7f3d18055540 con 0x7f3d280b3cd0 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d0c0678f0 msgr2=0x7f3d0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d0c0678f0 0x7f3d0c069db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009af0 tx=0x7f3d24008040 comp rx=0 tx=0).stop 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 msgr2=0x7f3d28154ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 0x7f3d28154ee0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800c3a0 tx=0x7f3d1800c860 comp rx=0 tx=0).stop 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 shutdown_connections 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d0c0678f0 0x7f3d0c069db0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d2815bb40 0x7f3d28156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d280b3cd0 0x7f3d28154ee0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 --2- 172.21.15.70:0/893011846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2806ed40 0x7f3d281537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:39.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 >> 172.21.15.70:0/893011846 conn(0x7f3d2806b1f0 msgr2=0x7f3d28161090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:39.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 shutdown_connections 2024-04-22T01:15:39.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:39.985+0000 7f3d2f64c640 1 -- 172.21.15.70:0/893011846 wait complete. 2024-04-22T01:15:40.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-22T01:15:40.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.121+0000 7efdd29aa640 1 Processor -- start 2024-04-22T01:15:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.121+0000 7efdd29aa640 1 -- start start 2024-04-22T01:15:40.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd29aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdcc15a320 0x7efdcc06f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd29aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 0x7efdcc072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd29aa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdcc06da70 con 0x7efdcc071b70 2024-04-22T01:15:40.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd29aa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efdcc06dd50 con 0x7efdcc15a320 2024-04-22T01:15:40.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd29aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc06e030 con 0x7efdcc0ce420 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 0x7efdcc072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdcbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdcc15a320 0x7efdcc06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 0x7efdcc072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:44408/0 (socket says 172.21.15.70:44408) 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 -- 172.21.15.70:0/752535224 learned_addr learned my addr 172.21.15.70:0/752535224 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 -- 172.21.15.70:0/752535224 >> v2:172.21.15.98:3300/0 conn(0x7efdcc15a320 msgr2=0x7efdcc06f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 --2- 172.21.15.70:0/752535224 >> v2:172.21.15.98:3300/0 conn(0x7efdcc15a320 0x7efdcc06f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 -- 172.21.15.70:0/752535224 >> v1:172.21.15.98:6789/0 conn(0x7efdcc071b70 legacy=0x7efdcc0b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdd0f47640 1 -- 172.21.15.70:0/752535224 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc06e310 con 0x7efdcc0ce420 2024-04-22T01:15:40.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.125+0000 7efdcbfff640 1 --2- 172.21.15.70:0/752535224 >> v2:172.21.15.98:3300/0 conn(0x7efdcc15a320 0x7efdcc06f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:40.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd0f47640 1 --2- 172.21.15.70:0/752535224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 0x7efdcc072000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdc4009e50 tx=0x7efdc401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=925498cc16a6f09c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdcaffd640 1 -- 172.21.15.70:0/752535224 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc40a5790 con 0x7efdcc0ce420 2024-04-22T01:15:40.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdcaffd640 1 -- 172.21.15.70:0/752535224 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc40a5930 con 0x7efdcc0ce420 2024-04-22T01:15:40.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdcaffd640 1 -- 172.21.15.70:0/752535224 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc40a5d00 con 0x7efdcc0ce420 2024-04-22T01:15:40.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- 172.21.15.70:0/752535224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 msgr2=0x7efdcc072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 --2- 172.21.15.70:0/752535224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 0x7efdcc072000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdc4009e50 tx=0x7efdc401ca30 comp rx=0 tx=0).stop 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- 172.21.15.70:0/752535224 shutdown_connections 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 --2- 172.21.15.70:0/752535224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc0ce420 0x7efdcc072000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 --2- 172.21.15.70:0/752535224 >> v2:172.21.15.98:3300/0 conn(0x7efdcc15a320 0x7efdcc06f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- 172.21.15.70:0/752535224 >> 172.21.15.70:0/752535224 conn(0x7efdcc0b1d40 msgr2=0x7efdcc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- 172.21.15.70:0/752535224 shutdown_connections 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- 172.21.15.70:0/752535224 wait complete. 2024-04-22T01:15:40.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 Processor -- start 2024-04-22T01:15:40.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- start start 2024-04-22T01:15:40.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc071b70 0x7efdcc14ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 0x7efdcc145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc15a320 0x7efdcc0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc06f810 con 0x7efdcc0ce420 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc06f690 con 0x7efdcc15a320 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdd29aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc0b4470 con 0x7efdcc071b70 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 0x7efdcc145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 0x7efdcc145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39996/0 (socket says 172.21.15.70:39996) 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.129+0000 7efdcb7fe640 1 -- 172.21.15.70:0/1905004022 learned_addr learned my addr 172.21.15.70:0/1905004022 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcbfff640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc071b70 0x7efdcc14ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdd0f47640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc15a320 0x7efdcc0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcb7fe640 1 -- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc071b70 msgr2=0x7efdcc14ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcb7fe640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc071b70 0x7efdcc14ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcb7fe640 1 -- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc15a320 msgr2=0x7efdcc0ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcb7fe640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc15a320 0x7efdcc0ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcb7fe640 1 -- 172.21.15.70:0/1905004022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc14b1b0 con 0x7efdcc0ce420 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcb7fe640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 0x7efdcc145e00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c420 tx=0x7efdc000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00052d0 con 0x7efdcc0ce420 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc0005470 con 0x7efdcc0ce420 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc1fa0b0 con 0x7efdcc0ce420 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0007260 con 0x7efdcc0ce420 2024-04-22T01:15:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1fa670 con 0x7efdcc0ce420 2024-04-22T01:15:40.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdcc068210 con 0x7efdcc0ce420 2024-04-22T01:15:40.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efdc0007400 con 0x7efdcc0ce420 2024-04-22T01:15:40.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdc8ff9640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb80678c0 0x7efdb8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efdc0085500 con 0x7efdcc0ce420 2024-04-22T01:15:40.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcbfff640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb80678c0 0x7efdb8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.133+0000 7efdcbfff640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb80678c0 0x7efdb8069d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009a50 tx=0x7efdbc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.137+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7efdc0055cd0 con 0x7efdcc0ce420 2024-04-22T01:15:40.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.297+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7efdcc159330 con 0x7efdcc0ce420 2024-04-22T01:15:40.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdc8ff9640 1 -- 172.21.15.70:0/1905004022 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7efdc00555d0 con 0x7efdcc0ce420 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb80678c0 msgr2=0x7efdb8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb80678c0 0x7efdb8069d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009a50 tx=0x7efdbc008040 comp rx=0 tx=0).stop 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 msgr2=0x7efdcc145e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 0x7efdcc145e00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c420 tx=0x7efdc000c8e0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 shutdown_connections 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb80678c0 0x7efdb8069d80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc15a320 0x7efdcc0ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdcc0ce420 0x7efdcc145e00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 --2- 172.21.15.70:0/1905004022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdcc071b70 0x7efdcc14ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 >> 172.21.15.70:0/1905004022 conn(0x7efdcc0b1d40 msgr2=0x7efdcc157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:40.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 shutdown_connections 2024-04-22T01:15:40.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.305+0000 7efdd29aa640 1 -- 172.21.15.70:0/1905004022 wait complete. 2024-04-22T01:15:40.325 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:15:40.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:15:40.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:15:40.407 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:15:40.408 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:15:40.408 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:40.414 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:15:40.414 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:40.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 Processor -- start 2024-04-22T01:15:40.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 -- start start 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f62fc15aa30 0x7f62fc0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62fc06dbd0 con 0x7f62fc15bb50 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f62fc06de70 con 0x7f62fc15aa30 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6303e0a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc06e150 con 0x7f62fc0aee40 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f62fc15aa30 0x7f62fc0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f62fc15aa30 0x7f62fc0b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40012/0 (socket says 172.21.15.70:40012) 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 -- 172.21.15.70:0/2015959531 learned_addr learned my addr 172.21.15.70:0/2015959531 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f63023a7640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 -- 172.21.15.70:0/2015959531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 msgr2=0x7f62fc0b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc0b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 -- 172.21.15.70:0/2015959531 >> v1:172.21.15.98:6789/0 conn(0x7f62fc15bb50 legacy=0x7f62fc0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f6301ba6640 1 -- 172.21.15.70:0/2015959531 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc06e430 con 0x7f62fc15aa30 2024-04-22T01:15:40.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.625+0000 7f63023a7640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc0b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:40.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6301ba6640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0b4720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009870 tx=0x7f62ec01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6488a88dfdc3126a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6300ba4640 1 -- 172.21.15.70:0/2015959531 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0aa430 con 0x7f62fc15aa30 2024-04-22T01:15:40.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6300ba4640 1 -- 172.21.15.70:0/2015959531 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ec0af070 con 0x7f62fc15aa30 2024-04-22T01:15:40.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6300ba4640 1 -- 172.21.15.70:0/2015959531 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0aa7f0 con 0x7f62fc15aa30 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- 172.21.15.70:0/2015959531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 msgr2=0x7f62fc0b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0b4720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009870 tx=0x7f62ec01c4f0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- 172.21.15.70:0/2015959531 shutdown_connections 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc0b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 --2- 172.21.15.70:0/2015959531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0b4720 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- 172.21.15.70:0/2015959531 >> 172.21.15.70:0/2015959531 conn(0x7f62fc06b1f0 msgr2=0x7f62fc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- 172.21.15.70:0/2015959531 shutdown_connections 2024-04-22T01:15:40.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- 172.21.15.70:0/2015959531 wait complete. 2024-04-22T01:15:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 Processor -- start 2024-04-22T01:15:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- start start 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc14a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62fc15bb50 0x7f62fc0aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc06ead0 con 0x7f62fc15aa30 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc06e950 con 0x7f62fc15bb50 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6303e0a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc167a90 con 0x7f62fc0aee40 2024-04-22T01:15:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0a9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40016/0 (socket says 172.21.15.70:40016) 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 -- 172.21.15.70:0/1303224071 learned_addr learned my addr 172.21.15.70:0/1303224071 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63023a7640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62fc15bb50 0x7f62fc0aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f6301ba6640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc14a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 -- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 msgr2=0x7f62fc14a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc14a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 -- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62fc15bb50 msgr2=0x7f62fc0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62fc15bb50 0x7f62fc0aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 -- 172.21.15.70:0/1303224071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc0b02e0 con 0x7f62fc15aa30 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.629+0000 7f63013a5640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0a9da0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c450 tx=0x7f62f800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8012830 con 0x7f62fc15aa30 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f8017070 con 0x7f62fc15aa30 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62fc0b05c0 con 0x7f62fc15aa30 2024-04-22T01:15:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8012c30 con 0x7f62fc15aa30 2024-04-22T01:15:40.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62fc0b0b80 con 0x7f62fc15aa30 2024-04-22T01:15:40.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62fc068210 con 0x7f62fc15aa30 2024-04-22T01:15:40.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f62f801c460 con 0x7f62fc15aa30 2024-04-22T01:15:40.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f62f27fc640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62e00678c0 0x7f62e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f62f8085dd0 con 0x7f62fc15aa30 2024-04-22T01:15:40.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f6301ba6640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62e00678c0 0x7f62e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.633+0000 7f6301ba6640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62e00678c0 0x7f62e0069d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0ae000 tx=0x7f62ec0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.637+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f62f80565a0 con 0x7f62fc15aa30 2024-04-22T01:15:40.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.801+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62fc06f190 con 0x7f62fc15aa30 2024-04-22T01:15:40.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.801+0000 7f62f27fc640 1 -- 172.21.15.70:0/1303224071 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f62f8087070 con 0x7f62fc15aa30 2024-04-22T01:15:40.803 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:40.803 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62e00678c0 msgr2=0x7f62e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62e00678c0 0x7f62e0069d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0ae000 tx=0x7f62ec0a7000 comp rx=0 tx=0).stop 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 msgr2=0x7f62fc0a9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0a9da0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c450 tx=0x7f62f800c910 comp rx=0 tx=0).stop 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 shutdown_connections 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62e00678c0 0x7f62e0069d80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62fc15bb50 0x7f62fc0aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62fc15aa30 0x7f62fc0a9da0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 --2- 172.21.15.70:0/1303224071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62fc0aee40 0x7f62fc14a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 >> 172.21.15.70:0/1303224071 conn(0x7f62fc06b1f0 msgr2=0x7f62fc06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 shutdown_connections 2024-04-22T01:15:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.805+0000 7f6303e0a640 1 -- 172.21.15.70:0/1303224071 wait complete. 2024-04-22T01:15:40.822 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:40.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 Processor -- start 2024-04-22T01:15:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 -- start start 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e4415de10 0x7f8e440d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4415e1f0 0x7f8e440d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e440d2610 0x7f8e440d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8e440b8bc0 con 0x7f8e4415de10 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8e440b8ea0 con 0x7f8e440d2610 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e489a2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e440b9180 con 0x7f8e4415e1f0 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e4415de10 0x7f8e440d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e4415de10 0x7f8e440d0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40026/0 (socket says 172.21.15.70:40026) 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 -- 172.21.15.70:0/3516318905 learned_addr learned my addr 172.21.15.70:0/3516318905 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e42d9d640 1 --2- 172.21.15.70:0/3516318905 >> v2:172.21.15.98:3300/0 conn(0x7f8e440d2610 0x7f8e440d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e41d9b640 1 --2- 172.21.15.70:0/3516318905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4415e1f0 0x7f8e440d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 -- 172.21.15.70:0/3516318905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4415e1f0 msgr2=0x7f8e440d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 --2- 172.21.15.70:0/3516318905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4415e1f0 0x7f8e440d11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 -- 172.21.15.70:0/3516318905 >> v2:172.21.15.98:3300/0 conn(0x7f8e440d2610 msgr2=0x7f8e440d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 --2- 172.21.15.70:0/3516318905 >> v2:172.21.15.98:3300/0 conn(0x7f8e440d2610 0x7f8e440d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.941+0000 7f8e4259c640 1 -- 172.21.15.70:0/3516318905 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e440d1e10 con 0x7f8e4415de10 2024-04-22T01:15:40.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e4259c640 1 --2- 172.21.15.70:0/3516318905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e440d0ca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8e440cad80 tx=0x7f8e3401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119b93fd33b391d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e4159a640 1 -- 172.21.15.70:0/3516318905 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e340a7070 con 0x7f8e4415de10 2024-04-22T01:15:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e4159a640 1 -- 172.21.15.70:0/3516318905 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e340b0070 con 0x7f8e4415de10 2024-04-22T01:15:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e4159a640 1 -- 172.21.15.70:0/3516318905 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e340ab890 con 0x7f8e4415de10 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 -- 172.21.15.70:0/3516318905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 msgr2=0x7f8e440d0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3516318905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e440d0ca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8e440cad80 tx=0x7f8e3401c490 comp rx=0 tx=0).stop 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 -- 172.21.15.70:0/3516318905 shutdown_connections 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3516318905 >> v2:172.21.15.98:3300/0 conn(0x7f8e440d2610 0x7f8e440d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3516318905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4415e1f0 0x7f8e440d11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3516318905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e440d0ca0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 -- 172.21.15.70:0/3516318905 >> 172.21.15.70:0/3516318905 conn(0x7f8e44072d20 msgr2=0x7f8e44073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 -- 172.21.15.70:0/3516318905 shutdown_connections 2024-04-22T01:15:40.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 -- 172.21.15.70:0/3516318905 wait complete. 2024-04-22T01:15:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 Processor -- start 2024-04-22T01:15:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.945+0000 7f8e489a2640 1 -- start start 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e440d2610 0x7f8e44159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e441f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4415e1f0 0x7f8e44153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e440b7a00 con 0x7f8e4415de10 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e440b7880 con 0x7f8e4415e1f0 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e440b7b80 con 0x7f8e440d2610 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e441f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e441f6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40034/0 (socket says 172.21.15.70:40034) 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 -- 172.21.15.70:0/3906125884 learned_addr learned my addr 172.21.15.70:0/3906125884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e4259c640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e440d2610 0x7f8e44159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e42d9d640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4415e1f0 0x7f8e44153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 -- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e440d2610 msgr2=0x7f8e44159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e440d2610 0x7f8e44159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 -- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4415e1f0 msgr2=0x7f8e44153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4415e1f0 0x7f8e44153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 -- 172.21.15.70:0/3906125884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4415f580 con 0x7f8e4415de10 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e41d9b640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e441f6eb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800d5f0 tx=0x7f8e3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e380125f0 con 0x7f8e4415de10 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e38019070 con 0x7f8e4415de10 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4415f860 con 0x7f8e4415de10 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e380129d0 con 0x7f8e4415de10 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4415fe20 con 0x7f8e4415de10 2024-04-22T01:15:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e44160300 con 0x7f8e4415de10 2024-04-22T01:15:40.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8e38012b70 con 0x7f8e4415de10 2024-04-22T01:15:40.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e32ffd640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e240678c0 0x7f8e24069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:40.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8e380864c0 con 0x7f8e4415de10 2024-04-22T01:15:40.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e4259c640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e240678c0 0x7f8e24069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:40.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.949+0000 7f8e4259c640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e240678c0 0x7f8e24069d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8e340af000 tx=0x7f8e340a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:40.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:40.953+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8e44160300 con 0x7f8e4415de10 2024-04-22T01:15:41.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.101+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8e440b4530 con 0x7f8e4415de10 2024-04-22T01:15:41.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e32ffd640 1 -- 172.21.15.70:0/3906125884 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f8e38056c00 con 0x7f8e4415de10 2024-04-22T01:15:41.105 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e240678c0 msgr2=0x7f8e24069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e240678c0 0x7f8e24069d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8e340af000 tx=0x7f8e340a7000 comp rx=0 tx=0).stop 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 msgr2=0x7f8e441f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e441f6eb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800d5f0 tx=0x7f8e3800dab0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 shutdown_connections 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e240678c0 0x7f8e24069d80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4415e1f0 0x7f8e44153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e4415de10 0x7f8e441f6eb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 --2- 172.21.15.70:0/3906125884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e440d2610 0x7f8e44159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 >> 172.21.15.70:0/3906125884 conn(0x7f8e44072d20 msgr2=0x7f8e440b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 shutdown_connections 2024-04-22T01:15:41.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.105+0000 7f8e489a2640 1 -- 172.21.15.70:0/3906125884 wait complete. 2024-04-22T01:15:41.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.233+0000 7f1cb1d98640 1 Processor -- start 2024-04-22T01:15:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.233+0000 7f1cb1d98640 1 -- start start 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cb1d98640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cac0b4450 0x7f1cac071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cb1d98640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac15c290 con 0x7f1cac0cd220 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cb1d98640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac15c570 con 0x7f1cac0720b0 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cb1d98640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1cac15c850 con 0x7f1cac0b4450 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1cac0cd220 0x7f1cac0b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59384/0 (socket says 172.21.15.70:59384) 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 -- 172.21.15.70:0/2250070130 learned_addr learned my addr 172.21.15.70:0/2250070130 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cab7fe640 1 --2- 172.21.15.70:0/2250070130 >> v2:172.21.15.98:3300/0 conn(0x7f1cac0b4450 0x7f1cac071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989139971 0 0) 0x7f1cac15c290 con 0x7f1cac0cd220 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c94003530 con 0x7f1cac0cd220 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1169492318 0 0) 0x7f1c94003530 con 0x7f1cac0cd220 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 >> v2:172.21.15.98:3300/0 conn(0x7f1cac0b4450 msgr2=0x7f1cac071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 --2- 172.21.15.70:0/2250070130 >> v2:172.21.15.98:3300/0 conn(0x7f1cac0b4450 0x7f1cac071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 >> v1:172.21.15.98:6789/0 conn(0x7f1cac0720b0 legacy=0x7f1cac0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac15cb30 con 0x7f1cac0cd220 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1ca4001a00 con 0x7f1cac0cd220 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 >> v1:172.21.15.70:6789/0 conn(0x7f1cac0cd220 legacy=0x7f1cac0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940039c0 0x7f1c94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c94004810 0x7f1c94004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c940051d0 0x7f1c9400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac15c850 con 0x7f1c940051d0 2024-04-22T01:15:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9400c120 con 0x7f1c94004810 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9400c320 con 0x7f1c940039c0 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cab7fe640 1 --2- 172.21.15.70:0/2250070130 >> v2:172.21.15.98:3300/0 conn(0x7f1cac0b4450 0x7f1cac071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c940051d0 0x7f1c9400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caaffd640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c94004810 0x7f1c94004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 -- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940039c0 msgr2=0x7f1c94003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940039c0 0x7f1c94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 -- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c94004810 msgr2=0x7f1c94004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c94004810 0x7f1c94004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 -- 172.21.15.70:0/2250070130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c94003530 con 0x7f1c940051d0 2024-04-22T01:15:41.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 -- 172.21.15.70:0/2250070130 reap_dead start 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1cabfff640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c940051d0 0x7f1c9400b9d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400a840 tx=0x7f1ca401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c38ec5b29a1261 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca400a460 con 0x7f1c940051d0 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca40ac640 con 0x7f1c940051d0 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.237+0000 7f1caa7fc640 1 -- 172.21.15.70:0/2250070130 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca40aca30 con 0x7f1c940051d0 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2250070130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c940051d0 msgr2=0x7f1c9400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c940051d0 0x7f1c9400b9d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400a840 tx=0x7f1ca401d360 comp rx=0 tx=0).stop 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2250070130 shutdown_connections 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2250070130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c940051d0 0x7f1c9400b9d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2250070130 >> 172.21.15.70:0/2250070130 conn(0x7f1cac0b1d40 msgr2=0x7f1cac0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2250070130 shutdown_connections 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2250070130 wait complete. 2024-04-22T01:15:41.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 Processor -- start 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- start start 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940051d0 0x7f1cac150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 0x7f1cac14a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac14d650 0x7f1cac14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac0b49b0 con 0x7f1cac15b780 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac0b4830 con 0x7f1cac14d650 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1cac0cd650 con 0x7f1c940051d0 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 0x7f1cac14a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 0x7f1cac14a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40058/0 (socket says 172.21.15.70:40058) 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 -- 172.21.15.70:0/2161309942 learned_addr learned my addr 172.21.15.70:0/2161309942 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cabfff640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac14d650 0x7f1cac14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cab7fe640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940051d0 0x7f1cac150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 -- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940051d0 msgr2=0x7f1cac150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940051d0 0x7f1cac150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 -- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac14d650 msgr2=0x7f1cac14db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac14d650 0x7f1cac14db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 -- 172.21.15.70:0/2161309942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac151600 con 0x7f1cac15b780 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1caaffd640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 0x7f1cac14a490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000d470 tx=0x7f1ca000d930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0018070 con 0x7f1cac15b780 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca0003620 con 0x7f1cac15b780 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cac1518e0 con 0x7f1cac15b780 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca00039d0 con 0x7f1cac15b780 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cac14b240 con 0x7f1cac15b780 2024-04-22T01:15:41.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.241+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cac068210 con 0x7f1cac15b780 2024-04-22T01:15:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.245+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1ca001b070 con 0x7f1cac15b780 2024-04-22T01:15:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.245+0000 7f1c93fff640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1c980678f0 0x7f1c98069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.245+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1ca00856d0 con 0x7f1cac15b780 2024-04-22T01:15:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.245+0000 7f1cab7fe640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1c980678f0 0x7f1c98069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.245+0000 7f1cab7fe640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1c980678f0 0x7f1c98069db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d7f0 tx=0x7f1c9c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.249+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1ca0055850 con 0x7f1cac15b780 2024-04-22T01:15:41.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.405+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1cac14a9d0 con 0x7f1cac15b780 2024-04-22T01:15:41.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.405+0000 7f1c93fff640 1 -- 172.21.15.70:0/2161309942 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f1ca0059b50 con 0x7f1cac15b780 2024-04-22T01:15:41.407 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1c980678f0 msgr2=0x7f1c98069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1c980678f0 0x7f1c98069db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d7f0 tx=0x7f1c9c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 msgr2=0x7f1cac14a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 0x7f1cac14a490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000d470 tx=0x7f1ca000d930 comp rx=0 tx=0).stop 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 shutdown_connections 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1c980678f0 0x7f1c98069db0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cac14d650 0x7f1cac14db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cac15b780 0x7f1cac14a490 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 --2- 172.21.15.70:0/2161309942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c940051d0 0x7f1cac150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 >> 172.21.15.70:0/2161309942 conn(0x7f1cac0b1d40 msgr2=0x7f1cac14e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:41.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 shutdown_connections 2024-04-22T01:15:41.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.409+0000 7f1cb1d98640 1 -- 172.21.15.70:0/2161309942 wait complete. 2024-04-22T01:15:41.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:41.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.533+0000 7f84b656a640 1 Processor -- start 2024-04-22T01:15:41.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.533+0000 7f84b656a640 1 -- start start 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84b656a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84b00d01b0 0x7f84b00d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84b656a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84b015e2d0 con 0x7f84b015def0 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84b656a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84b015e570 con 0x7f84b006d480 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84b656a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f84b015e810 con 0x7f84b00d01b0 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84affff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f84b015def0 0x7f84b00cf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59386/0 (socket says 172.21.15.70:59386) 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84affff640 1 -- 172.21.15.70:0/713597983 learned_addr learned my addr 172.21.15.70:0/713597983 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060885204 0 0) 0x7f84b015e2d0 con 0x7f84b015def0 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8498003530 con 0x7f84b015def0 2024-04-22T01:15:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 --2- 172.21.15.70:0/713597983 >> v2:172.21.15.98:3300/0 conn(0x7f84b00d01b0 0x7f84b00d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2188426863 0 0) 0x7f8498003530 con 0x7f84b015def0 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 >> v2:172.21.15.98:3300/0 conn(0x7f84b00d01b0 msgr2=0x7f84b00d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 --2- 172.21.15.70:0/713597983 >> v2:172.21.15.98:3300/0 conn(0x7f84b00d01b0 0x7f84b00d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 >> v1:172.21.15.98:6789/0 conn(0x7f84b006d480 legacy=0x7f84b006d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b015eaf0 con 0x7f84b015def0 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f84a8001a00 con 0x7f84b015def0 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 >> v1:172.21.15.70:6789/0 conn(0x7f84b015def0 legacy=0x7f84b00cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84980039f0 0x7f8498003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8498004810 0x7f8498004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498005220 0x7f849800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f84b015e810 con 0x7f8498004810 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f849800c150 con 0x7f84980039f0 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f849800c350 con 0x7f8498005220 2024-04-22T01:15:41.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8498004810 0x7f8498004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84affff640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84980039f0 0x7f8498003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84b4b07640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498005220 0x7f849800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 -- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498005220 msgr2=0x7f849800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498005220 0x7f849800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 -- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84980039f0 msgr2=0x7f8498003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84980039f0 0x7f8498003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84af7fe640 1 -- 172.21.15.70:0/713597983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8498003530 con 0x7f8498004810 2024-04-22T01:15:41.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.537+0000 7f84b4b07640 1 -- 172.21.15.70:0/713597983 reap_dead start 2024-04-22T01:15:41.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8498004810 0x7f8498004c90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f84a400ba00 tx=0x7f84a401c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86f14fba7dcd5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40b2070 con 0x7f8498004810 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a40ad640 con 0x7f8498004810 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84aeffd640 1 -- 172.21.15.70:0/713597983 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a400b600 con 0x7f8498004810 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- 172.21.15.70:0/713597983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8498004810 msgr2=0x7f8498004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8498004810 0x7f8498004c90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f84a400ba00 tx=0x7f84a401c640 comp rx=0 tx=0).stop 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- 172.21.15.70:0/713597983 shutdown_connections 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 --2- 172.21.15.70:0/713597983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8498004810 0x7f8498004c90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- 172.21.15.70:0/713597983 >> 172.21.15.70:0/713597983 conn(0x7f84b00b36c0 msgr2=0x7f84b00b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- 172.21.15.70:0/713597983 shutdown_connections 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- 172.21.15.70:0/713597983 wait complete. 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 Processor -- start 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- start start 2024-04-22T01:15:41.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498004810 0x7f84b0156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 0x7f84b00b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00c9120 0x7f84b01e7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f84b009ae50 con 0x7f84b00c8ca0 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84b015f0b0 con 0x7f84b00c9120 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b656a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84b009afd0 con 0x7f8498004810 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 0x7f84b00b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 0x7f84b00b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40078/0 (socket says 172.21.15.70:40078) 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 -- 172.21.15.70:0/2853595416 learned_addr learned my addr 172.21.15.70:0/2853595416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b4b07640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00c9120 0x7f84b01e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84affff640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498004810 0x7f84b0156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 -- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498004810 msgr2=0x7f84b0156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498004810 0x7f84b0156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 -- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00c9120 msgr2=0x7f84b01e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00c9120 0x7f84b01e7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 -- 172.21.15.70:0/2853595416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b0157230 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84b4b07640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00c9120 0x7f84b01e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84af7fe640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 0x7f84b00b50e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f84a4097f40 tx=0x7f8498005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40bc030 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.541+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a40ad440 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b00b5b30 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40b2040 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b00b60f0 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84b0068920 con 0x7f84b00c8ca0 2024-04-22T01:15:41.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f84a40bf070 con 0x7f84b00c8ca0 2024-04-22T01:15:41.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84acff9640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f849c0678f0 0x7f849c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f84a4121070 con 0x7f84b00c8ca0 2024-04-22T01:15:41.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84affff640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f849c0678f0 0x7f849c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.545+0000 7f84affff640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f849c0678f0 0x7f849c069db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f84a8001590 tx=0x7f84a80014c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.549+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f84a4016350 con 0x7f84b00c8ca0 2024-04-22T01:15:41.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.697+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84b00b45d0 con 0x7f84b00c8ca0 2024-04-22T01:15:41.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84acff9640 1 -- 172.21.15.70:0/2853595416 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f84a40f1890 con 0x7f84b00c8ca0 2024-04-22T01:15:41.701 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f849c0678f0 msgr2=0x7f849c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f849c0678f0 0x7f849c069db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f84a8001590 tx=0x7f84a80014c0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 msgr2=0x7f84b00b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 0x7f84b00b50e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f84a4097f40 tx=0x7f8498005840 comp rx=0 tx=0).stop 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 shutdown_connections 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f849c0678f0 0x7f849c069db0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b00c9120 0x7f84b01e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84b00c8ca0 0x7f84b00b50e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 --2- 172.21.15.70:0/2853595416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8498004810 0x7f84b0156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 >> 172.21.15.70:0/2853595416 conn(0x7f84b00b36c0 msgr2=0x7f84b00cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:41.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 shutdown_connections 2024-04-22T01:15:41.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.701+0000 7f84b656a640 1 -- 172.21.15.70:0/2853595416 wait complete. 2024-04-22T01:15:41.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:41.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.837+0000 7f97708e8640 1 Processor -- start 2024-04-22T01:15:41.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.837+0000 7f97708e8640 1 -- start start 2024-04-22T01:15:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f97708e8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 0x7f976c0d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f97708e8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f976c15c0d0 0x7f976c099d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f97708e8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976c0ce010 con 0x7f976c09a270 2024-04-22T01:15:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f97708e8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f976c0ce2f0 con 0x7f976c15c0d0 2024-04-22T01:15:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f97708e8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f976c0b10b0 con 0x7f976c15bcf0 2024-04-22T01:15:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f976ad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f976c09a270 0x7f976c0b1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59400/0 (socket says 172.21.15.70:59400) 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f976ad9d640 1 -- 172.21.15.70:0/426476915 learned_addr learned my addr 172.21.15.70:0/426476915 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768915398 0 0) 0x7f976c0ce010 con 0x7f976c09a270 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f976a59c640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 0x7f976c0d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 --2- 172.21.15.70:0/426476915 >> v2:172.21.15.98:3300/0 conn(0x7f976c15c0d0 0x7f976c099d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9758003530 con 0x7f976c09a270 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3843788688 0 0) 0x7f9758003530 con 0x7f976c09a270 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 msgr2=0x7f976c0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 0x7f976c0d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 >> v2:172.21.15.98:3300/0 conn(0x7f976c15c0d0 msgr2=0x7f976c099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 --2- 172.21.15.70:0/426476915 >> v2:172.21.15.98:3300/0 conn(0x7f976c15c0d0 0x7f976c099d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c0b1390 con 0x7f976c09a270 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f9764001a00 con 0x7f976c09a270 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 >> v1:172.21.15.70:6789/0 conn(0x7f976c09a270 legacy=0x7f976c0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97580039c0 0x7f9758003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9758004810 0x7f9758004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 0x7f97580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0ce2f0 con 0x7f9758004810 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0b10b0 con 0x7f97580039c0 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f975800b9d0 con 0x7f976c15bcf0 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9758004810 0x7f9758004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f976ad9d640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 0x7f97580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f976a59c640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97580039c0 0x7f9758003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 -- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 msgr2=0x7f97580051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15bcf0 0x7f97580051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 -- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97580039c0 msgr2=0x7f9758003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97580039c0 0x7f9758003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 -- 172.21.15.70:0/426476915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9758003530 con 0x7f9758004810 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9763fff640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9758004810 0x7f9758004c90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9754009ae0 tx=0x7f975401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d803790e37f6d7c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540b2070 con 0x7f9758004810 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97540a55c0 con 0x7f9758004810 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.841+0000 7f9769d9b640 1 -- 172.21.15.70:0/426476915 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540096b0 con 0x7f9758004810 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/426476915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9758004810 msgr2=0x7f9758004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 --2- 172.21.15.70:0/426476915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9758004810 0x7f9758004c90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9754009ae0 tx=0x7f975401cab0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f976ad9d640 1 -- 172.21.15.70:0/426476915 reap_dead start 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/426476915 shutdown_connections 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/426476915 >> 172.21.15.70:0/426476915 conn(0x7f976c0b3700 msgr2=0x7f976c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/426476915 shutdown_connections 2024-04-22T01:15:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/426476915 wait complete. 2024-04-22T01:15:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 Processor -- start 2024-04-22T01:15:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- start start 2024-04-22T01:15:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9758004810 0x7f976c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 0x7f976c0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c06d7b0 0x7f976c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0d0a60 con 0x7f976c06cbd0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0d08e0 con 0x7f976c06d7b0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f976c0d0be0 con 0x7f9758004810 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 0x7f976c0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 0x7f976c0b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40106/0 (socket says 172.21.15.70:40106) 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 -- 172.21.15.70:0/549988079 learned_addr learned my addr 172.21.15.70:0/549988079 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f976a59c640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9758004810 0x7f976c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 -- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9758004810 msgr2=0x7f976c158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9758004810 0x7f976c158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f976ad9d640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c06d7b0 0x7f976c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 -- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c06d7b0 msgr2=0x7f976c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c06d7b0 0x7f976c06dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 -- 172.21.15.70:0/549988079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c06d340 con 0x7f976c06cbd0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9763fff640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 0x7f976c0b4830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9754009ae0 tx=0x7f9758005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540bc030 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97540a5df0 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976c0b62a0 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540b2040 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.845+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976c0b57c0 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.849+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976c068920 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.849+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f97540bf070 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.849+0000 7f9762ffd640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.849+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9754120000 con 0x7f976c06cbd0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.849+0000 7f976a59c640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.849+0000 7f976a59c640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f975c009850 tx=0x7f975c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:41.853+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f97540167c0 con 0x7f976c06cbd0 2024-04-22T01:15:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.013+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f976c0cd5d0 con 0x7f976c06cbd0 2024-04-22T01:15:42.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.013+0000 7f9762ffd640 1 -- 172.21.15.70:0/549988079 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f97540f08f0 con 0x7f976c06cbd0 2024-04-22T01:15:42.015 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 msgr2=0x7f974c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f975c009850 tx=0x7f975c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 msgr2=0x7f976c0b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 0x7f976c0b4830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9754009ae0 tx=0x7f9758005840 comp rx=0 tx=0).stop 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 shutdown_connections 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c06d7b0 0x7f976c06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c06cbd0 0x7f976c0b4830 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 --2- 172.21.15.70:0/549988079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9758004810 0x7f976c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 >> 172.21.15.70:0/549988079 conn(0x7f976c0b3700 msgr2=0x7f976c071230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 shutdown_connections 2024-04-22T01:15:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.017+0000 7f97708e8640 1 -- 172.21.15.70:0/549988079 wait complete. 2024-04-22T01:15:42.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:42.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 Processor -- start 2024-04-22T01:15:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 -- start start 2024-04-22T01:15:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a815bb50 0x7f08a80cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f08a806f480 0x7f08a8071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a80cd7a0 con 0x7f08a815bf30 2024-04-22T01:15:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f08a815f8b0 con 0x7f08a806f480 2024-04-22T01:15:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08afcab640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f08a815fb00 con 0x7f08a815bb50 2024-04-22T01:15:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ae248640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f08a806f480 0x7f08a8071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ad246640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f08a815bf30 0x7f08a806ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59414/0 (socket says 172.21.15.70:59414) 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ad246640 1 -- 172.21.15.70:0/3383432037 learned_addr learned my addr 172.21.15.70:0/3383432037 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ada47640 1 --2- 172.21.15.70:0/3383432037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a815bb50 0x7f08a80cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ae248640 1 -- 172.21.15.70:0/3383432037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a815bb50 msgr2=0x7f08a80cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ae248640 1 --2- 172.21.15.70:0/3383432037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a815bb50 0x7f08a80cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ae248640 1 -- 172.21.15.70:0/3383432037 >> v1:172.21.15.70:6789/0 conn(0x7f08a815bf30 legacy=0x7f08a806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ae248640 1 -- 172.21.15.70:0/3383432037 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a815fde0 con 0x7f08a806f480 2024-04-22T01:15:42.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08ae248640 1 --2- 172.21.15.70:0/3383432037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a806f480 0x7f08a8071af0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009d90 tx=0x7f08a001c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42accaa131a44087 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.153+0000 7f08aca45640 1 -- 172.21.15.70:0/3383432037 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a00b2070 con 0x7f08a806f480 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08aca45640 1 -- 172.21.15.70:0/3383432037 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a00a5820 con 0x7f08a806f480 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08aca45640 1 -- 172.21.15.70:0/3383432037 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a00a5bf0 con 0x7f08a806f480 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/3383432037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a806f480 msgr2=0x7f08a8071af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 --2- 172.21.15.70:0/3383432037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a806f480 0x7f08a8071af0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009d90 tx=0x7f08a001c800 comp rx=0 tx=0).stop 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/3383432037 shutdown_connections 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 --2- 172.21.15.70:0/3383432037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a806f480 0x7f08a8071af0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 --2- 172.21.15.70:0/3383432037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a815bb50 0x7f08a80cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/3383432037 >> 172.21.15.70:0/3383432037 conn(0x7f08a80cf9e0 msgr2=0x7f08a80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/3383432037 shutdown_connections 2024-04-22T01:15:42.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/3383432037 wait complete. 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 Processor -- start 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- start start 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a806f480 0x7f08a8141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 0x7f08a8143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08a815bf30 0x7f08a81441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8160760 con 0x7f08a815bb50 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81605e0 con 0x7f08a815bf30 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f08a80cf460 con 0x7f08a806f480 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 0x7f08a8143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 0x7f08a8143c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40116/0 (socket says 172.21.15.70:40116) 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 -- 172.21.15.70:0/2476360737 learned_addr learned my addr 172.21.15.70:0/2476360737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ada47640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a806f480 0x7f08a8141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ae248640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08a815bf30 0x7f08a81441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 -- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a806f480 msgr2=0x7f08a8141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a806f480 0x7f08a8141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 -- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08a815bf30 msgr2=0x7f08a81441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08a815bf30 0x7f08a81441a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 -- 172.21.15.70:0/2476360737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a80b39e0 con 0x7f08a815bb50 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ada47640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a806f480 0x7f08a8141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08ad246640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 0x7f08a8143c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0898009df0 tx=0x7f089800c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898019070 con 0x7f08a815bb50 2024-04-22T01:15:42.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08980078f0 con 0x7f08a815bb50 2024-04-22T01:15:42.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a80b3cc0 con 0x7f08a815bb50 2024-04-22T01:15:42.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898007ce0 con 0x7f08a815bb50 2024-04-22T01:15:42.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.157+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a80b4280 con 0x7f08a815bb50 2024-04-22T01:15:42.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.161+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a804f510 con 0x7f08a815bb50 2024-04-22T01:15:42.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.161+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f089801c070 con 0x7f08a815bb50 2024-04-22T01:15:42.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.161+0000 7f089e7fc640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f088c0678c0 0x7f088c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.161+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0898087d60 con 0x7f08a815bb50 2024-04-22T01:15:42.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.161+0000 7f08ada47640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f088c0678c0 0x7f088c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.161+0000 7f08ada47640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f088c0678c0 0x7f088c069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009840 tx=0x7f08a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.165+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f08980584a0 con 0x7f08a815bb50 2024-04-22T01:15:42.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f08a80d0740 con 0x7f08a815bb50 2024-04-22T01:15:42.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f089e7fc640 1 -- 172.21.15.70:0/2476360737 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f0898057da0 con 0x7f08a815bb50 2024-04-22T01:15:42.326 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f088c0678c0 msgr2=0x7f088c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f088c0678c0 0x7f088c069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009840 tx=0x7f08a4008040 comp rx=0 tx=0).stop 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 msgr2=0x7f08a8143c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 0x7f08a8143c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0898009df0 tx=0x7f089800c430 comp rx=0 tx=0).stop 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 shutdown_connections 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f088c0678c0 0x7f088c069d80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08a815bf30 0x7f08a81441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08a815bb50 0x7f08a8143c60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 --2- 172.21.15.70:0/2476360737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08a806f480 0x7f08a8141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 >> 172.21.15.70:0/2476360737 conn(0x7f08a80cf9e0 msgr2=0x7f08a806b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:42.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 shutdown_connections 2024-04-22T01:15:42.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.325+0000 7f08afcab640 1 -- 172.21.15.70:0/2476360737 wait complete. 2024-04-22T01:15:42.343 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:15:42.343 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:42.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:42.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:15:42.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:15:42.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.509+0000 7f2dcb404640 1 Processor -- start 2024-04-22T01:15:42.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dcb404640 1 -- start start 2024-04-22T01:15:42.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dcb404640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2dc41455b0 0x7f2dc40cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dcb404640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2dc4072230 0x7f2dc406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dcb404640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dc40d0250 con 0x7f2dc40cc150 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dcb404640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2dc40cd7b0 con 0x7f2dc4072230 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dcb404640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2dc40cc710 con 0x7f2dc41455b0 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2dc4072230 0x7f2dc406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2dc4072230 0x7f2dc406e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40128/0 (socket says 172.21.15.70:40128) 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 -- 172.21.15.70:0/967950269 learned_addr learned my addr 172.21.15.70:0/967950269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc91a0640 1 --2- 172.21.15.70:0/967950269 >> v2:172.21.15.98:3300/0 conn(0x7f2dc41455b0 0x7f2dc40cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 -- 172.21.15.70:0/967950269 >> v2:172.21.15.98:3300/0 conn(0x7f2dc41455b0 msgr2=0x7f2dc40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 --2- 172.21.15.70:0/967950269 >> v2:172.21.15.98:3300/0 conn(0x7f2dc41455b0 0x7f2dc40cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 -- 172.21.15.70:0/967950269 >> v1:172.21.15.98:6789/0 conn(0x7f2dc40cc150 legacy=0x7f2dc4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 -- 172.21.15.70:0/967950269 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc40cc9f0 con 0x7f2dc4072230 2024-04-22T01:15:42.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.513+0000 7f2dc99a1640 1 --2- 172.21.15.70:0/967950269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc4072230 0x7f2dc406e200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40d0700 tx=0x7f2dbc01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9fc96dc844dc12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dbbfff640 1 -- 172.21.15.70:0/967950269 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0a5590 con 0x7f2dc4072230 2024-04-22T01:15:42.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dbbfff640 1 -- 172.21.15.70:0/967950269 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0a5730 con 0x7f2dc4072230 2024-04-22T01:15:42.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dbbfff640 1 -- 172.21.15.70:0/967950269 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0a5b00 con 0x7f2dc4072230 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/967950269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc4072230 msgr2=0x7f2dc406e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 --2- 172.21.15.70:0/967950269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc4072230 0x7f2dc406e200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40d0700 tx=0x7f2dbc01c8d0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/967950269 shutdown_connections 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 --2- 172.21.15.70:0/967950269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc4072230 0x7f2dc406e200 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 --2- 172.21.15.70:0/967950269 >> v2:172.21.15.98:3300/0 conn(0x7f2dc41455b0 0x7f2dc40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/967950269 >> 172.21.15.70:0/967950269 conn(0x7f2dc406b1f0 msgr2=0x7f2dc406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/967950269 shutdown_connections 2024-04-22T01:15:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/967950269 wait complete. 2024-04-22T01:15:42.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 Processor -- start 2024-04-22T01:15:42.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- start start 2024-04-22T01:15:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc4072230 0x7f2dc413d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc40cc150 0x7f2dc4158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 0x7f2dc4159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc406f2e0 con 0x7f2dc41455b0 2024-04-22T01:15:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc406f160 con 0x7f2dc40cc150 2024-04-22T01:15:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc4168b40 con 0x7f2dc4072230 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 0x7f2dc4159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 0x7f2dc4159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40138/0 (socket says 172.21.15.70:40138) 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 -- 172.21.15.70:0/784483162 learned_addr learned my addr 172.21.15.70:0/784483162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc899f640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc40cc150 0x7f2dc4158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc91a0640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc4072230 0x7f2dc413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 -- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc4072230 msgr2=0x7f2dc413d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc4072230 0x7f2dc413d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 -- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc40cc150 msgr2=0x7f2dc4158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc40cc150 0x7f2dc4158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 -- 172.21.15.70:0/784483162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc415a070 con 0x7f2dc41455b0 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc899f640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc40cc150 0x7f2dc4158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dc99a1640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 0x7f2dc4159b30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0194e0 tx=0x7f2dbc019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0bc020 con 0x7f2dc41455b0 2024-04-22T01:15:42.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0b8690 con 0x7f2dc41455b0 2024-04-22T01:15:42.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc415a2f0 con 0x7f2dc41455b0 2024-04-22T01:15:42.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0a5590 con 0x7f2dc41455b0 2024-04-22T01:15:42.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.517+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc413f470 con 0x7f2dc41455b0 2024-04-22T01:15:42.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.521+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc4068210 con 0x7f2dc41455b0 2024-04-22T01:15:42.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.521+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2dbc0bf020 con 0x7f2dc41455b0 2024-04-22T01:15:42.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.521+0000 7f2db9ffb640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2da80678c0 0x7f2da8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.521+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2dbc120e40 con 0x7f2dc41455b0 2024-04-22T01:15:42.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.521+0000 7f2dc91a0640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2da80678c0 0x7f2da8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.521+0000 7f2dc91a0640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2da80678c0 0x7f2da8069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4144f40 tx=0x7f2dc0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.525+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2dbc0c6030 con 0x7f2dc41455b0 2024-04-22T01:15:42.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.689+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2dc415f510 con 0x7f2dc41455b0 2024-04-22T01:15:42.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2db9ffb640 1 -- 172.21.15.70:0/784483162 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f2dc415a070 con 0x7f2dc41455b0 2024-04-22T01:15:42.693 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:42.693 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2da80678c0 msgr2=0x7f2da8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2da80678c0 0x7f2da8069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4144f40 tx=0x7f2dc0008040 comp rx=0 tx=0).stop 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 msgr2=0x7f2dc4159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 0x7f2dc4159b30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0194e0 tx=0x7f2dbc019670 comp rx=0 tx=0).stop 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 shutdown_connections 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2da80678c0 0x7f2da8069d80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2dc41455b0 0x7f2dc4159b30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc40cc150 0x7f2dc4158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 --2- 172.21.15.70:0/784483162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc4072230 0x7f2dc413d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 >> 172.21.15.70:0/784483162 conn(0x7f2dc406b1f0 msgr2=0x7f2dc413c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:42.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 shutdown_connections 2024-04-22T01:15:42.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.693+0000 7f2dcb404640 1 -- 172.21.15.70:0/784483162 wait complete. 2024-04-22T01:15:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:42.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:42.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.869+0000 7f637d472640 1 Processor -- start 2024-04-22T01:15:42.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f637d472640 1 -- start start 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f637d472640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780b4450 0x7f6378071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f637d472640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f63780720b0 0x7f63780ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f637d472640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637815c290 con 0x7f63780cd050 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f637d472640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6378072510 con 0x7f63780720b0 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f637d472640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63780b3a50 con 0x7f63780b4450 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63777fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f63780cd050 0x7f637815f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59428/0 (socket says 172.21.15.70:59428) 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63777fe640 1 -- 172.21.15.70:0/494945530 learned_addr learned my addr 172.21.15.70:0/494945530 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63767fc640 1 --2- 172.21.15.70:0/494945530 >> v2:172.21.15.70:3300/0 conn(0x7f63780720b0 0x7f63780ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f6376ffd640 1 --2- 172.21.15.70:0/494945530 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780b4450 0x7f6378071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63767fc640 1 -- 172.21.15.70:0/494945530 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780b4450 msgr2=0x7f6378071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63767fc640 1 --2- 172.21.15.70:0/494945530 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780b4450 0x7f6378071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63767fc640 1 -- 172.21.15.70:0/494945530 >> v1:172.21.15.70:6789/0 conn(0x7f63780cd050 legacy=0x7f637815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63767fc640 1 -- 172.21.15.70:0/494945530 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637815c570 con 0x7f63780720b0 2024-04-22T01:15:42.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f6375ffb640 1 -- 172.21.15.70:0/494945530 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 885839096 0 0) 0x7f637815c290 con 0x7f63780cd050 2024-04-22T01:15:42.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f63767fc640 1 --2- 172.21.15.70:0/494945530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780720b0 0x7f63780ccb10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f63600097c0 tx=0x7f636001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af37e20d3ed6bfad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f6375ffb640 1 -- 172.21.15.70:0/494945530 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63600b2070 con 0x7f63780720b0 2024-04-22T01:15:42.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f6375ffb640 1 -- 172.21.15.70:0/494945530 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63600ad640 con 0x7f63780720b0 2024-04-22T01:15:42.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.873+0000 7f6375ffb640 1 -- 172.21.15.70:0/494945530 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63600ada10 con 0x7f63780720b0 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/494945530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780720b0 msgr2=0x7f63780ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 --2- 172.21.15.70:0/494945530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780720b0 0x7f63780ccb10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f63600097c0 tx=0x7f636001c470 comp rx=0 tx=0).stop 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/494945530 shutdown_connections 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 --2- 172.21.15.70:0/494945530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780720b0 0x7f63780ccb10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 --2- 172.21.15.70:0/494945530 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780b4450 0x7f6378071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/494945530 >> 172.21.15.70:0/494945530 conn(0x7f63780b1d40 msgr2=0x7f63780cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/494945530 shutdown_connections 2024-04-22T01:15:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/494945530 wait complete. 2024-04-22T01:15:42.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 Processor -- start 2024-04-22T01:15:42.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- start start 2024-04-22T01:15:42.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780720b0 0x7f63781510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 0x7f63781575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63780cd050 0x7f637814a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63780b4830 con 0x7f63780b4450 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63780cd480 con 0x7f63780cd050 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63780b49b0 con 0x7f63780720b0 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 0x7f63781575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 0x7f63781575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40156/0 (socket says 172.21.15.70:40156) 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 -- 172.21.15.70:0/1952140306 learned_addr learned my addr 172.21.15.70:0/1952140306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f6376ffd640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780720b0 0x7f63781510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 -- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780720b0 msgr2=0x7f63781510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63777fe640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63780cd050 0x7f637814a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780720b0 0x7f63781510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 -- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63780cd050 msgr2=0x7f637814a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63780cd050 0x7f637814a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 -- 172.21.15.70:0/1952140306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637814c560 con 0x7f63780b4450 2024-04-22T01:15:42.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f63767fc640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 0x7f63781575c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63600b0040 tx=0x7f636001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63600b2050 con 0x7f63780b4450 2024-04-22T01:15:42.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63600b3040 con 0x7f63780b4450 2024-04-22T01:15:42.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6378151750 con 0x7f63780b4450 2024-04-22T01:15:42.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360004220 con 0x7f63780b4450 2024-04-22T01:15:42.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.877+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6378151d10 con 0x7f63780b4450 2024-04-22T01:15:42.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.881+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6378068210 con 0x7f63780b4450 2024-04-22T01:15:42.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.881+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f63600c0060 con 0x7f63780b4450 2024-04-22T01:15:42.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.881+0000 7f636f7fe640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63580678c0 0x7f6358069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:42.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.881+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f636011fd60 con 0x7f63780b4450 2024-04-22T01:15:42.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.881+0000 7f6376ffd640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63580678c0 0x7f6358069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:42.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.881+0000 7f6376ffd640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63580678c0 0x7f6358069d80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f636800b9d0 tx=0x7f6368008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:42.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:42.885+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6360016090 con 0x7f63780b4450 2024-04-22T01:15:43.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.045+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f63780d01e0 con 0x7f63780b4450 2024-04-22T01:15:43.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.045+0000 7f636f7fe640 1 -- 172.21.15.70:0/1952140306 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f63600f0530 con 0x7f63780b4450 2024-04-22T01:15:43.047 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:43.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.045+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63580678c0 msgr2=0x7f6358069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63580678c0 0x7f6358069d80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f636800b9d0 tx=0x7f6368008040 comp rx=0 tx=0).stop 2024-04-22T01:15:43.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 msgr2=0x7f63781575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 0x7f63781575c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63600b0040 tx=0x7f636001acb0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 shutdown_connections 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f63580678c0 0x7f6358069d80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63780cd050 0x7f637814a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63780b4450 0x7f63781575c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 --2- 172.21.15.70:0/1952140306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63780720b0 0x7f63781510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 >> 172.21.15.70:0/1952140306 conn(0x7f63780b1d40 msgr2=0x7f63780b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 shutdown_connections 2024-04-22T01:15:43.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.049+0000 7f637d472640 1 -- 172.21.15.70:0/1952140306 wait complete. 2024-04-22T01:15:43.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 Processor -- start 2024-04-22T01:15:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 -- start start 2024-04-22T01:15:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c0c06b1f0 0x7f7c0c06b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 0x7f7c0c0d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c06bb10 con 0x7f7c0c15c480 2024-04-22T01:15:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0c0b0520 con 0x7f7c0c06b1f0 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c11198640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c0b0800 con 0x7f7c0c0d08c0 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7c0c15c480 0x7f7c0c15c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59444/0 (socket says 172.21.15.70:59444) 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0a59c640 1 -- 172.21.15.70:0/1353880640 learned_addr learned my addr 172.21.15.70:0/1353880640 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1353880640 >> v2:172.21.15.98:3300/0 conn(0x7f7c0c06b1f0 0x7f7c0c06b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0b59e640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 0x7f7c0c0d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728811895 0 0) 0x7f7c0c06bb10 con 0x7f7c0c15c480 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bf4003530 con 0x7f7c0c15c480 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3337386883 0 0) 0x7f7bf4003530 con 0x7f7c0c15c480 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 msgr2=0x7f7c0c0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 0x7f7c0c0d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 >> v2:172.21.15.98:3300/0 conn(0x7f7c0c06b1f0 msgr2=0x7f7c0c06b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 --2- 172.21.15.70:0/1353880640 >> v2:172.21.15.98:3300/0 conn(0x7f7c0c06b1f0 0x7f7c0c06b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c0b0ae0 con 0x7f7c0c15c480 2024-04-22T01:15:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7c00001680 con 0x7f7c0c15c480 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 >> v1:172.21.15.70:6789/0 conn(0x7f7c0c15c480 legacy=0x7f7c0c15c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bf40039c0 0x7f7bf4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4004810 0x7f7bf4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 0x7f7bf40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0b0520 con 0x7f7bf40039c0 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0b0800 con 0x7f7bf4004810 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf400b9d0 con 0x7f7c0c0d08c0 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bf40039c0 0x7f7bf4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 -- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 msgr2=0x7f7bf40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c0d08c0 0x7f7bf40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 -- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4004810 msgr2=0x7f7bf4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4004810 0x7f7bf4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.189+0000 7f7c0ad9d640 1 -- 172.21.15.70:0/1353880640 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf4003530 con 0x7f7bf40039c0 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bf40039c0 0x7f7bf4003de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009ae0 tx=0x7f7bfc01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b2754d819d3be5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0096b0 con 0x7f7bf40039c0 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bfc0a8410 con 0x7f7bf40039c0 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c09d9b640 1 -- 172.21.15.70:0/1353880640 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0a8800 con 0x7f7bf40039c0 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 -- 172.21.15.70:0/1353880640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bf40039c0 msgr2=0x7f7bf4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 --2- 172.21.15.70:0/1353880640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bf40039c0 0x7f7bf4003de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009ae0 tx=0x7f7bfc01cab0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c0b59e640 1 -- 172.21.15.70:0/1353880640 reap_dead start 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 -- 172.21.15.70:0/1353880640 shutdown_connections 2024-04-22T01:15:43.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 -- 172.21.15.70:0/1353880640 >> 172.21.15.70:0/1353880640 conn(0x7f7c0c099a60 msgr2=0x7f7c0c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 -- 172.21.15.70:0/1353880640 shutdown_connections 2024-04-22T01:15:43.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 -- 172.21.15.70:0/1353880640 wait complete. 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 Processor -- start 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.193+0000 7f7c11198640 1 -- start start 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf40039c0 0x7f7c0c0c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 0x7f7c0c0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c153ac0 0x7f7c0c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0d1660 con 0x7f7c0c1583d0 2024-04-22T01:15:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0d14e0 con 0x7f7bf40039c0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c0d17e0 con 0x7f7c0c153ac0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 0x7f7c0c0c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 0x7f7c0c0c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40176/0 (socket says 172.21.15.70:40176) 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 -- 172.21.15.70:0/1957604303 learned_addr learned my addr 172.21.15.70:0/1957604303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0b59e640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c153ac0 0x7f7c0c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf40039c0 0x7f7c0c0c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 -- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c153ac0 msgr2=0x7f7c0c153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c153ac0 0x7f7c0c153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 -- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf40039c0 msgr2=0x7f7c0c0c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf40039c0 0x7f7c0c0c4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 -- 172.21.15.70:0/1957604303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c15a5b0 con 0x7f7c0c1583d0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0a59c640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 0x7f7c0c0c4b50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c800 tx=0x7f7c0000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00018070 con 0x7f7c0c1583d0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c00007500 con 0x7f7c0c1583d0 2024-04-22T01:15:43.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c1f6150 con 0x7f7c0c1583d0 2024-04-22T01:15:43.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00004a30 con 0x7f7c0c1583d0 2024-04-22T01:15:43.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c154480 con 0x7f7c0c1583d0 2024-04-22T01:15:43.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c0c068920 con 0x7f7c0c1583d0 2024-04-22T01:15:43.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c0001b070 con 0x7f7c0c1583d0 2024-04-22T01:15:43.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7bf37fe640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7bf80678f0 0x7f7bf8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7c00085620 con 0x7f7c0c1583d0 2024-04-22T01:15:43.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7bf80678f0 0x7f7bf8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.197+0000 7f7c0ad9d640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7bf80678f0 0x7f7bf8069db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0af000 tx=0x7f7bfc0aa040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.201+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7c00055df0 con 0x7f7c0c1583d0 2024-04-22T01:15:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.365+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7c0c06c0a0 con 0x7f7c0c1583d0 2024-04-22T01:15:43.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.365+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1957604303 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f7c000556f0 con 0x7f7c0c1583d0 2024-04-22T01:15:43.368 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:43.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7bf80678f0 msgr2=0x7f7bf8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7bf80678f0 0x7f7bf8069db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0af000 tx=0x7f7bfc0aa040 comp rx=0 tx=0).stop 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 msgr2=0x7f7c0c0c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 0x7f7c0c0c4b50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c800 tx=0x7f7c0000ccc0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 shutdown_connections 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7bf80678f0 0x7f7bf8069db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c0c153ac0 0x7f7c0c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c1583d0 0x7f7c0c0c4b50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 --2- 172.21.15.70:0/1957604303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf40039c0 0x7f7c0c0c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 >> 172.21.15.70:0/1957604303 conn(0x7f7c0c099a60 msgr2=0x7f7c0c15b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 shutdown_connections 2024-04-22T01:15:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.369+0000 7f7c11198640 1 -- 172.21.15.70:0/1957604303 wait complete. 2024-04-22T01:15:43.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:43.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.509+0000 7fc110445640 1 Processor -- start 2024-04-22T01:15:43.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.509+0000 7fc110445640 1 -- start start 2024-04-22T01:15:43.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc110445640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 0x7fc1080ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc110445640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10815c290 con 0x7fc1080cd050 2024-04-22T01:15:43.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc110445640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10815c570 con 0x7fc1080b4450 2024-04-22T01:15:43.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc110445640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc10815c850 con 0x7fc108072280 2024-04-22T01:15:43.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10e9e2640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc1080cd050 0x7fc1080b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59450/0 (socket says 172.21.15.70:59450) 2024-04-22T01:15:43.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10e9e2640 1 -- 172.21.15.70:0/3878670067 learned_addr learned my addr 172.21.15.70:0/3878670067 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 0x7fc1080ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776346376 0 0) 0x7fc10815c290 con 0x7fc1080cd050 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0f4003530 con 0x7fc1080cd050 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1948218546 0 0) 0x7fc10815c570 con 0x7fc1080b4450 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc10815c290 con 0x7fc1080b4450 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3643127778 0 0) 0x7fc0f4003530 con 0x7fc1080cd050 2024-04-22T01:15:43.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 msgr2=0x7fc1080ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 0x7fc1080ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 >> v1:172.21.15.98:6789/0 conn(0x7fc1080b4450 legacy=0x7fc108071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10815cb30 con 0x7fc1080cd050 2024-04-22T01:15:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fc100001a00 con 0x7fc1080cd050 2024-04-22T01:15:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 >> v1:172.21.15.70:6789/0 conn(0x7fc1080cd050 legacy=0x7fc1080b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 0x7fc0f4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f4004130 0x7fc0f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0f4004d50 0x7fc0f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc10815c850 con 0x7fc0f4004130 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f400bb10 con 0x7fc0f4004d50 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f400bd10 con 0x7fc108072280 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f4004130 0x7fc0f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10e9e2640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0f4004d50 0x7fc0f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10e1e1640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 0x7fc0f4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 -- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 msgr2=0x7fc0f4003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc108072280 0x7fc0f4003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 -- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0f4004d50 msgr2=0x7fc0f40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0f4004d50 0x7fc0f40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.513+0000 7fc10d9e0640 1 -- 172.21.15.70:0/3878670067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f4003530 con 0x7fc0f4004130 2024-04-22T01:15:43.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10d9e0640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f4004130 0x7fc0f4004810 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc104009ae0 tx=0x7fc10401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709753a27b8c1035 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1040b2070 con 0x7fc0f4004130 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1040a55a0 con 0x7fc0f4004130 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10d1df640 1 -- 172.21.15.70:0/3878670067 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1040096b0 con 0x7fc0f4004130 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- 172.21.15.70:0/3878670067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f4004130 msgr2=0x7fc0f4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 --2- 172.21.15.70:0/3878670067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f4004130 0x7fc0f4004810 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc104009ae0 tx=0x7fc10401c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 -- 172.21.15.70:0/3878670067 reap_dead start 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- 172.21.15.70:0/3878670067 shutdown_connections 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- 172.21.15.70:0/3878670067 >> 172.21.15.70:0/3878670067 conn(0x7fc1080b1d40 msgr2=0x7fc1080cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- 172.21.15.70:0/3878670067 shutdown_connections 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- 172.21.15.70:0/3878670067 wait complete. 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 Processor -- start 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- start start 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f4004130 0x7fc1081492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1081f5050 0x7fc1080affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 0x7fc1080b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080b4830 con 0x7fc1081f54d0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080cd4a0 con 0x7fc1081f5050 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc110445640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc1080b49b0 con 0x7fc0f4004130 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 0x7fc1080b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 0x7fc1080b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40200/0 (socket says 172.21.15.70:40200) 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 -- 172.21.15.70:0/2999927601 learned_addr learned my addr 172.21.15.70:0/2999927601 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10d9e0640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1081f5050 0x7fc1080affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e1e1640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f4004130 0x7fc1081492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 -- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f4004130 msgr2=0x7fc1081492e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f4004130 0x7fc1081492e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 -- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1081f5050 msgr2=0x7fc1080affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1081f5050 0x7fc1080affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 -- 172.21.15.70:0/2999927601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1081f5aa0 con 0x7fc1081f54d0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.517+0000 7fc10e9e2640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 0x7fc1080b04e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc1080b27c0 tx=0x7fc10000e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10000eb10 con 0x7fc1081f54d0 2024-04-22T01:15:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc100018070 con 0x7fc1081f54d0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1081f5d80 con 0x7fc1081f54d0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100013420 con 0x7fc1081f54d0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1081502e0 con 0x7fc1081f54d0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1081504e0 con 0x7fc1081f54d0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc1000135c0 con 0x7fc1081f54d0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc0feffd640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0ec0678f0 0x7fc0ec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc10e1e1640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0ec0678f0 0x7fc0ec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc100086370 con 0x7fc1081f54d0 2024-04-22T01:15:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.521+0000 7fc10e1e1640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0ec0678f0 0x7fc0ec069db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8007af0 tx=0x7fc0f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.525+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc1081504e0 con 0x7fc1081f54d0 2024-04-22T01:15:43.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.685+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc108068210 con 0x7fc1081f54d0 2024-04-22T01:15:43.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.685+0000 7fc0feffd640 1 -- 172.21.15.70:0/2999927601 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fc100056ab0 con 0x7fc1081f54d0 2024-04-22T01:15:43.687 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0ec0678f0 msgr2=0x7fc0ec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0ec0678f0 0x7fc0ec069db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8007af0 tx=0x7fc0f8009040 comp rx=0 tx=0).stop 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 msgr2=0x7fc1080b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 0x7fc1080b04e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc1080b27c0 tx=0x7fc10000e160 comp rx=0 tx=0).stop 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 shutdown_connections 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0ec0678f0 0x7fc0ec069db0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1081f54d0 0x7fc1080b04e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1081f5050 0x7fc1080affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 --2- 172.21.15.70:0/2999927601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f4004130 0x7fc1081492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 >> 172.21.15.70:0/2999927601 conn(0x7fc1080b1d40 msgr2=0x7fc1080744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 shutdown_connections 2024-04-22T01:15:43.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.689+0000 7fc110445640 1 -- 172.21.15.70:0/2999927601 wait complete. 2024-04-22T01:15:43.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:43.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8ffcd1640 1 Processor -- start 2024-04-22T01:15:43.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8ffcd1640 1 -- start start 2024-04-22T01:15:43.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8ffcd1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8f80cdd70 0x7fa8f806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8ffcd1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f806b860 con 0x7fa8f80b21a0 2024-04-22T01:15:43.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8ffcd1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f806bb40 con 0x7fa8f815df10 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8ffcd1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f8161a60 con 0x7fa8f80cdd70 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa8f80b21a0 0x7fa8f80cb130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59456/0 (socket says 172.21.15.70:59456) 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 -- 172.21.15.70:0/329763897 learned_addr learned my addr 172.21.15.70:0/329763897 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2080853444 0 0) 0x7fa8f806b860 con 0x7fa8f80b21a0 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e4003530 con 0x7fa8f80b21a0 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fe26e640 1 --2- 172.21.15.70:0/329763897 >> v2:172.21.15.98:3300/0 conn(0x7fa8f80cdd70 0x7fa8f806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1919064506 0 0) 0x7fa8e4003530 con 0x7fa8f80b21a0 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 >> v2:172.21.15.98:3300/0 conn(0x7fa8f80cdd70 msgr2=0x7fa8f806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 --2- 172.21.15.70:0/329763897 >> v2:172.21.15.98:3300/0 conn(0x7fa8f80cdd70 0x7fa8f806b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 >> v1:172.21.15.98:6789/0 conn(0x7fa8f815df10 legacy=0x7fa8f80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f8161d40 con 0x7fa8f80b21a0 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fa8e8001680 con 0x7fa8f80b21a0 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 >> v1:172.21.15.70:6789/0 conn(0x7fa8f80b21a0 legacy=0x7fa8f80cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e40039c0 0x7fa8e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8e4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e40051d0 0x7fa8e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f8161a60 con 0x7fa8e4004810 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e400c170 con 0x7fa8e40039c0 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e400c370 con 0x7fa8e40051d0 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8e4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fda6d640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e40039c0 0x7fa8e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fe26e640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e40051d0 0x7fa8e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 -- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e40051d0 msgr2=0x7fa8e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e40051d0 0x7fa8e400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 -- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e40039c0 msgr2=0x7fa8e4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e40039c0 0x7fa8e4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fd26c640 1 -- 172.21.15.70:0/329763897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e4003530 con 0x7fa8e4004810 2024-04-22T01:15:43.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.825+0000 7fa8fe26e640 1 -- 172.21.15.70:0/329763897 reap_dead start 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8e4004c90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800c240 tx=0x7fa8e801cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d83d7b95f8b5ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8007d20 con 0x7fa8e4004810 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e80a9070 con 0x7fa8e4004810 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fca6b640 1 -- 172.21.15.70:0/329763897 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8005da0 con 0x7fa8e4004810 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/329763897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 msgr2=0x7fa8e4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8e4004c90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800c240 tx=0x7fa8e801cbd0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/329763897 shutdown_connections 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/329763897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8e4004c90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/329763897 >> 172.21.15.70:0/329763897 conn(0x7fa8f8099a60 msgr2=0x7fa8f8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/329763897 shutdown_connections 2024-04-22T01:15:43.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/329763897 wait complete. 2024-04-22T01:15:43.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 Processor -- start 2024-04-22T01:15:43.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- start start 2024-04-22T01:15:43.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8f8159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8f80c5020 0x7fa8f81fac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8f81fb1b0 0x7fa8f81fb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f80cb940 con 0x7fa8e4004810 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f8162950 con 0x7fa8f80c5020 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8ffcd1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f80cbac0 con 0x7fa8f81fb1b0 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fda6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8f8159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fda6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8f8159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40208/0 (socket says 172.21.15.70:40208) 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.829+0000 7fa8fda6d640 1 -- 172.21.15.70:0/182367596 learned_addr learned my addr 172.21.15.70:0/182367596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8f80c5020 0x7fa8f81fac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fe26e640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8f81fb1b0 0x7fa8f81fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fda6d640 1 -- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8f81fb1b0 msgr2=0x7fa8f81fb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fda6d640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8f81fb1b0 0x7fa8f81fb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fda6d640 1 -- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8f80c5020 msgr2=0x7fa8f81fac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fda6d640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8f80c5020 0x7fa8f81fac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fda6d640 1 -- 172.21.15.70:0/182367596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f81fbb70 con 0x7fa8e4004810 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fda6d640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8f8159b00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e480 tx=0x7fa8f400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f40052e0 con 0x7fa8e4004810 2024-04-22T01:15:43.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f4018070 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4003840 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f815a610 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f80c9f90 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f8068920 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8f40039e0 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ee7fc640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8dc0678c0 0x7fa8dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8dc0678c0 0x7fa8dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa8f4085cf0 con 0x7fa8e4004810 2024-04-22T01:15:43.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.833+0000 7fa8fd26c640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8dc0678c0 0x7fa8dc069d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80ab000 tx=0x7fa8e80b1040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:43.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.837+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa8f408a070 con 0x7fa8e4004810 2024-04-22T01:15:43.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.985+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa8f815d6d0 con 0x7fa8e4004810 2024-04-22T01:15:43.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ee7fc640 1 -- 172.21.15.70:0/182367596 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fa8f40564c0 con 0x7fa8e4004810 2024-04-22T01:15:43.990 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:43.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8dc0678c0 msgr2=0x7fa8dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8dc0678c0 0x7fa8dc069d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80ab000 tx=0x7fa8e80b1040 comp rx=0 tx=0).stop 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 msgr2=0x7fa8f8159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8f8159b00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e480 tx=0x7fa8f400e940 comp rx=0 tx=0).stop 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 shutdown_connections 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8dc0678c0 0x7fa8dc069d80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8f81fb1b0 0x7fa8f81fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8f80c5020 0x7fa8f81fac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 --2- 172.21.15.70:0/182367596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e4004810 0x7fa8f8159b00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 >> 172.21.15.70:0/182367596 conn(0x7fa8f8099a60 msgr2=0x7fa8f80cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 shutdown_connections 2024-04-22T01:15:43.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:43.989+0000 7fa8ffcd1640 1 -- 172.21.15.70:0/182367596 wait complete. 2024-04-22T01:15:44.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:44.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 Processor -- start 2024-04-22T01:15:44.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 -- start start 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2358161b90 0x7f2358162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23580cecd0 0x7f23580cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235806b450 con 0x7f23581615a0 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2358162550 con 0x7f2358161b90 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235f500640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2358161980 con 0x7f23580cecd0 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2358161b90 0x7f2358162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2358161b90 0x7f2358162010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40214/0 (socket says 172.21.15.70:40214) 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 -- 172.21.15.70:0/1830441879 learned_addr learned my addr 172.21.15.70:0/1830441879 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 -- 172.21.15.70:0/1830441879 >> v2:172.21.15.98:3300/0 conn(0x7f23580cecd0 msgr2=0x7f23580cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 --2- 172.21.15.70:0/1830441879 >> v2:172.21.15.98:3300/0 conn(0x7f23580cecd0 0x7f23580cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 -- 172.21.15.70:0/1830441879 >> v1:172.21.15.98:6789/0 conn(0x7f23581615a0 legacy=0x7f23580b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:44.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 -- 172.21.15.70:0/1830441879 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23580b77d0 con 0x7f2358161b90 2024-04-22T01:15:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.121+0000 7f235ca9b640 1 --2- 172.21.15.70:0/1830441879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358161b90 0x7f2358162010 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f23580d2410 tx=0x7f234801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb321220bbc3f457 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f234ffff640 1 -- 172.21.15.70:0/1830441879 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23480b0070 con 0x7f2358161b90 2024-04-22T01:15:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f234ffff640 1 -- 172.21.15.70:0/1830441879 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23480ab600 con 0x7f2358161b90 2024-04-22T01:15:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f234ffff640 1 -- 172.21.15.70:0/1830441879 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23480ab9d0 con 0x7f2358161b90 2024-04-22T01:15:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/1830441879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358161b90 msgr2=0x7f2358162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 --2- 172.21.15.70:0/1830441879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358161b90 0x7f2358162010 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f23580d2410 tx=0x7f234801c4e0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/1830441879 shutdown_connections 2024-04-22T01:15:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 --2- 172.21.15.70:0/1830441879 >> v2:172.21.15.98:3300/0 conn(0x7f23580cecd0 0x7f23580cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 --2- 172.21.15.70:0/1830441879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358161b90 0x7f2358162010 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/1830441879 >> 172.21.15.70:0/1830441879 conn(0x7f235806c910 msgr2=0x7f235806ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/1830441879 shutdown_connections 2024-04-22T01:15:44.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/1830441879 wait complete. 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 Processor -- start 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- start start 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 0x7f23581551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581615a0 0x7f23580b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b57b0 0x7f23580b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23580d2130 con 0x7f23580cecd0 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23580ce830 con 0x7f23581615a0 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f23580d3140 con 0x7f23580b57b0 2024-04-22T01:15:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 0x7f23581551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 0x7f23581551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40224/0 (socket says 172.21.15.70:40224) 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 -- 172.21.15.70:0/553901563 learned_addr learned my addr 172.21.15.70:0/553901563 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 -- 172.21.15.70:0/553901563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b57b0 msgr2=0x7f23580b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b57b0 0x7f23580b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 -- 172.21.15.70:0/553901563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581615a0 msgr2=0x7f23580b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581615a0 0x7f23580b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 -- 172.21.15.70:0/553901563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23580ceb00 con 0x7f23580cecd0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235d29c640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 0x7f23581551c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f235400b3e0 tx=0x7f235400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354015020 con 0x7f23580cecd0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2354011850 con 0x7f23580cecd0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23580b41d0 con 0x7f23580cecd0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354011c40 con 0x7f23580cecd0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.125+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23580b4790 con 0x7f23580cecd0 2024-04-22T01:15:44.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.129+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235804f510 con 0x7f23580cecd0 2024-04-22T01:15:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.129+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2354018020 con 0x7f23580cecd0 2024-04-22T01:15:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.129+0000 7f234dffb640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f233c0678f0 0x7f233c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.129+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2354085f90 con 0x7f23580cecd0 2024-04-22T01:15:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.129+0000 7f235ca9b640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f233c0678f0 0x7f233c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.129+0000 7f235ca9b640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f233c0678f0 0x7f233c069db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23480ad000 tx=0x7f2348008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.133+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f235401f020 con 0x7f23580cecd0 2024-04-22T01:15:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.301+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f23580747c0 con 0x7f23580cecd0 2024-04-22T01:15:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.301+0000 7f234dffb640 1 -- 172.21.15.70:0/553901563 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f23540172c0 con 0x7f23580cecd0 2024-04-22T01:15:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f233c0678f0 msgr2=0x7f233c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f233c0678f0 0x7f233c069db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23480ad000 tx=0x7f2348008040 comp rx=0 tx=0).stop 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 msgr2=0x7f23581551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 0x7f23581551c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f235400b3e0 tx=0x7f235400b8a0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 shutdown_connections 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f233c0678f0 0x7f233c069db0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b57b0 0x7f23580b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581615a0 0x7f23580b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 --2- 172.21.15.70:0/553901563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580cecd0 0x7f23581551c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 >> 172.21.15.70:0/553901563 conn(0x7f235806c910 msgr2=0x7f2358159f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 shutdown_connections 2024-04-22T01:15:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.305+0000 7f235f500640 1 -- 172.21.15.70:0/553901563 wait complete. 2024-04-22T01:15:44.323 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:15:44.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:44.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:15:44.335 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:15:44.335 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:15:44.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:44.335 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:15:44.382 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:15:44.382 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:44.388 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:15:44.389 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:44.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:44.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.553+0000 7fe07fa76640 1 Processor -- start 2024-04-22T01:15:44.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.553+0000 7fe07fa76640 1 -- start start 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07fa76640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe0780cc150 0x7fe078071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07fa76640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0780cd7b0 con 0x7fe0781455b0 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07fa76640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0780cc710 con 0x7fe078072060 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07fa76640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe078072490 con 0x7fe0780cc150 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d812640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe0781455b0 0x7fe0780cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59462/0 (socket says 172.21.15.70:59462) 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d812640 1 -- 172.21.15.70:0/3842576166 learned_addr learned my addr 172.21.15.70:0/3842576166 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d011640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe0780cc150 0x7fe078071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d011640 1 -- 172.21.15.70:0/3842576166 >> v1:172.21.15.98:6789/0 conn(0x7fe078072060 legacy=0x7fe07806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d011640 1 -- 172.21.15.70:0/3842576166 >> v1:172.21.15.70:6789/0 conn(0x7fe0781455b0 legacy=0x7fe0780cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d011640 1 -- 172.21.15.70:0/3842576166 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0780cc9f0 con 0x7fe0780cc150 2024-04-22T01:15:44.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07c810640 1 -- 172.21.15.70:0/3842576166 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1829095312 0 0) 0x7fe0780cd7b0 con 0x7fe0781455b0 2024-04-22T01:15:44.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07d011640 1 --2- 172.21.15.70:0/3842576166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0780cc150 0x7fe078071b20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe0680097c0 tx=0x7fe06801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1f8d9810955db4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07c810640 1 -- 172.21.15.70:0/3842576166 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0680b0070 con 0x7fe0780cc150 2024-04-22T01:15:44.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07c810640 1 -- 172.21.15.70:0/3842576166 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0680ab600 con 0x7fe0780cc150 2024-04-22T01:15:44.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07c810640 1 -- 172.21.15.70:0/3842576166 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0680ab9d0 con 0x7fe0780cc150 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07fa76640 1 -- 172.21.15.70:0/3842576166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0780cc150 msgr2=0x7fe078071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.557+0000 7fe07fa76640 1 --2- 172.21.15.70:0/3842576166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0780cc150 0x7fe078071b20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe0680097c0 tx=0x7fe06801c340 comp rx=0 tx=0).stop 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/3842576166 shutdown_connections 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 --2- 172.21.15.70:0/3842576166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0780cc150 0x7fe078071b20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/3842576166 >> 172.21.15.70:0/3842576166 conn(0x7fe07806b1f0 msgr2=0x7fe07806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/3842576166 shutdown_connections 2024-04-22T01:15:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/3842576166 wait complete. 2024-04-22T01:15:44.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 Processor -- start 2024-04-22T01:15:44.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- start start 2024-04-22T01:15:44.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe078072060 0x7fe078159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0780cc150 0x7fe07815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 0x7fe07815e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe078169ee0 con 0x7fe0781455b0 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe078169d60 con 0x7fe078072060 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe07816a060 con 0x7fe0780cc150 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 0x7fe07815e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 0x7fe07815e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40250/0 (socket says 172.21.15.70:40250) 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 -- 172.21.15.70:0/1889142073 learned_addr learned my addr 172.21.15.70:0/1889142073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 -- 172.21.15.70:0/1889142073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0780cc150 msgr2=0x7fe07815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0780cc150 0x7fe07815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 -- 172.21.15.70:0/1889142073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe078072060 msgr2=0x7fe078159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe078072060 0x7fe078159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 -- 172.21.15.70:0/1889142073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07813e1c0 con 0x7fe0781455b0 2024-04-22T01:15:44.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07e013640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 0x7fe07815e900 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe07000b990 tx=0x7fe07000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070019070 con 0x7fe0781455b0 2024-04-22T01:15:44.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe07000cba0 con 0x7fe0781455b0 2024-04-22T01:15:44.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070014440 con 0x7fe0781455b0 2024-04-22T01:15:44.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07813e4a0 con 0x7fe0781455b0 2024-04-22T01:15:44.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe07813ea60 con 0x7fe0781455b0 2024-04-22T01:15:44.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.561+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe078068210 con 0x7fe0781455b0 2024-04-22T01:15:44.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.565+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe07001c070 con 0x7fe0781455b0 2024-04-22T01:15:44.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.565+0000 7fe06e7fc640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe05c0678c0 0x7fe05c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.565+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe070085dd0 con 0x7fe0781455b0 2024-04-22T01:15:44.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.565+0000 7fe07d812640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe05c0678c0 0x7fe05c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.565+0000 7fe07d812640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe05c0678c0 0x7fe05c069d80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0740015a0 tx=0x7fe074007970 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.569+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe070056500 con 0x7fe0781455b0 2024-04-22T01:15:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.729+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe07815f510 con 0x7fe0781455b0 2024-04-22T01:15:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe06e7fc640 1 -- 172.21.15.70:0/1889142073 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7fe070055e00 con 0x7fe0781455b0 2024-04-22T01:15:44.734 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:44.734 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:44.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe05c0678c0 msgr2=0x7fe05c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe05c0678c0 0x7fe05c069d80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0740015a0 tx=0x7fe074007970 comp rx=0 tx=0).stop 2024-04-22T01:15:44.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 msgr2=0x7fe07815e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 0x7fe07815e900 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe07000b990 tx=0x7fe07000be50 comp rx=0 tx=0).stop 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 shutdown_connections 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe05c0678c0 0x7fe05c069d80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0781455b0 0x7fe07815e900 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0780cc150 0x7fe07815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 --2- 172.21.15.70:0/1889142073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe078072060 0x7fe078159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 >> 172.21.15.70:0/1889142073 conn(0x7fe07806b1f0 msgr2=0x7fe078142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 shutdown_connections 2024-04-22T01:15:44.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.733+0000 7fe07fa76640 1 -- 172.21.15.70:0/1889142073 wait complete. 2024-04-22T01:15:44.753 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:44.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.864+0000 7fec3b512640 1 Processor -- start 2024-04-22T01:15:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.864+0000 7fec3b512640 1 -- start start 2024-04-22T01:15:44.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340b4450 0x7fec34071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fec340cd220 0x7fec340b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec3415c290 con 0x7fec340720b0 2024-04-22T01:15:44.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fec34072510 con 0x7fec340cd220 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fec3415c570 con 0x7fec340b4450 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec38aad640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fec340720b0 0x7fec340ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59466/0 (socket says 172.21.15.70:59466) 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec39aaf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fec340cd220 0x7fec340b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec38aad640 1 -- 172.21.15.70:0/1818581763 learned_addr learned my addr 172.21.15.70:0/1818581763 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec39aaf640 1 -- 172.21.15.70:0/1818581763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340b4450 msgr2=0x7fec34071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec392ae640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340b4450 0x7fec34071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec2bfff640 1 -- 172.21.15.70:0/1818581763 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288097681 0 0) 0x7fec3415c290 con 0x7fec340720b0 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec39aaf640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340b4450 0x7fec34071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec39aaf640 1 -- 172.21.15.70:0/1818581763 >> v1:172.21.15.70:6789/0 conn(0x7fec340720b0 legacy=0x7fec340ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec39aaf640 1 -- 172.21.15.70:0/1818581763 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3415c850 con 0x7fec340cd220 2024-04-22T01:15:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec392ae640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340b4450 0x7fec34071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:44.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec39aaf640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340cd220 0x7fec340b2a40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009d90 tx=0x7fec2c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66e8fda2f95e184 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec2bfff640 1 -- 172.21.15.70:0/1818581763 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0b2070 con 0x7fec340cd220 2024-04-22T01:15:44.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec2bfff640 1 -- 172.21.15.70:0/1818581763 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec2c0a5990 con 0x7fec340cd220 2024-04-22T01:15:44.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec2bfff640 1 -- 172.21.15.70:0/1818581763 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0a5d60 con 0x7fec340cd220 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 -- 172.21.15.70:0/1818581763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340cd220 msgr2=0x7fec340b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.868+0000 7fec3b512640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340cd220 0x7fec340b2a40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009d90 tx=0x7fec2c01c940 comp rx=0 tx=0).stop 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1818581763 shutdown_connections 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340cd220 0x7fec340b2a40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 --2- 172.21.15.70:0/1818581763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340b4450 0x7fec34071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1818581763 >> 172.21.15.70:0/1818581763 conn(0x7fec340b1d40 msgr2=0x7fec340cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:44.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1818581763 shutdown_connections 2024-04-22T01:15:44.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1818581763 wait complete. 2024-04-22T01:15:44.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 Processor -- start 2024-04-22T01:15:44.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- start start 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 0x7fec3414bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec340b4450 0x7fec34152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340cd220 0x7fec34154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fec340cd980 con 0x7fec340720b0 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec340cd800 con 0x7fec340b4450 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fec340b3790 con 0x7fec340cd220 2024-04-22T01:15:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 0x7fec3414bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 0x7fec3414bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40278/0 (socket says 172.21.15.70:40278) 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 -- 172.21.15.70:0/1412975560 learned_addr learned my addr 172.21.15.70:0/1412975560 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 -- 172.21.15.70:0/1412975560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340cd220 msgr2=0x7fec34154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340cd220 0x7fec34154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 -- 172.21.15.70:0/1412975560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec340b4450 msgr2=0x7fec34152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec340b4450 0x7fec34152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 -- 172.21.15.70:0/1412975560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec34151270 con 0x7fec340720b0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec392ae640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 0x7fec3414bad0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c7f0 tx=0x7fec3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30007a30 con 0x7fec340720b0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec30007bd0 con 0x7fec340720b0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec34151550 con 0x7fec340720b0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec300113f0 con 0x7fec340720b0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec34151b10 con 0x7fec340720b0 2024-04-22T01:15:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.872+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec34068210 con 0x7fec340720b0 2024-04-22T01:15:44.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.876+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fec30011590 con 0x7fec340720b0 2024-04-22T01:15:44.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.876+0000 7fec29ffb640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fec180678c0 0x7fec18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:44.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.876+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fec300862a0 con 0x7fec340720b0 2024-04-22T01:15:44.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.876+0000 7fec38aad640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fec180678c0 0x7fec18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:44.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.876+0000 7fec38aad640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fec180678c0 0x7fec18069d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fec24000c10 tx=0x7fec24007910 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:44.880+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fec30015020 con 0x7fec340720b0 2024-04-22T01:15:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.028+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec340d01e0 con 0x7fec340720b0 2024-04-22T01:15:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.028+0000 7fec29ffb640 1 -- 172.21.15.70:0/1412975560 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fec300569d0 con 0x7fec340720b0 2024-04-22T01:15:45.031 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fec180678c0 msgr2=0x7fec18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fec180678c0 0x7fec18069d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fec24000c10 tx=0x7fec24007910 comp rx=0 tx=0).stop 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 msgr2=0x7fec3414bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 0x7fec3414bad0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c7f0 tx=0x7fec3000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 shutdown_connections 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fec180678c0 0x7fec18069d80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec340cd220 0x7fec34154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec340b4450 0x7fec34152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 --2- 172.21.15.70:0/1412975560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec340720b0 0x7fec3414bad0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 >> 172.21.15.70:0/1412975560 conn(0x7fec340b1d40 msgr2=0x7fec3415ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 shutdown_connections 2024-04-22T01:15:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.032+0000 7fec3b512640 1 -- 172.21.15.70:0/1412975560 wait complete. 2024-04-22T01:15:45.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:45.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.156+0000 7f592e640640 1 Processor -- start 2024-04-22T01:15:45.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.156+0000 7f592e640640 1 -- start start 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592e640640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59280d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592e640640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59280b9ce0 con 0x7f59280cea10 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592e640640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59280b75e0 con 0x7f592815de20 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592e640640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f59280b78c0 con 0x7f59280b98a0 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f59280cea10 0x7f59280ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59478/0 (socket says 172.21.15.70:59478) 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 -- 172.21.15.70:0/599719862 learned_addr learned my addr 172.21.15.70:0/599719862 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f59277fe640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59280d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 818338914 0 0) 0x7f59280b9ce0 con 0x7f59280cea10 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5910003530 con 0x7f59280cea10 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2733758164 0 0) 0x7f5910003530 con 0x7f59280cea10 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 msgr2=0x7f59280d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59280d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 >> v1:172.21.15.98:6789/0 conn(0x7f592815de20 legacy=0x7f59280b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59280b7ba0 con 0x7f59280cea10 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f5920001a00 con 0x7f59280cea10 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 >> v1:172.21.15.70:6789/0 conn(0x7f59280cea10 legacy=0x7f59280ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59100039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f70 0x7f5910004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5910004d50 0x7f59100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59280b78c0 con 0x7f5910004d50 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f591000bb40 con 0x7f5910003f70 2024-04-22T01:15:45.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f591000bd40 con 0x7f59280b98a0 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5910004d50 0x7f59100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f59277fe640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f70 0x7f5910004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5927fff640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59100039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 -- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 msgr2=0x7f59100039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59100039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 -- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f70 msgr2=0x7f5910004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f70 0x7f5910004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 -- 172.21.15.70:0/599719862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5910003530 con 0x7f5910004d50 2024-04-22T01:15:45.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5927fff640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59280b98a0 0x7f59100039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:45.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f592cbdd640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5910004d50 0x7f59100051d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f592000a840 tx=0x7f592001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4245f248f36ea34 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592000a460 con 0x7f5910004d50 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59200ad650 con 0x7f5910004d50 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.160+0000 7f5926ffd640 1 -- 172.21.15.70:0/599719862 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59200ada40 con 0x7f5910004d50 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/599719862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5910004d50 msgr2=0x7f59100051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 --2- 172.21.15.70:0/599719862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5910004d50 0x7f59100051d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f592000a840 tx=0x7f592001d360 comp rx=0 tx=0).stop 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 -- 172.21.15.70:0/599719862 reap_dead start 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/599719862 shutdown_connections 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/599719862 >> 172.21.15.70:0/599719862 conn(0x7f5928072d20 msgr2=0x7f5928073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/599719862 shutdown_connections 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/599719862 wait complete. 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 Processor -- start 2024-04-22T01:15:45.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- start start 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928158380 0x7f59280ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5928158760 0x7f59280cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 0x7f59280cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f592815e200 con 0x7f59280cbac0 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59280b8160 con 0x7f5928158380 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f59280d1d80 con 0x7f5928158760 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 0x7f59280cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 0x7f59280cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40300/0 (socket says 172.21.15.70:40300) 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 -- 172.21.15.70:0/90323644 learned_addr learned my addr 172.21.15.70:0/90323644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 -- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5928158760 msgr2=0x7f59280cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f5927fff640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928158380 0x7f59280ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f59277fe640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5928158760 0x7f59280cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5928158760 0x7f59280cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 -- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928158380 msgr2=0x7f59280ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928158380 0x7f59280ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 -- 172.21.15.70:0/90323644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59280cc830 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592cbdd640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 0x7f59280cbf50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f592000a840 tx=0x7f5910005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59200bb070 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5920001800 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59280c7a80 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920001210 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59280c8040 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.164+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59280c7870 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.168+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59200be070 con 0x7f59280cbac0 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.168+0000 7f5924ff9640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59140678c0 0x7f5914069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.168+0000 7f5927fff640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59140678c0 0x7f5914069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.168+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5920120140 con 0x7f59280cbac0 2024-04-22T01:15:45.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.168+0000 7f5927fff640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59140678c0 0x7f5914069d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f59280d44e0 tx=0x7f5918009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.172+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5920017070 con 0x7f59280cbac0 2024-04-22T01:15:45.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5928068210 con 0x7f59280cbac0 2024-04-22T01:15:45.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f5924ff9640 1 -- 172.21.15.70:0/90323644 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f59200f0910 con 0x7f59280cbac0 2024-04-22T01:15:45.334 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:45.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59140678c0 msgr2=0x7f5914069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59140678c0 0x7f5914069d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f59280d44e0 tx=0x7f5918009040 comp rx=0 tx=0).stop 2024-04-22T01:15:45.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 msgr2=0x7f59280cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 0x7f59280cbf50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f592000a840 tx=0x7f5910005840 comp rx=0 tx=0).stop 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 shutdown_connections 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59140678c0 0x7f5914069d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59280cbac0 0x7f59280cbf50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5928158760 0x7f59280cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 --2- 172.21.15.70:0/90323644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928158380 0x7f59280ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 >> 172.21.15.70:0/90323644 conn(0x7f5928072d20 msgr2=0x7f59280b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 shutdown_connections 2024-04-22T01:15:45.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.332+0000 7f592e640640 1 -- 172.21.15.70:0/90323644 wait complete. 2024-04-22T01:15:45.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:45.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.472+0000 7fd761e29640 1 Processor -- start 2024-04-22T01:15:45.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.472+0000 7fd761e29640 1 -- start start 2024-04-22T01:15:45.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd75c0cd050 0x7fd75c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd75c15c290 con 0x7fd75c0b4450 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd75c15c570 con 0x7fd75c0cd050 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd75c15c850 con 0x7fd75c072280 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd75c0cd050 0x7fd75c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd75c0b4450 0x7fd75c071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59488/0 (socket says 172.21.15.70:59488) 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75b7fe640 1 -- 172.21.15.70:0/601070957 learned_addr learned my addr 172.21.15.70:0/601070957 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75affd640 1 --2- 172.21.15.70:0/601070957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75a7fc640 1 -- 172.21.15.70:0/601070957 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375986604 0 0) 0x7fd75c15c290 con 0x7fd75c0b4450 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75a7fc640 1 -- 172.21.15.70:0/601070957 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd748003530 con 0x7fd75c0b4450 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75a7fc640 1 -- 172.21.15.70:0/601070957 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 81426398 0 0) 0x7fd748003530 con 0x7fd75c0b4450 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75bfff640 1 -- 172.21.15.70:0/601070957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 msgr2=0x7fd75c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75bfff640 1 --2- 172.21.15.70:0/601070957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75bfff640 1 -- 172.21.15.70:0/601070957 >> v1:172.21.15.70:6789/0 conn(0x7fd75c0b4450 legacy=0x7fd75c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75bfff640 1 -- 172.21.15.70:0/601070957 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75c15cb30 con 0x7fd75c0cd050 2024-04-22T01:15:45.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75bfff640 1 --2- 172.21.15.70:0/601070957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c0b2a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0097c0 tx=0x7fd74c01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d680a4afd59238f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75a7fc640 1 -- 172.21.15.70:0/601070957 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0b0070 con 0x7fd75c0cd050 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75a7fc640 1 -- 172.21.15.70:0/601070957 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd74c0ab400 con 0x7fd75c0cd050 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd75a7fc640 1 -- 172.21.15.70:0/601070957 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754001a00 con 0x7fd75c0cd050 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- 172.21.15.70:0/601070957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 msgr2=0x7fd75c0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 --2- 172.21.15.70:0/601070957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c0b2a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0097c0 tx=0x7fd74c01c330 comp rx=0 tx=0).stop 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- 172.21.15.70:0/601070957 shutdown_connections 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 --2- 172.21.15.70:0/601070957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c0b2a40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 --2- 172.21.15.70:0/601070957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- 172.21.15.70:0/601070957 >> 172.21.15.70:0/601070957 conn(0x7fd75c0b1d40 msgr2=0x7fd75c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- 172.21.15.70:0/601070957 shutdown_connections 2024-04-22T01:15:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.476+0000 7fd761e29640 1 -- 172.21.15.70:0/601070957 wait complete. 2024-04-22T01:15:45.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 Processor -- start 2024-04-22T01:15:45.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- start start 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd75c0b4450 0x7fd75c1380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c15ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c0cd980 con 0x7fd75c0cd050 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c0cd800 con 0x7fd75c0b4450 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd75c0b3790 con 0x7fd75c072280 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c15ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c15ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40316/0 (socket says 172.21.15.70:40316) 2024-04-22T01:15:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 -- 172.21.15.70:0/1320383165 learned_addr learned my addr 172.21.15.70:0/1320383165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75affd640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd75c0b4450 0x7fd75c1380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75b7fe640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 -- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 msgr2=0x7fd75c06f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c06f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 -- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd75c0b4450 msgr2=0x7fd75c1380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd75c0b4450 0x7fd75c1380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 -- 172.21.15.70:0/1320383165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75c148070 con 0x7fd75c0cd050 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd75bfff640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c15ac30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd74c01a9e0 tx=0x7fd74c01ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0b0050 con 0x7fd75c0cd050 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd74c0bc070 con 0x7fd75c0cd050 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd75c148350 con 0x7fd75c0cd050 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c006e20 con 0x7fd75c0cd050 2024-04-22T01:15:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd75c148910 con 0x7fd75c0cd050 2024-04-22T01:15:45.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.480+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75c068210 con 0x7fd75c0cd050 2024-04-22T01:15:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.484+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd74c0b7850 con 0x7fd75c0cd050 2024-04-22T01:15:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.484+0000 7fd753fff640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd73c0678f0 0x7fd73c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.484+0000 7fd75b7fe640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd73c0678f0 0x7fd73c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.484+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd74c120080 con 0x7fd75c0cd050 2024-04-22T01:15:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.484+0000 7fd75b7fe640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd73c0678f0 0x7fd73c069db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd75400e270 tx=0x7fd754009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.488+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd74c0c5020 con 0x7fd75c0cd050 2024-04-22T01:15:45.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.644+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd75c0d01e0 con 0x7fd75c0cd050 2024-04-22T01:15:45.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.644+0000 7fd753fff640 1 -- 172.21.15.70:0/1320383165 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fd74c015fb0 con 0x7fd75c0cd050 2024-04-22T01:15:45.648 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd73c0678f0 msgr2=0x7fd73c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd73c0678f0 0x7fd73c069db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd75400e270 tx=0x7fd754009040 comp rx=0 tx=0).stop 2024-04-22T01:15:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 msgr2=0x7fd75c15ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c15ac30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd74c01a9e0 tx=0x7fd74c01ab70 comp rx=0 tx=0).stop 2024-04-22T01:15:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 shutdown_connections 2024-04-22T01:15:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd73c0678f0 0x7fd73c069db0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd75c0cd050 0x7fd75c15ac30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd75c0b4450 0x7fd75c1380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 --2- 172.21.15.70:0/1320383165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd75c072280 0x7fd75c06f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 >> 172.21.15.70:0/1320383165 conn(0x7fd75c0b1d40 msgr2=0x7fd75c1558c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 shutdown_connections 2024-04-22T01:15:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.648+0000 7fd761e29640 1 -- 172.21.15.70:0/1320383165 wait complete. 2024-04-22T01:15:45.667 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:45.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.784+0000 7fb18d35f640 1 Processor -- start 2024-04-22T01:15:45.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.784+0000 7fb18d35f640 1 -- start start 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb18d35f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb188072280 0x7fb1880ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb18d35f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb18815c290 con 0x7fb1880b4450 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb18d35f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb18815c570 con 0x7fb1880cd050 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb18d35f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb18815c850 con 0x7fb188072280 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb1867fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb188072280 0x7fb1880ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb186ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb1880b4450 0x7fb188071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59492/0 (socket says 172.21.15.70:59492) 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb1867fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb188072280 0x7fb1880ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40326/0 (socket says 172.21.15.70:40326) 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb186ffd640 1 -- 172.21.15.70:0/2606195201 learned_addr learned my addr 172.21.15.70:0/2606195201 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb1867fc640 1 -- 172.21.15.70:0/2606195201 >> v1:172.21.15.98:6789/0 conn(0x7fb1880cd050 legacy=0x7fb1880b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:45.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb1867fc640 1 -- 172.21.15.70:0/2606195201 >> v1:172.21.15.70:6789/0 conn(0x7fb1880b4450 legacy=0x7fb188071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb185ffb640 1 -- 172.21.15.70:0/2606195201 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867108035 0 0) 0x7fb18815c290 con 0x7fb1880b4450 2024-04-22T01:15:45.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb1867fc640 1 -- 172.21.15.70:0/2606195201 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18815cb30 con 0x7fb188072280 2024-04-22T01:15:45.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.788+0000 7fb1867fc640 1 --2- 172.21.15.70:0/2606195201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb1880ccb10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0097c0 tx=0x7fb17c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a34c9865678e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb185ffb640 1 -- 172.21.15.70:0/2606195201 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0b0070 con 0x7fb188072280 2024-04-22T01:15:45.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb185ffb640 1 -- 172.21.15.70:0/2606195201 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb17c0ab600 con 0x7fb188072280 2024-04-22T01:15:45.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb185ffb640 1 -- 172.21.15.70:0/2606195201 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0ab9d0 con 0x7fb188072280 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/2606195201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 msgr2=0x7fb1880ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 --2- 172.21.15.70:0/2606195201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb1880ccb10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0097c0 tx=0x7fb17c01c3d0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/2606195201 shutdown_connections 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 --2- 172.21.15.70:0/2606195201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb1880ccb10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/2606195201 >> 172.21.15.70:0/2606195201 conn(0x7fb1880b1d40 msgr2=0x7fb1880cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/2606195201 shutdown_connections 2024-04-22T01:15:45.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/2606195201 wait complete. 2024-04-22T01:15:45.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 Processor -- start 2024-04-22T01:15:45.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- start start 2024-04-22T01:15:45.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb188146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1880b4450 0x7fb188146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1880cd050 0x7fb1880afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1880cd980 con 0x7fb188072280 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1880cd800 con 0x7fb1880cd050 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb188169b20 con 0x7fb1880b4450 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb188146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb188146800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40330/0 (socket says 172.21.15.70:40330) 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 -- 172.21.15.70:0/570193696 learned_addr learned my addr 172.21.15.70:0/570193696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 -- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1880b4450 msgr2=0x7fb188146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1880b4450 0x7fb188146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 -- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1880cd050 msgr2=0x7fb1880afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb1877fe640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1880cd050 0x7fb1880afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1880cd050 0x7fb1880afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 -- 172.21.15.70:0/570193696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb188146d40 con 0x7fb188072280 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb186ffd640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb188146800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c520 tx=0x7fb17800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178018070 con 0x7fb188072280 2024-04-22T01:15:45.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1780074e0 con 0x7fb188072280 2024-04-22T01:15:45.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18814bd10 con 0x7fb188072280 2024-04-22T01:15:45.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178004a10 con 0x7fb188072280 2024-04-22T01:15:45.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.792+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1881501f0 con 0x7fb188072280 2024-04-22T01:15:45.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.796+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb188068210 con 0x7fb188072280 2024-04-22T01:15:45.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.796+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb17801b070 con 0x7fb188072280 2024-04-22T01:15:45.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.796+0000 7fb16f7fe640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1740678c0 0x7fb174069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:45.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.796+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb1780853e0 con 0x7fb188072280 2024-04-22T01:15:45.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.796+0000 7fb1867fc640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1740678c0 0x7fb174069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:45.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.796+0000 7fb1867fc640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1740678c0 0x7fb174069d80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0ad000 tx=0x7fb17c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:45.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.800+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb178055bb0 con 0x7fb188072280 2024-04-22T01:15:45.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb1880b0ca0 con 0x7fb188072280 2024-04-22T01:15:45.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb16f7fe640 1 -- 172.21.15.70:0/570193696 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fb1780554b0 con 0x7fb188072280 2024-04-22T01:15:45.959 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:45.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1740678c0 msgr2=0x7fb174069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1740678c0 0x7fb174069d80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0ad000 tx=0x7fb17c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:45.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 msgr2=0x7fb188146800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:45.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb188146800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c520 tx=0x7fb17800c9e0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 shutdown_connections 2024-04-22T01:15:45.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1740678c0 0x7fb174069d80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1880cd050 0x7fb1880afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1880b4450 0x7fb188146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 --2- 172.21.15.70:0/570193696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb188072280 0x7fb188146800 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:45.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 >> 172.21.15.70:0/570193696 conn(0x7fb1880b1d40 msgr2=0x7fb1880cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:45.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 shutdown_connections 2024-04-22T01:15:45.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:45.956+0000 7fb18d35f640 1 -- 172.21.15.70:0/570193696 wait complete. 2024-04-22T01:15:45.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:46.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.096+0000 7f0a29736640 1 Processor -- start 2024-04-22T01:15:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.096+0000 7f0a29736640 1 -- start start 2024-04-22T01:15:46.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.096+0000 7f0a29736640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a240d0d30 0x7f0a240b0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.096+0000 7f0a29736640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a240ce7c0 con 0x7f0a240cf580 2024-04-22T01:15:46.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.096+0000 7f0a29736640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a240ceaa0 con 0x7f0a2415ace0 2024-04-22T01:15:46.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.096+0000 7f0a29736640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2415b0c0 con 0x7f0a240d0d30 2024-04-22T01:15:46.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a237fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a240d0d30 0x7f0a240b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a227fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0a240cf580 0x7f0a240d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59506/0 (socket says 172.21.15.70:59506) 2024-04-22T01:15:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a227fc640 1 -- 172.21.15.70:0/1472042808 learned_addr learned my addr 172.21.15.70:0/1472042808 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a237fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a240d0d30 0x7f0a240b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40340/0 (socket says 172.21.15.70:40340) 2024-04-22T01:15:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a237fe640 1 -- 172.21.15.70:0/1472042808 >> v1:172.21.15.98:6789/0 conn(0x7f0a2415ace0 legacy=0x7f0a240cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a237fe640 1 -- 172.21.15.70:0/1472042808 >> v1:172.21.15.70:6789/0 conn(0x7f0a240cf580 legacy=0x7f0a240d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a237fe640 1 -- 172.21.15.70:0/1472042808 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a240ced80 con 0x7f0a240d0d30 2024-04-22T01:15:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a21ffb640 1 -- 172.21.15.70:0/1472042808 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497054506 0 0) 0x7f0a240ce7c0 con 0x7f0a240cf580 2024-04-22T01:15:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a237fe640 1 --2- 172.21.15.70:0/1472042808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a240b0650 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009d90 tx=0x7f0a1c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1b3d9aa46786ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a21ffb640 1 -- 172.21.15.70:0/1472042808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c0b2070 con 0x7f0a240d0d30 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a21ffb640 1 -- 172.21.15.70:0/1472042808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a1c0a5760 con 0x7f0a240d0d30 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a21ffb640 1 -- 172.21.15.70:0/1472042808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c0a5b30 con 0x7f0a240d0d30 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 -- 172.21.15.70:0/1472042808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 msgr2=0x7f0a240b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 --2- 172.21.15.70:0/1472042808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a240b0650 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009d90 tx=0x7f0a1c01c8a0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 -- 172.21.15.70:0/1472042808 shutdown_connections 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 --2- 172.21.15.70:0/1472042808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a240b0650 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 -- 172.21.15.70:0/1472042808 >> 172.21.15.70:0/1472042808 conn(0x7f0a24099a60 msgr2=0x7f0a24099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 -- 172.21.15.70:0/1472042808 shutdown_connections 2024-04-22T01:15:46.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.100+0000 7f0a29736640 1 -- 172.21.15.70:0/1472042808 wait complete. 2024-04-22T01:15:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 Processor -- start 2024-04-22T01:15:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- start start 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a240cf580 0x7f0a2415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a2415c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a2415ace0 0x7f0a2415d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a24168950 con 0x7f0a240d0d30 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a241687d0 con 0x7f0a240cf580 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a24168ad0 con 0x7f0a2415ace0 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a2415c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a2415c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40348/0 (socket says 172.21.15.70:40348) 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 -- 172.21.15.70:0/3629439583 learned_addr learned my addr 172.21.15.70:0/3629439583 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a237fe640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a2415ace0 0x7f0a2415d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 -- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a2415ace0 msgr2=0x7f0a2415d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a22ffd640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a240cf580 0x7f0a2415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a2415ace0 0x7f0a2415d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 -- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a240cf580 msgr2=0x7f0a2415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a240cf580 0x7f0a2415a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 -- 172.21.15.70:0/3629439583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a24159b90 con 0x7f0a240d0d30 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a227fc640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a2415c010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a180011f0 tx=0x7f0a1800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18018070 con 0x7f0a240d0d30 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a180074e0 con 0x7f0a240d0d30 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18004a10 con 0x7f0a240d0d30 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2415b250 con 0x7f0a240d0d30 2024-04-22T01:15:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2415b810 con 0x7f0a240d0d30 2024-04-22T01:15:46.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2404f510 con 0x7f0a240d0d30 2024-04-22T01:15:46.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.104+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a1801b070 con 0x7f0a240d0d30 2024-04-22T01:15:46.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.108+0000 7f0a0b7fe640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a100678c0 0x7f0a10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.108+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0a180853d0 con 0x7f0a240d0d30 2024-04-22T01:15:46.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.108+0000 7f0a22ffd640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a100678c0 0x7f0a10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.108+0000 7f0a22ffd640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a100678c0 0x7f0a10069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400a750 tx=0x7f0a14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.112+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0a18055b10 con 0x7f0a240d0d30 2024-04-22T01:15:46.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.280+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0a240d06c0 con 0x7f0a240d0d30 2024-04-22T01:15:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a0b7fe640 1 -- 172.21.15.70:0/3629439583 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f0a18055410 con 0x7f0a240d0d30 2024-04-22T01:15:46.285 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:46.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a100678c0 msgr2=0x7f0a10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a100678c0 0x7f0a10069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400a750 tx=0x7f0a14008040 comp rx=0 tx=0).stop 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 msgr2=0x7f0a2415c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a2415c010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a180011f0 tx=0x7f0a1800c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 shutdown_connections 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a100678c0 0x7f0a10069d80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a2415ace0 0x7f0a2415d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a240d0d30 0x7f0a2415c010 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 --2- 172.21.15.70:0/3629439583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a240cf580 0x7f0a2415a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 >> 172.21.15.70:0/3629439583 conn(0x7f0a24099a60 msgr2=0x7f0a2406b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 shutdown_connections 2024-04-22T01:15:46.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.284+0000 7f0a29736640 1 -- 172.21.15.70:0/3629439583 wait complete. 2024-04-22T01:15:46.303 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:15:46.304 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:46.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:46.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.428+0000 7ff1c2414640 1 Processor -- start 2024-04-22T01:15:46.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.428+0000 7ff1c2414640 1 -- start start 2024-04-22T01:15:46.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c2414640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1bc072230 0x7ff1bc0aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c2414640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc0cf4a0 con 0x7ff1bc15a550 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c2414640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc06f260 con 0x7ff1bc0aeae0 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c2414640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff1bc06f010 con 0x7ff1bc072230 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1bc072230 0x7ff1bc0aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1bc072230 0x7ff1bc0aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50972/0 (socket says 172.21.15.70:50972) 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1bbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff1bc15a550 0x7ff1bc0cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38732/0 (socket says 172.21.15.70:38732) 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 -- 172.21.15.70:0/992162943 learned_addr learned my addr 172.21.15.70:0/992162943 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 -- 172.21.15.70:0/992162943 >> v1:172.21.15.98:6789/0 conn(0x7ff1bc0aeae0 legacy=0x7ff1bc071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 -- 172.21.15.70:0/992162943 >> v1:172.21.15.70:6789/0 conn(0x7ff1bc15a550 legacy=0x7ff1bc0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 -- 172.21.15.70:0/992162943 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc06f540 con 0x7ff1bc072230 2024-04-22T01:15:46.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c09b1640 1 --2- 172.21.15.70:0/992162943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0aef50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009e50 tx=0x7ff1b401db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baec60abfdcec2a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1baffd640 1 -- 172.21.15.70:0/992162943 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40a67b0 con 0x7ff1bc072230 2024-04-22T01:15:46.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1baffd640 1 -- 172.21.15.70:0/992162943 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b40a6950 con 0x7ff1bc072230 2024-04-22T01:15:46.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1baffd640 1 -- 172.21.15.70:0/992162943 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40a6d20 con 0x7ff1bc072230 2024-04-22T01:15:46.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c2414640 1 -- 172.21.15.70:0/992162943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 msgr2=0x7ff1bc0aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.432+0000 7ff1c2414640 1 --2- 172.21.15.70:0/992162943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0aef50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009e50 tx=0x7ff1b401db00 comp rx=0 tx=0).stop 2024-04-22T01:15:46.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- 172.21.15.70:0/992162943 shutdown_connections 2024-04-22T01:15:46.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 --2- 172.21.15.70:0/992162943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0aef50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- 172.21.15.70:0/992162943 >> 172.21.15.70:0/992162943 conn(0x7ff1bc06b1f0 msgr2=0x7ff1bc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:46.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- 172.21.15.70:0/992162943 shutdown_connections 2024-04-22T01:15:46.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- 172.21.15.70:0/992162943 wait complete. 2024-04-22T01:15:46.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 Processor -- start 2024-04-22T01:15:46.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- start start 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1bc0aeae0 0x7ff1bc0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1bc15a550 0x7ff1bc156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc0b3150 con 0x7ff1bc072230 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc0b2fd0 con 0x7ff1bc0aeae0 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff1bc0b32d0 con 0x7ff1bc15a550 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0b0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50988/0 (socket says 172.21.15.70:50988) 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 -- 172.21.15.70:0/3086923524 learned_addr learned my addr 172.21.15.70:0/3086923524 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:46.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 -- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1bc15a550 msgr2=0x7ff1bc156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c09b1640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1bc15a550 0x7ff1bc156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bb7fe640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1bc0aeae0 0x7ff1bc0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1bc15a550 0x7ff1bc156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 -- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1bc0aeae0 msgr2=0x7ff1bc0b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1bc0aeae0 0x7ff1bc0b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 -- 172.21.15.70:0/3086923524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc14bd30 con 0x7ff1bc072230 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1bbfff640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0b0630 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c3a0 tx=0x7ff1ac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac0074e0 con 0x7ff1bc072230 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac0076a0 con 0x7ff1bc072230 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1bc14c080 con 0x7ff1bc072230 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac004a30 con 0x7ff1bc072230 2024-04-22T01:15:46.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.436+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1bc14c640 con 0x7ff1bc072230 2024-04-22T01:15:46.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.440+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1bc068210 con 0x7ff1bc072230 2024-04-22T01:15:46.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.440+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff1ac01a020 con 0x7ff1bc072230 2024-04-22T01:15:46.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.440+0000 7ff1b8ff9640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1a80678c0 0x7ff1a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.440+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff1ac0857f0 con 0x7ff1bc072230 2024-04-22T01:15:46.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.440+0000 7ff1bb7fe640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1a80678c0 0x7ff1a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.440+0000 7ff1bb7fe640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1a80678c0 0x7ff1a8069d80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0003d60 tx=0x7ff1b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.444+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff1ac055f30 con 0x7ff1bc072230 2024-04-22T01:15:46.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.592+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1bc155b70 con 0x7ff1bc072230 2024-04-22T01:15:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.592+0000 7ff1b8ff9640 1 -- 172.21.15.70:0/3086923524 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7ff1ac055830 con 0x7ff1bc072230 2024-04-22T01:15:46.597 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:46.597 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1a80678c0 msgr2=0x7ff1a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1a80678c0 0x7ff1a8069d80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0003d60 tx=0x7ff1b0009040 comp rx=0 tx=0).stop 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 msgr2=0x7ff1bc0b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0b0630 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c3a0 tx=0x7ff1ac00c860 comp rx=0 tx=0).stop 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 shutdown_connections 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1a80678c0 0x7ff1a8069d80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1bc15a550 0x7ff1bc156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1bc0aeae0 0x7ff1bc0b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 --2- 172.21.15.70:0/3086923524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1bc072230 0x7ff1bc0b0630 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 >> 172.21.15.70:0/3086923524 conn(0x7ff1bc06b1f0 msgr2=0x7ff1bc15c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 shutdown_connections 2024-04-22T01:15:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.596+0000 7ff1c2414640 1 -- 172.21.15.70:0/3086923524 wait complete. 2024-04-22T01:15:46.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:46.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 Processor -- start 2024-04-22T01:15:46.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 -- start start 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3db00b4450 0x7f3db0071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3db00720b0 0x7f3db00ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db015c290 con 0x7f3db00cd050 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3db00b3a50 con 0x7f3db00720b0 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3db561e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3db015c570 con 0x7f3db00b4450 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3db00720b0 0x7f3db00ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3db00720b0 0x7f3db00ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50992/0 (socket says 172.21.15.70:50992) 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 -- 172.21.15.70:0/3048458813 learned_addr learned my addr 172.21.15.70:0/3048458813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3daeffd640 1 --2- 172.21.15.70:0/3048458813 >> v2:172.21.15.98:3300/0 conn(0x7f3db00b4450 0x7f3db0071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 -- 172.21.15.70:0/3048458813 >> v2:172.21.15.98:3300/0 conn(0x7f3db00b4450 msgr2=0x7f3db0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 --2- 172.21.15.70:0/3048458813 >> v2:172.21.15.98:3300/0 conn(0x7f3db00b4450 0x7f3db0071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 -- 172.21.15.70:0/3048458813 >> v1:172.21.15.98:6789/0 conn(0x7f3db00cd050 legacy=0x7f3db015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3dae7fc640 1 -- 172.21.15.70:0/3048458813 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db015c850 con 0x7f3db00720b0 2024-04-22T01:15:46.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.736+0000 7f3daeffd640 1 --2- 172.21.15.70:0/3048458813 >> v2:172.21.15.98:3300/0 conn(0x7f3db00b4450 0x7f3db0071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:46.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3dae7fc640 1 --2- 172.21.15.70:0/3048458813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00720b0 0x7f3db00ccb10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3db006fce0 tx=0x7f3d9801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edacb4d21493745 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3dadffb640 1 -- 172.21.15.70:0/3048458813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d980b0070 con 0x7f3db00720b0 2024-04-22T01:15:46.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3dadffb640 1 -- 172.21.15.70:0/3048458813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d980ab600 con 0x7f3db00720b0 2024-04-22T01:15:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3dadffb640 1 -- 172.21.15.70:0/3048458813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d980ab9d0 con 0x7f3db00720b0 2024-04-22T01:15:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 -- 172.21.15.70:0/3048458813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00720b0 msgr2=0x7f3db00ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 --2- 172.21.15.70:0/3048458813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00720b0 0x7f3db00ccb10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3db006fce0 tx=0x7f3d9801c520 comp rx=0 tx=0).stop 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 -- 172.21.15.70:0/3048458813 shutdown_connections 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 --2- 172.21.15.70:0/3048458813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00720b0 0x7f3db00ccb10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 --2- 172.21.15.70:0/3048458813 >> v2:172.21.15.98:3300/0 conn(0x7f3db00b4450 0x7f3db0071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 -- 172.21.15.70:0/3048458813 >> 172.21.15.70:0/3048458813 conn(0x7f3db00b1d40 msgr2=0x7f3db00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 -- 172.21.15.70:0/3048458813 shutdown_connections 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 -- 172.21.15.70:0/3048458813 wait complete. 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 Processor -- start 2024-04-22T01:15:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.740+0000 7f3db561e640 1 -- start start 2024-04-22T01:15:46.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3db00720b0 0x7f3db01453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3db00b4450 0x7f3db0143700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 0x7f3db0142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3db00b49b0 con 0x7f3db00cd050 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3db00b4830 con 0x7f3db00b4450 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3db00cd800 con 0x7f3db00720b0 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 0x7f3db0142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 0x7f3db0142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51002/0 (socket says 172.21.15.70:51002) 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 -- 172.21.15.70:0/2708135183 learned_addr learned my addr 172.21.15.70:0/2708135183 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daeffd640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3db00720b0 0x7f3db01453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 -- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3db00720b0 msgr2=0x7f3db01453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3dae7fc640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3db00b4450 0x7f3db0143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3db00720b0 0x7f3db01453b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 -- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3db00b4450 msgr2=0x7f3db0143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3db00b4450 0x7f3db0143700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 -- 172.21.15.70:0/2708135183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db0143510 con 0x7f3db00cd050 2024-04-22T01:15:46.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daf7fe640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 0x7f3db0142fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3da800c460 tx=0x7f3da800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8016020 con 0x7f3db00cd050 2024-04-22T01:15:46.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da8005220 con 0x7f3db00cd050 2024-04-22T01:15:46.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db0146d00 con 0x7f3db00cd050 2024-04-22T01:15:46.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8007030 con 0x7f3db00cd050 2024-04-22T01:15:46.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db0147560 con 0x7f3db00cd050 2024-04-22T01:15:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db0068210 con 0x7f3db00cd050 2024-04-22T01:15:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3da80071d0 con 0x7f3db00cd050 2024-04-22T01:15:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3da77fe640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d900678f0 0x7f3d90069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daeffd640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d900678f0 0x7f3d90069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3da8085970 con 0x7f3db00cd050 2024-04-22T01:15:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.744+0000 7f3daeffd640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d900678f0 0x7f3d90069db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009f70 tx=0x7f3da000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:46.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.748+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3da8020020 con 0x7f3db00cd050 2024-04-22T01:15:46.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.908+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3db00cc190 con 0x7f3db00cd050 2024-04-22T01:15:46.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.908+0000 7f3da77fe640 1 -- 172.21.15.70:0/2708135183 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f3da80074c0 con 0x7f3db00cd050 2024-04-22T01:15:46.913 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d900678f0 msgr2=0x7f3d90069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d900678f0 0x7f3d90069db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009f70 tx=0x7f3da000b040 comp rx=0 tx=0).stop 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 msgr2=0x7f3db0142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 0x7f3db0142fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3da800c460 tx=0x7f3da800c920 comp rx=0 tx=0).stop 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 shutdown_connections 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d900678f0 0x7f3d90069db0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3db00cd050 0x7f3db0142fd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3db00b4450 0x7f3db0143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 --2- 172.21.15.70:0/2708135183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3db00720b0 0x7f3db01453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:46.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 >> 172.21.15.70:0/2708135183 conn(0x7f3db00b1d40 msgr2=0x7f3db00b2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:46.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 shutdown_connections 2024-04-22T01:15:46.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:46.912+0000 7f3db561e640 1 -- 172.21.15.70:0/2708135183 wait complete. 2024-04-22T01:15:46.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:47.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.040+0000 7f83b07d8640 1 Processor -- start 2024-04-22T01:15:47.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.040+0000 7f83b07d8640 1 -- start start 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83b07d8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83a80cd220 0x7f83a8166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83b07d8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a815c290 con 0x7f83a8072280 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83b07d8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a815c570 con 0x7f83a80b4450 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83b07d8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f83a80b3a50 con 0x7f83a80cd220 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83aed75640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83a80cd220 0x7f83a8166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83add73640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f83a8072280 0x7f83a80ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38744/0 (socket says 172.21.15.70:38744) 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83add73640 1 -- 172.21.15.70:0/562715178 learned_addr learned my addr 172.21.15.70:0/562715178 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:47.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83aed75640 1 -- 172.21.15.70:0/562715178 >> v1:172.21.15.98:6789/0 conn(0x7f83a80b4450 legacy=0x7f83a8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:47.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83aed75640 1 -- 172.21.15.70:0/562715178 >> v1:172.21.15.70:6789/0 conn(0x7f83a8072280 legacy=0x7f83a80ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83aed75640 1 -- 172.21.15.70:0/562715178 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a815c850 con 0x7f83a80cd220 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83aed75640 1 --2- 172.21.15.70:0/562715178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a80cd220 0x7f83a8166b00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009d90 tx=0x7f83a001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a70118ddd621000 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83ad572640 1 -- 172.21.15.70:0/562715178 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00b2070 con 0x7f83a80cd220 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83ad572640 1 -- 172.21.15.70:0/562715178 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a00a5870 con 0x7f83a80cd220 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83ad572640 1 -- 172.21.15.70:0/562715178 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00a5c40 con 0x7f83a80cd220 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83b07d8640 1 -- 172.21.15.70:0/562715178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a80cd220 msgr2=0x7f83a8166b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.044+0000 7f83b07d8640 1 --2- 172.21.15.70:0/562715178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a80cd220 0x7f83a8166b00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009d90 tx=0x7f83a001c930 comp rx=0 tx=0).stop 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/562715178 shutdown_connections 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 --2- 172.21.15.70:0/562715178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a80cd220 0x7f83a8166b00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/562715178 >> 172.21.15.70:0/562715178 conn(0x7f83a80b1d40 msgr2=0x7f83a80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/562715178 shutdown_connections 2024-04-22T01:15:47.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/562715178 wait complete. 2024-04-22T01:15:47.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 Processor -- start 2024-04-22T01:15:47.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- start start 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 0x7f83a8146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a80b4450 0x7f83a814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a80cd220 0x7f83a815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83a80cd980 con 0x7f83a8072280 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83a80cd800 con 0x7f83a80b4450 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83a8169860 con 0x7f83a80cd220 2024-04-22T01:15:47.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 0x7f83a8146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 0x7f83a8146dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51020/0 (socket says 172.21.15.70:51020) 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 -- 172.21.15.70:0/803622188 learned_addr learned my addr 172.21.15.70:0/803622188 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83add73640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a80b4450 0x7f83a814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83aed75640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a80cd220 0x7f83a815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 -- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a80cd220 msgr2=0x7f83a815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a80cd220 0x7f83a815d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 -- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a80b4450 msgr2=0x7f83a814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a80b4450 0x7f83a814aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 -- 172.21.15.70:0/803622188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a814d8d0 con 0x7f83a8072280 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83ae574640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 0x7f83a8146dc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f83a400c450 tx=0x7f83a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4012830 con 0x7f83a8072280 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a4017070 con 0x7f83a8072280 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a814dbb0 con 0x7f83a8072280 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4012c30 con 0x7f83a8072280 2024-04-22T01:15:47.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a814e170 con 0x7f83a8072280 2024-04-22T01:15:47.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.048+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a8068210 con 0x7f83a8072280 2024-04-22T01:15:47.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.052+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83a401c460 con 0x7f83a8072280 2024-04-22T01:15:47.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.052+0000 7f839effd640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f838c0678c0 0x7f838c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.052+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f83a4085db0 con 0x7f83a8072280 2024-04-22T01:15:47.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.052+0000 7f83add73640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f838c0678c0 0x7f838c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.052+0000 7f83add73640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f838c0678c0 0x7f838c069d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f83a814a320 tx=0x7f8398007e90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.056+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f83a408a070 con 0x7f83a8072280 2024-04-22T01:15:47.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.200+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f83a80d01e0 con 0x7f83a8072280 2024-04-22T01:15:47.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f839effd640 1 -- 172.21.15.70:0/803622188 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f83a4055e00 con 0x7f83a8072280 2024-04-22T01:15:47.206 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f838c0678c0 msgr2=0x7f838c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f838c0678c0 0x7f838c069d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f83a814a320 tx=0x7f8398007e90 comp rx=0 tx=0).stop 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 msgr2=0x7f83a8146dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 0x7f83a8146dc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f83a400c450 tx=0x7f83a400c910 comp rx=0 tx=0).stop 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 shutdown_connections 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f838c0678c0 0x7f838c069d80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a80cd220 0x7f83a815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a80b4450 0x7f83a814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 --2- 172.21.15.70:0/803622188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83a8072280 0x7f83a8146dc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 >> 172.21.15.70:0/803622188 conn(0x7f83a80b1d40 msgr2=0x7f83a8071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 shutdown_connections 2024-04-22T01:15:47.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.204+0000 7f83b07d8640 1 -- 172.21.15.70:0/803622188 wait complete. 2024-04-22T01:15:47.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 Processor -- start 2024-04-22T01:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 -- start start 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc63c0ce920 0x7fc63c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c0cedd0 0x7fc63c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63c0b3800 con 0x7fc63c15e6e0 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc63c0b3ae0 con 0x7fc63c0ce920 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc643c5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c0cde40 con 0x7fc63c0cedd0 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6411f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc63c0ce920 0x7fc63c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6419fa640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc63c15e6e0 0x7fc63c15eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38750/0 (socket says 172.21.15.70:38750) 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6419fa640 1 -- 172.21.15.70:0/3567561020 learned_addr learned my addr 172.21.15.70:0/3567561020 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6421fb640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c0cedd0 0x7fc63c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6411f9640 1 -- 172.21.15.70:0/3567561020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c0cedd0 msgr2=0x7fc63c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6411f9640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c0cedd0 0x7fc63c1551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6411f9640 1 -- 172.21.15.70:0/3567561020 >> v1:172.21.15.70:6789/0 conn(0x7fc63c15e6e0 legacy=0x7fc63c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6411f9640 1 -- 172.21.15.70:0/3567561020 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c0ce120 con 0x7fc63c0ce920 2024-04-22T01:15:47.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6409f8640 1 -- 172.21.15.70:0/3567561020 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851972424 0 0) 0x7fc63c0b3800 con 0x7fc63c15e6e0 2024-04-22T01:15:47.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.332+0000 7fc6421fb640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c0cedd0 0x7fc63c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:47.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc6411f9640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0ce920 0x7fc63c154c60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0097c0 tx=0x7fc62c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec23d1df16c38bd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc6409f8640 1 -- 172.21.15.70:0/3567561020 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0aa5f0 con 0x7fc63c0ce920 2024-04-22T01:15:47.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc6409f8640 1 -- 172.21.15.70:0/3567561020 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c0af070 con 0x7fc63c0ce920 2024-04-22T01:15:47.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc6409f8640 1 -- 172.21.15.70:0/3567561020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0aa9b0 con 0x7fc63c0ce920 2024-04-22T01:15:47.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 -- 172.21.15.70:0/3567561020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0ce920 msgr2=0x7fc63c154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0ce920 0x7fc63c154c60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0097c0 tx=0x7fc62c01c470 comp rx=0 tx=0).stop 2024-04-22T01:15:47.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 -- 172.21.15.70:0/3567561020 shutdown_connections 2024-04-22T01:15:47.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c0cedd0 0x7fc63c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 --2- 172.21.15.70:0/3567561020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0ce920 0x7fc63c154c60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 -- 172.21.15.70:0/3567561020 >> 172.21.15.70:0/3567561020 conn(0x7fc63c0b1d40 msgr2=0x7fc63c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 -- 172.21.15.70:0/3567561020 shutdown_connections 2024-04-22T01:15:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 -- 172.21.15.70:0/3567561020 wait complete. 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 Processor -- start 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.336+0000 7fc643c5e640 1 -- start start 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0ce920 0x7fc63c15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 0x7fc63c1f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c15e6e0 0x7fc63c1f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c0b2e40 con 0x7fc63c0cedd0 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c0b2cc0 con 0x7fc63c0ce920 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c0b2fc0 con 0x7fc63c15e6e0 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 0x7fc63c1f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 0x7fc63c1f8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51036/0 (socket says 172.21.15.70:51036) 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 -- 172.21.15.70:0/2285568943 learned_addr learned my addr 172.21.15.70:0/2285568943 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:47.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6419fa640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0ce920 0x7fc63c15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 -- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c15e6e0 msgr2=0x7fc63c1f87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c15e6e0 0x7fc63c1f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 -- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0ce920 msgr2=0x7fc63c15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0ce920 0x7fc63c15a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 -- 172.21.15.70:0/2285568943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c1550f0 con 0x7fc63c0cedd0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6411f9640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 0x7fc63c1f8270 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc62c01a0a0 tx=0x7fc62c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0be610 con 0x7fc63c0cedd0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c0be7d0 con 0x7fc63c0cedd0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c152bb0 con 0x7fc63c0cedd0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0bebc0 con 0x7fc63c0cedd0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c155670 con 0x7fc63c0cedd0 2024-04-22T01:15:47.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c068210 con 0x7fc63c0cedd0 2024-04-22T01:15:47.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc62c0bc020 con 0x7fc63c0cedd0 2024-04-22T01:15:47.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6327fc640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6200678c0 0x7fc620069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc62c120ba0 con 0x7fc63c0cedd0 2024-04-22T01:15:47.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6419fa640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6200678c0 0x7fc620069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.340+0000 7fc6419fa640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6200678c0 0x7fc620069d80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc638000f70 tx=0x7fc638009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.344+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc62c0c3060 con 0x7fc63c0cedd0 2024-04-22T01:15:47.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.492+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc63c0d08e0 con 0x7fc63c0cedd0 2024-04-22T01:15:47.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.492+0000 7fc6327fc640 1 -- 172.21.15.70:0/2285568943 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fc62c016090 con 0x7fc63c0cedd0 2024-04-22T01:15:47.496 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6200678c0 msgr2=0x7fc620069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6200678c0 0x7fc620069d80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc638000f70 tx=0x7fc638009040 comp rx=0 tx=0).stop 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 msgr2=0x7fc63c1f8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 0x7fc63c1f8270 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc62c01a0a0 tx=0x7fc62c01acb0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 shutdown_connections 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6200678c0 0x7fc620069d80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c15e6e0 0x7fc63c1f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63c0cedd0 0x7fc63c1f8270 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 --2- 172.21.15.70:0/2285568943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0ce920 0x7fc63c15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 >> 172.21.15.70:0/2285568943 conn(0x7fc63c0b1d40 msgr2=0x7fc63c158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 shutdown_connections 2024-04-22T01:15:47.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.496+0000 7fc643c5e640 1 -- 172.21.15.70:0/2285568943 wait complete. 2024-04-22T01:15:47.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:47.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 Processor -- start 2024-04-22T01:15:47.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 -- start start 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f545815de20 0x7f54580b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54580cea10 0x7f54580ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54580cf3e0 con 0x7f54580b6db0 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f54580b75e0 con 0x7f54580cea10 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545cc72640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f54580b78c0 con 0x7f545815de20 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5455d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f54580b6db0 0x7f54580d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38754/0 (socket says 172.21.15.70:38754) 2024-04-22T01:15:47.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5455d9b640 1 -- 172.21.15.70:0/50228988 learned_addr learned my addr 172.21.15.70:0/50228988 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5456d9d640 1 --2- 172.21.15.70:0/50228988 >> v2:172.21.15.70:3300/0 conn(0x7f54580cea10 0x7f54580ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545659c640 1 --2- 172.21.15.70:0/50228988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f545815de20 0x7f54580b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5456d9d640 1 -- 172.21.15.70:0/50228988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f545815de20 msgr2=0x7f54580b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5456d9d640 1 --2- 172.21.15.70:0/50228988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f545815de20 0x7f54580b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5456d9d640 1 -- 172.21.15.70:0/50228988 >> v1:172.21.15.70:6789/0 conn(0x7f54580b6db0 legacy=0x7f54580d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f545559a640 1 -- 172.21.15.70:0/50228988 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456834702 0 0) 0x7f54580cf3e0 con 0x7f54580b6db0 2024-04-22T01:15:47.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.632+0000 7f5456d9d640 1 -- 172.21.15.70:0/50228988 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54580b7ba0 con 0x7f54580cea10 2024-04-22T01:15:47.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f5456d9d640 1 --2- 172.21.15.70:0/50228988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54580cea10 0x7f54580ceea0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5450009dc0 tx=0x7f545001f140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31c44a21d4d7154 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545559a640 1 -- 172.21.15.70:0/50228988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54500b4070 con 0x7f54580cea10 2024-04-22T01:15:47.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545559a640 1 -- 172.21.15.70:0/50228988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54500af600 con 0x7f54580cea10 2024-04-22T01:15:47.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545559a640 1 -- 172.21.15.70:0/50228988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54500af9d0 con 0x7f54580cea10 2024-04-22T01:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- 172.21.15.70:0/50228988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54580cea10 msgr2=0x7f54580ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 --2- 172.21.15.70:0/50228988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54580cea10 0x7f54580ceea0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5450009dc0 tx=0x7f545001f140 comp rx=0 tx=0).stop 2024-04-22T01:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- 172.21.15.70:0/50228988 shutdown_connections 2024-04-22T01:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 --2- 172.21.15.70:0/50228988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54580cea10 0x7f54580ceea0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 --2- 172.21.15.70:0/50228988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f545815de20 0x7f54580b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- 172.21.15.70:0/50228988 >> 172.21.15.70:0/50228988 conn(0x7f5458072d20 msgr2=0x7f5458073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- 172.21.15.70:0/50228988 shutdown_connections 2024-04-22T01:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- 172.21.15.70:0/50228988 wait complete. 2024-04-22T01:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 Processor -- start 2024-04-22T01:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- start start 2024-04-22T01:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54580b6db0 0x7f5458151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54580cea10 0x7f54580d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 0x7f545815c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f545815e200 con 0x7f545815de20 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54580b8160 con 0x7f54580b6db0 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.636+0000 7f545cc72640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f54580b71f0 con 0x7f54580cea10 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 0x7f545815c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 0x7f545815c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51058/0 (socket says 172.21.15.70:51058) 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 -- 172.21.15.70:0/322251005 learned_addr learned my addr 172.21.15.70:0/322251005 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545659c640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54580b6db0 0x7f5458151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5455d9b640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54580cea10 0x7f54580d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 -- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54580cea10 msgr2=0x7f54580d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54580cea10 0x7f54580d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 -- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54580b6db0 msgr2=0x7f5458151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54580b6db0 0x7f5458151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 -- 172.21.15.70:0/322251005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5458154930 con 0x7f545815de20 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545659c640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54580b6db0 0x7f5458151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:47.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5455d9b640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54580cea10 0x7f54580d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5456d9d640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 0x7f545815c0b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f545001d7f0 tx=0x7f545001bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54500b4040 con 0x7f545815de20 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54500c1070 con 0x7f545815de20 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5458154bd0 con 0x7f545815de20 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450009450 con 0x7f545815de20 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5458155190 con 0x7f545815de20 2024-04-22T01:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5458068210 con 0x7f545815de20 2024-04-22T01:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f54500ba6b0 con 0x7f545815de20 2024-04-22T01:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5446ffd640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54380678c0 0x7f5438069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5450125520 con 0x7f545815de20 2024-04-22T01:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545659c640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54380678c0 0x7f5438069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.640+0000 7f545659c640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54380678c0 0x7f5438069d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5448009a40 tx=0x7f5448008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.644+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5450018ca0 con 0x7f545815de20 2024-04-22T01:15:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.804+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f54580ce7c0 con 0x7f545815de20 2024-04-22T01:15:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.804+0000 7f5446ffd640 1 -- 172.21.15.70:0/322251005 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f54500f5d60 con 0x7f545815de20 2024-04-22T01:15:47.808 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54380678c0 msgr2=0x7f5438069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54380678c0 0x7f5438069d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5448009a40 tx=0x7f5448008040 comp rx=0 tx=0).stop 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 msgr2=0x7f545815c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 0x7f545815c0b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f545001d7f0 tx=0x7f545001bee0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 shutdown_connections 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54380678c0 0x7f5438069d80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545815de20 0x7f545815c0b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54580cea10 0x7f54580d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 --2- 172.21.15.70:0/322251005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54580b6db0 0x7f5458151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 >> 172.21.15.70:0/322251005 conn(0x7f5458072d20 msgr2=0x7f54580c9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 shutdown_connections 2024-04-22T01:15:47.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.808+0000 7f545cc72640 1 -- 172.21.15.70:0/322251005 wait complete. 2024-04-22T01:15:47.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:47.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 Processor -- start 2024-04-22T01:15:47.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 -- start start 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc015ace0 0x7fcdc00cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcdc00cbf90 0x7fcdc00d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcdc00cc3b0 0x7fcdc00d0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcdc015b0c0 con 0x7fcdc00cbf90 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcdc00ce7c0 con 0x7fcdc00cc3b0 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdc4efd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc00ceaa0 con 0x7fcdc015ace0 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcdc00cbf90 0x7fcdc00d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcdc00cbf90 0x7fcdc00d0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51074/0 (socket says 172.21.15.70:51074) 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 -- 172.21.15.70:0/3755079420 learned_addr learned my addr 172.21.15.70:0/3755079420 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbe59c640 1 --2- 172.21.15.70:0/3755079420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc015ace0 0x7fcdc00cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbed9d640 1 --2- 172.21.15.70:0/3755079420 >> v2:172.21.15.98:3300/0 conn(0x7fcdc00cc3b0 0x7fcdc00d0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 -- 172.21.15.70:0/3755079420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc015ace0 msgr2=0x7fcdc00cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:47.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 --2- 172.21.15.70:0/3755079420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc015ace0 0x7fcdc00cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 -- 172.21.15.70:0/3755079420 >> v2:172.21.15.98:3300/0 conn(0x7fcdc00cc3b0 msgr2=0x7fcdc00d0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 --2- 172.21.15.70:0/3755079420 >> v2:172.21.15.98:3300/0 conn(0x7fcdc00cc3b0 0x7fcdc00d0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbdd9b640 1 -- 172.21.15.70:0/3755079420 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc00ced80 con 0x7fcdc00cbf90 2024-04-22T01:15:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.940+0000 7fcdbed9d640 1 --2- 172.21.15.70:0/3755079420 >> v2:172.21.15.98:3300/0 conn(0x7fcdc00cc3b0 0x7fcdc00d0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbdd9b640 1 --2- 172.21.15.70:0/3755079420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc00d0620 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00b6a60 tx=0x7fcdb401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a75757d8114f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbd59a640 1 -- 172.21.15.70:0/3755079420 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40b0070 con 0x7fcdc00cbf90 2024-04-22T01:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbd59a640 1 -- 172.21.15.70:0/3755079420 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb40ab600 con 0x7fcdc00cbf90 2024-04-22T01:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbd59a640 1 -- 172.21.15.70:0/3755079420 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40ab9d0 con 0x7fcdc00cbf90 2024-04-22T01:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3755079420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 msgr2=0x7fcdc00d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3755079420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc00d0620 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00b6a60 tx=0x7fcdb401c490 comp rx=0 tx=0).stop 2024-04-22T01:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3755079420 shutdown_connections 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3755079420 >> v2:172.21.15.98:3300/0 conn(0x7fcdc00cc3b0 0x7fcdc00d0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3755079420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc00d0620 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3755079420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc015ace0 0x7fcdc00cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3755079420 >> 172.21.15.70:0/3755079420 conn(0x7fcdc0099a60 msgr2=0x7fcdc0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3755079420 shutdown_connections 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3755079420 wait complete. 2024-04-22T01:15:47.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 Processor -- start 2024-04-22T01:15:47.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- start start 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc0152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc00cc3b0 0x7fcdc00b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdc015ace0 0x7fcdc006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc00b11f0 con 0x7fcdc00cbf90 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc00b1070 con 0x7fcdc015ace0 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdc4efd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc00b1370 con 0x7fcdc00cc3b0 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc0152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc0152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51090/0 (socket says 172.21.15.70:51090) 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 -- 172.21.15.70:0/3758574276 learned_addr learned my addr 172.21.15.70:0/3758574276 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbdd9b640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc00cc3b0 0x7fcdc00b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbed9d640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdc015ace0 0x7fcdc006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 -- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc00cc3b0 msgr2=0x7fcdc00b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc00cc3b0 0x7fcdc00b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 -- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdc015ace0 msgr2=0x7fcdc006b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdc015ace0 0x7fcdc006b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.944+0000 7fcdbe59c640 1 -- 172.21.15.70:0/3758574276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc00b51e0 con 0x7fcdc00cbf90 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdbe59c640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc0152e90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000d800 tx=0x7fcdb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000b6c0 con 0x7fcdc00cbf90 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb0019070 con 0x7fcdc00cbf90 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000ba60 con 0x7fcdc00cbf90 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc00b4f00 con 0x7fcdc00cbf90 2024-04-22T01:15:47.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc01f4f10 con 0x7fcdc00cbf90 2024-04-22T01:15:47.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc004f510 con 0x7fcdc00cbf90 2024-04-22T01:15:47.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcdb000bc00 con 0x7fcdc00cbf90 2024-04-22T01:15:47.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdaeffd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcda00678c0 0x7fcda0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:47.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdbdd9b640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcda00678c0 0x7fcda0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcdb0086550 con 0x7fcdc00cbf90 2024-04-22T01:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.948+0000 7fcdbdd9b640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcda00678c0 0x7fcda0069d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40ad000 tx=0x7fcdb4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:47.952+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcdb0056c90 con 0x7fcdc00cbf90 2024-04-22T01:15:48.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.128+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcdc01f4a20 con 0x7fcdc00cbf90 2024-04-22T01:15:48.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.128+0000 7fcdaeffd640 1 -- 172.21.15.70:0/3758574276 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fcdb0056590 con 0x7fcdc00cbf90 2024-04-22T01:15:48.131 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:48.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcda00678c0 msgr2=0x7fcda0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcda00678c0 0x7fcda0069d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40ad000 tx=0x7fcdb4008040 comp rx=0 tx=0).stop 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 msgr2=0x7fcdc0152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc0152e90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000d800 tx=0x7fcdb000dcc0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 shutdown_connections 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcda00678c0 0x7fcda0069d80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdc015ace0 0x7fcdc006b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdc00cc3b0 0x7fcdc00b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 --2- 172.21.15.70:0/3758574276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc00cbf90 0x7fcdc0152e90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 >> 172.21.15.70:0/3758574276 conn(0x7fcdc0099a60 msgr2=0x7fcdc0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 shutdown_connections 2024-04-22T01:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.132+0000 7fcdc4efd640 1 -- 172.21.15.70:0/3758574276 wait complete. 2024-04-22T01:15:48.150 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:15:48.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:48.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:15:48.185 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:15:48.185 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:15:48.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:48.186 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:15:48.260 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:15:48.260 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:48.266 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:15:48.266 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:48.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:48.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.480+0000 7f466073d640 1 Processor -- start 2024-04-22T01:15:48.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.480+0000 7f466073d640 1 -- start start 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46581455b0 0x7f46580cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f46580cc150 0x7f4658071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46580d0250 con 0x7f4658072060 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f46580724d0 con 0x7f46580cc150 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f46580cd7b0 con 0x7f46581455b0 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f46580cc150 0x7f4658071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f46580cc150 0x7f4658071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51104/0 (socket says 172.21.15.70:51104) 2024-04-22T01:15:48.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 -- 172.21.15.70:0/4096164998 learned_addr learned my addr 172.21.15.70:0/4096164998 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:48.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465e4d9640 1 --2- 172.21.15.70:0/4096164998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46581455b0 0x7f46580cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 -- 172.21.15.70:0/4096164998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46581455b0 msgr2=0x7f46580cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:48.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 --2- 172.21.15.70:0/4096164998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46581455b0 0x7f46580cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 -- 172.21.15.70:0/4096164998 >> v1:172.21.15.98:6789/0 conn(0x7f4658072060 legacy=0x7f465815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:48.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 -- 172.21.15.70:0/4096164998 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46580cc710 con 0x7f46580cc150 2024-04-22T01:15:48.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465dcd8640 1 --2- 172.21.15.70:0/4096164998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46580cc150 0x7f4658071b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4648009870 tx=0x7f464801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b61eab6056b6589 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:48.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465d4d7640 1 -- 172.21.15.70:0/4096164998 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46480b0070 con 0x7f46580cc150 2024-04-22T01:15:48.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465d4d7640 1 -- 172.21.15.70:0/4096164998 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46480ab600 con 0x7f46580cc150 2024-04-22T01:15:48.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f465d4d7640 1 -- 172.21.15.70:0/4096164998 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46480ab9d0 con 0x7f46580cc150 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 -- 172.21.15.70:0/4096164998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46580cc150 msgr2=0x7f4658071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.484+0000 7f466073d640 1 --2- 172.21.15.70:0/4096164998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46580cc150 0x7f4658071b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4648009870 tx=0x7f464801c400 comp rx=0 tx=0).stop 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/4096164998 shutdown_connections 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 --2- 172.21.15.70:0/4096164998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46580cc150 0x7f4658071b20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 --2- 172.21.15.70:0/4096164998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46581455b0 0x7f46580cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/4096164998 >> 172.21.15.70:0/4096164998 conn(0x7f465806b1f0 msgr2=0x7f465806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/4096164998 shutdown_connections 2024-04-22T01:15:48.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/4096164998 wait complete. 2024-04-22T01:15:48.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 Processor -- start 2024-04-22T01:15:48.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- start start 2024-04-22T01:15:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4658072060 0x7f465813a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46580cc150 0x7f4658141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 0x7f465813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4658168b40 con 0x7f46581455b0 2024-04-22T01:15:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f465806f210 con 0x7f46580cc150 2024-04-22T01:15:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4658168cc0 con 0x7f4658072060 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 0x7f465813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 0x7f465813f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51120/0 (socket says 172.21.15.70:51120) 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 -- 172.21.15.70:0/3840925653 learned_addr learned my addr 172.21.15.70:0/3840925653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465e4d9640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4658072060 0x7f465813a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465dcd8640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46580cc150 0x7f4658141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 -- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4658072060 msgr2=0x7f465813a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4658072060 0x7f465813a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 -- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46580cc150 msgr2=0x7f4658141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46580cc150 0x7f4658141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 -- 172.21.15.70:0/3840925653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465813afe0 con 0x7f46581455b0 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465dcd8640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46580cc150 0x7f4658141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f465ecda640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 0x7f465813f970 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f465000b970 tx=0x7f465000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650019070 con 0x7f46581455b0 2024-04-22T01:15:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f465000cb40 con 0x7f46581455b0 2024-04-22T01:15:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465813b2c0 con 0x7f46581455b0 2024-04-22T01:15:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650014440 con 0x7f46581455b0 2024-04-22T01:15:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f465813b880 con 0x7f46581455b0 2024-04-22T01:15:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.488+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4658068210 con 0x7f46581455b0 2024-04-22T01:15:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.492+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f465001c070 con 0x7f46581455b0 2024-04-22T01:15:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.492+0000 7f464effd640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f463c0678c0 0x7f463c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.492+0000 7f465e4d9640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f463c0678c0 0x7f463c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.492+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4650085f60 con 0x7f46581455b0 2024-04-22T01:15:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.492+0000 7f465e4d9640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f463c0678c0 0x7f463c069d80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f465400a970 tx=0x7f4654008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.496+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4650023080 con 0x7f46581455b0 2024-04-22T01:15:48.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.652+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f465815f510 con 0x7f46581455b0 2024-04-22T01:15:48.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f464effd640 1 -- 172.21.15.70:0/3840925653 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f4650056780 con 0x7f46581455b0 2024-04-22T01:15:48.659 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:48.659 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:48.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f463c0678c0 msgr2=0x7f463c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f463c0678c0 0x7f463c069d80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f465400a970 tx=0x7f4654008040 comp rx=0 tx=0).stop 2024-04-22T01:15:48.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 msgr2=0x7f465813f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 0x7f465813f970 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f465000b970 tx=0x7f465000be30 comp rx=0 tx=0).stop 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 shutdown_connections 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f463c0678c0 0x7f463c069d80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46581455b0 0x7f465813f970 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46580cc150 0x7f4658141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 --2- 172.21.15.70:0/3840925653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4658072060 0x7f465813a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 >> 172.21.15.70:0/3840925653 conn(0x7f465806b1f0 msgr2=0x7f465813c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:48.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 shutdown_connections 2024-04-22T01:15:48.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.656+0000 7f466073d640 1 -- 172.21.15.70:0/3840925653 wait complete. 2024-04-22T01:15:48.676 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:48.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:48.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.792+0000 7fb18c533640 1 Processor -- start 2024-04-22T01:15:48.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.792+0000 7fb18c533640 1 -- start start 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18c533640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 0x7fb184071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18c533640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb18406de60 con 0x7fb18406d8f0 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18c533640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb18406e140 con 0x7fb184158650 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18c533640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb18406e420 con 0x7fb18415ee80 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb189ace640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb18406d8f0 0x7fb18415f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38768/0 (socket says 172.21.15.70:38768) 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb189ace640 1 -- 172.21.15.70:0/1768875893 learned_addr learned my addr 172.21.15.70:0/1768875893 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:48.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18aad0640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 0x7fb184071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4289549151 0 0) 0x7fb18406de60 con 0x7fb18406d8f0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb170003530 con 0x7fb18406d8f0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1048161291 0 0) 0x7fb170003530 con 0x7fb18406d8f0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 msgr2=0x7fb184071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 0x7fb184071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 >> v1:172.21.15.98:6789/0 conn(0x7fb184158650 legacy=0x7fb18406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18406e700 con 0x7fb18406d8f0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fb174001680 con 0x7fb18406d8f0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 >> v1:172.21.15.70:6789/0 conn(0x7fb18406d8f0 legacy=0x7fb18415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb170003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 0x7fb170004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170004d50 0x7fb1700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb18406e420 con 0x7fb1700039f0 2024-04-22T01:15:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb17000bbb0 con 0x7fb170004d50 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb17000bdb0 con 0x7fb18415ee80 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb170003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb189ace640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 0x7fb170004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18aad0640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170004d50 0x7fb1700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 -- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 msgr2=0x7fb170004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb18415ee80 0x7fb170004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 -- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170004d50 msgr2=0x7fb1700051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170004d50 0x7fb1700051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 -- 172.21.15.70:0/1768875893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb170003530 con 0x7fb1700039f0 2024-04-22T01:15:48.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb170003e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb180001780 tx=0x7fb18001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f59a5dfafcc5708 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:48.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1800013a0 con 0x7fb1700039f0 2024-04-22T01:15:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1800052f0 con 0x7fb1700039f0 2024-04-22T01:15:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.796+0000 7fb1892cd640 1 -- 172.21.15.70:0/1768875893 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1800a4e60 con 0x7fb1700039f0 2024-04-22T01:15:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/1768875893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 msgr2=0x7fb170003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 --2- 172.21.15.70:0/1768875893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb170003e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb180001780 tx=0x7fb18001c420 comp rx=0 tx=0).stop 2024-04-22T01:15:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18aad0640 1 -- 172.21.15.70:0/1768875893 reap_dead start 2024-04-22T01:15:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/1768875893 shutdown_connections 2024-04-22T01:15:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/1768875893 >> 172.21.15.70:0/1768875893 conn(0x7fb1840b1d40 msgr2=0x7fb1840cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/1768875893 shutdown_connections 2024-04-22T01:15:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/1768875893 wait complete. 2024-04-22T01:15:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 Processor -- start 2024-04-22T01:15:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- start start 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb1841373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1841519a0 0x7fb18414ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb184149510 0x7fb184155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb18415fe50 con 0x7fb1700039f0 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb18415fcd0 con 0x7fb184149510 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb184169a40 con 0x7fb1841519a0 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb1841373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb1841373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51134/0 (socket says 172.21.15.70:51134) 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 -- 172.21.15.70:0/2642313138 learned_addr learned my addr 172.21.15.70:0/2642313138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb189ace640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1841519a0 0x7fb18414ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18aad0640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb184149510 0x7fb184155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 -- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1841519a0 msgr2=0x7fb18414ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1841519a0 0x7fb18414ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 -- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb184149510 msgr2=0x7fb184155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb184149510 0x7fb184155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 -- 172.21.15.70:0/2642313138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18414b340 con 0x7fb1700039f0 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18aad0640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb184149510 0x7fb184155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18a2cf640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb1841373d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb180005040 tx=0x7fb170005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180007a40 con 0x7fb1700039f0 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1800bb070 con 0x7fb1700039f0 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb184149d60 con 0x7fb1700039f0 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1800013a0 con 0x7fb1700039f0 2024-04-22T01:15:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18414a320 con 0x7fb1700039f0 2024-04-22T01:15:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.800+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb184068210 con 0x7fb1700039f0 2024-04-22T01:15:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.804+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb1800b4040 con 0x7fb1700039f0 2024-04-22T01:15:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.804+0000 7fb17affd640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1680678f0 0x7fb168069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.804+0000 7fb189ace640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1680678f0 0x7fb168069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.804+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb18011fda0 con 0x7fb1700039f0 2024-04-22T01:15:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.804+0000 7fb189ace640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1680678f0 0x7fb168069db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb174001ce0 tx=0x7fb174009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:48.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.808+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb18414b340 con 0x7fb1700039f0 2024-04-22T01:15:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.952+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb184152a30 con 0x7fb1700039f0 2024-04-22T01:15:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.952+0000 7fb17affd640 1 -- 172.21.15.70:0/2642313138 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb180016130 con 0x7fb1700039f0 2024-04-22T01:15:48.957 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1680678f0 msgr2=0x7fb168069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1680678f0 0x7fb168069db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb174001ce0 tx=0x7fb174009040 comp rx=0 tx=0).stop 2024-04-22T01:15:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 msgr2=0x7fb1841373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb1841373d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb180005040 tx=0x7fb170005840 comp rx=0 tx=0).stop 2024-04-22T01:15:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 shutdown_connections 2024-04-22T01:15:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1680678f0 0x7fb168069db0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb184149510 0x7fb184155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1841519a0 0x7fb18414ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 --2- 172.21.15.70:0/2642313138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1700039f0 0x7fb1841373d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 >> 172.21.15.70:0/2642313138 conn(0x7fb1840b1d40 msgr2=0x7fb1840cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 shutdown_connections 2024-04-22T01:15:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:48.956+0000 7fb18c533640 1 -- 172.21.15.70:0/2642313138 wait complete. 2024-04-22T01:15:48.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 Processor -- start 2024-04-22T01:15:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 -- start start 2024-04-22T01:15:49.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fccbc072280 0x7fccbc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 0x7fccbc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccbc15c290 con 0x7fccbc0b4450 2024-04-22T01:15:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fccbc15c570 con 0x7fccbc072280 2024-04-22T01:15:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccc1877640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc15c850 con 0x7fccbc0cd050 2024-04-22T01:15:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccba7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fccbc072280 0x7fccbc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccbaffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fccbc0b4450 0x7fccbc071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38772/0 (socket says 172.21.15.70:38772) 2024-04-22T01:15:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccbaffd640 1 -- 172.21.15.70:0/2427179047 learned_addr learned my addr 172.21.15.70:0/2427179047 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccba7fc640 1 --2- 172.21.15.70:0/2427179047 >> v2:172.21.15.70:3300/0 conn(0x7fccbc072280 0x7fccbc0ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51142/0 (socket says 172.21.15.70:51142) 2024-04-22T01:15:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccba7fc640 1 -- 172.21.15.70:0/2427179047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 msgr2=0x7fccbc0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccba7fc640 1 --2- 172.21.15.70:0/2427179047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 0x7fccbc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccba7fc640 1 -- 172.21.15.70:0/2427179047 >> v1:172.21.15.70:6789/0 conn(0x7fccbc0b4450 legacy=0x7fccbc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.084+0000 7fccba7fc640 1 -- 172.21.15.70:0/2427179047 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc15cb30 con 0x7fccbc072280 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccba7fc640 1 --2- 172.21.15.70:0/2427179047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc0ccb10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fccb00097c0 tx=0x7fccb001c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44affb0168515e57 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccb9ffb640 1 -- 172.21.15.70:0/2427179047 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00b0070 con 0x7fccbc072280 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccb9ffb640 1 -- 172.21.15.70:0/2427179047 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb00ab600 con 0x7fccbc072280 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccb9ffb640 1 -- 172.21.15.70:0/2427179047 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00ab9d0 con 0x7fccbc072280 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/2427179047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 msgr2=0x7fccbc0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 --2- 172.21.15.70:0/2427179047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc0ccb10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fccb00097c0 tx=0x7fccb001c2a0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/2427179047 shutdown_connections 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 --2- 172.21.15.70:0/2427179047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 0x7fccbc0b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 --2- 172.21.15.70:0/2427179047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc0ccb10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/2427179047 >> 172.21.15.70:0/2427179047 conn(0x7fccbc0b1d40 msgr2=0x7fccbc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/2427179047 shutdown_connections 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/2427179047 wait complete. 2024-04-22T01:15:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 Processor -- start 2024-04-22T01:15:49.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- start start 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc14ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccbc0b4450 0x7fccbc153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 0x7fccbc1505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc0cd980 con 0x7fccbc072280 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc0cd800 con 0x7fccbc0b4450 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc0b3790 con 0x7fccbc0cd050 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc14ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc14ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51158/0 (socket says 172.21.15.70:51158) 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 -- 172.21.15.70:0/3268062431 learned_addr learned my addr 172.21.15.70:0/3268062431 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 -- 172.21.15.70:0/3268062431 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 msgr2=0x7fccbc1505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 0x7fccbc1505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 -- 172.21.15.70:0/3268062431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccbc0b4450 msgr2=0x7fccbc153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccbc0b4450 0x7fccbc153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 -- 172.21.15.70:0/3268062431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc1546b0 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccbaffd640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc14ad90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fccac00c780 tx=0x7fccac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac018070 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccac0072d0 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac0047f0 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc154990 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.088+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc155230 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.092+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccbc154d80 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.092+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fccac01b070 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.092+0000 7fcca3fff640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcca80678c0 0x7fcca8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.092+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fccac0855c0 con 0x7fccbc072280 2024-04-22T01:15:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.092+0000 7fccba7fc640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcca80678c0 0x7fcca8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.092+0000 7fccba7fc640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcca80678c0 0x7fcca8069d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fccb00ad000 tx=0x7fccb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.096+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fccac055d00 con 0x7fccbc072280 2024-04-22T01:15:49.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fccbc068210 con 0x7fccbc072280 2024-04-22T01:15:49.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fcca3fff640 1 -- 172.21.15.70:0/3268062431 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fccac055600 con 0x7fccbc072280 2024-04-22T01:15:49.254 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:49.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcca80678c0 msgr2=0x7fcca8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcca80678c0 0x7fcca8069d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fccb00ad000 tx=0x7fccb0008040 comp rx=0 tx=0).stop 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 msgr2=0x7fccbc14ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc14ad90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fccac00c780 tx=0x7fccac00cc40 comp rx=0 tx=0).stop 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 shutdown_connections 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcca80678c0 0x7fcca8069d80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccbc0cd050 0x7fccbc1505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccbc0b4450 0x7fccbc153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 --2- 172.21.15.70:0/3268062431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccbc072280 0x7fccbc14ad90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 >> 172.21.15.70:0/3268062431 conn(0x7fccbc0b1d40 msgr2=0x7fccbc0d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 shutdown_connections 2024-04-22T01:15:49.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.252+0000 7fccc1877640 1 -- 172.21.15.70:0/3268062431 wait complete. 2024-04-22T01:15:49.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:49.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 Processor -- start 2024-04-22T01:15:49.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 -- start start 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7e74071b70 0x7f7e74072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e740cd410 0x7f7e740cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7406da70 con 0x7f7e7415a320 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7e7406f860 con 0x7f7e74071b70 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e78f42640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7406dd50 con 0x7f7e740cd410 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7e74071b70 0x7f7e74072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7e74071b70 0x7f7e74072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51160/0 (socket says 172.21.15.70:51160) 2024-04-22T01:15:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 -- 172.21.15.70:0/644969935 learned_addr learned my addr 172.21.15.70:0/644969935 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e72d9d640 1 --2- 172.21.15.70:0/644969935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e740cd410 0x7f7e740cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 -- 172.21.15.70:0/644969935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e740cd410 msgr2=0x7f7e740cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 --2- 172.21.15.70:0/644969935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e740cd410 0x7f7e740cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 -- 172.21.15.70:0/644969935 >> v1:172.21.15.70:6789/0 conn(0x7f7e7415a320 legacy=0x7f7e7406f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.388+0000 7f7e71d9b640 1 -- 172.21.15.70:0/644969935 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7406e030 con 0x7f7e74071b70 2024-04-22T01:15:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 --2- 172.21.15.70:0/644969935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e74071b70 0x7f7e74072000 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e6801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc065534d7992e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e7159a640 1 -- 172.21.15.70:0/644969935 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680b1070 con 0x7f7e74071b70 2024-04-22T01:15:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e7159a640 1 -- 172.21.15.70:0/644969935 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e680ac630 con 0x7f7e74071b70 2024-04-22T01:15:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e7159a640 1 -- 172.21.15.70:0/644969935 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680a8050 con 0x7f7e74071b70 2024-04-22T01:15:49.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/644969935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e74071b70 msgr2=0x7f7e74072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 --2- 172.21.15.70:0/644969935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e74071b70 0x7f7e74072000 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e6801c560 comp rx=0 tx=0).stop 2024-04-22T01:15:49.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/644969935 shutdown_connections 2024-04-22T01:15:49.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 --2- 172.21.15.70:0/644969935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e740cd410 0x7f7e740cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 --2- 172.21.15.70:0/644969935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e74071b70 0x7f7e74072000 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/644969935 >> 172.21.15.70:0/644969935 conn(0x7f7e740b1d40 msgr2=0x7f7e740cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/644969935 shutdown_connections 2024-04-22T01:15:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/644969935 wait complete. 2024-04-22T01:15:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 Processor -- start 2024-04-22T01:15:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- start start 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e74071b70 0x7f7e74141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 0x7f7e7414a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7415a320 0x7f7e74165f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740cf5a0 con 0x7f7e740cd410 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740cf420 con 0x7f7e7415a320 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7e740cddc0 con 0x7f7e74071b70 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 0x7f7e7414a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 0x7f7e7414a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51162/0 (socket says 172.21.15.70:51162) 2024-04-22T01:15:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 -- 172.21.15.70:0/2164991454 learned_addr learned my addr 172.21.15.70:0/2164991454 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e7259c640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e74071b70 0x7f7e74141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 -- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e74071b70 msgr2=0x7f7e74141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e74071b70 0x7f7e74141490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 -- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7415a320 msgr2=0x7f7e74165f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e72d9d640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7415a320 0x7f7e74165f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7415a320 0x7f7e74165f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 -- 172.21.15.70:0/2164991454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7414ad00 con 0x7f7e740cd410 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e71d9b640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 0x7f7e7414a320 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7e680af040 tx=0x7f7e68019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680bc070 con 0x7f7e740cd410 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e680ab5e0 con 0x7f7e740cd410 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7414afe0 con 0x7f7e740cd410 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680b1070 con 0x7f7e740cd410 2024-04-22T01:15:49.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.392+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7414b5a0 con 0x7f7e740cd410 2024-04-22T01:15:49.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.396+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e74068210 con 0x7f7e740cd410 2024-04-22T01:15:49.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.396+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7e680bf070 con 0x7f7e740cd410 2024-04-22T01:15:49.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.396+0000 7f7e62ffd640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7e540678c0 0x7f7e54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.396+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7e6811f9b0 con 0x7f7e740cd410 2024-04-22T01:15:49.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.396+0000 7f7e7259c640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7e540678c0 0x7f7e54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.396+0000 7f7e7259c640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7e540678c0 0x7f7e54069d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e64007fd0 tx=0x7f7e64009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.400+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7e68016150 con 0x7f7e740cd410 2024-04-22T01:15:49.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.544+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7e740b3340 con 0x7f7e740cd410 2024-04-22T01:15:49.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e62ffd640 1 -- 172.21.15.70:0/2164991454 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f7e680f00f0 con 0x7f7e740cd410 2024-04-22T01:15:49.550 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:49.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7e540678c0 msgr2=0x7f7e54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7e540678c0 0x7f7e54069d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e64007fd0 tx=0x7f7e64009040 comp rx=0 tx=0).stop 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 msgr2=0x7f7e7414a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 0x7f7e7414a320 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7e680af040 tx=0x7f7e68019300 comp rx=0 tx=0).stop 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 shutdown_connections 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7e540678c0 0x7f7e54069d80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7415a320 0x7f7e74165f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e740cd410 0x7f7e7414a320 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 --2- 172.21.15.70:0/2164991454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e74071b70 0x7f7e74141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 >> 172.21.15.70:0/2164991454 conn(0x7f7e740b1d40 msgr2=0x7f7e74158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 shutdown_connections 2024-04-22T01:15:49.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.548+0000 7f7e78f42640 1 -- 172.21.15.70:0/2164991454 wait complete. 2024-04-22T01:15:49.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:49.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574af81640 1 Processor -- start 2024-04-22T01:15:49.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574af81640 1 -- start start 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574af81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f574406b1f0 0x7f574406b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574af81640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57440d1a70 con 0x7f5744161970 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574af81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574406bb10 con 0x7f574406efb0 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574af81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f57440cd570 con 0x7f574406b1f0 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574951e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5744161970 0x7f5744161e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38788/0 (socket says 172.21.15.70:38788) 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574951e640 1 -- 172.21.15.70:0/731631023 learned_addr learned my addr 172.21.15.70:0/731631023 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 --2- 172.21.15.70:0/731631023 >> v2:172.21.15.98:3300/0 conn(0x7f574406b1f0 0x7f574406b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620381032 0 0) 0x7f57440d1a70 con 0x7f5744161970 2024-04-22T01:15:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5734003530 con 0x7f5744161970 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2559236421 0 0) 0x7f5734003530 con 0x7f5744161970 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 >> v2:172.21.15.98:3300/0 conn(0x7f574406b1f0 msgr2=0x7f574406b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 --2- 172.21.15.70:0/731631023 >> v2:172.21.15.98:3300/0 conn(0x7f574406b1f0 0x7f574406b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 >> v1:172.21.15.98:6789/0 conn(0x7f574406efb0 legacy=0x7f574406f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57440d0cb0 con 0x7f5744161970 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f573c001a00 con 0x7f5744161970 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 >> v1:172.21.15.70:6789/0 conn(0x7f5744161970 legacy=0x7f5744161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57340039c0 0x7f5734003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734004810 0x7f5734004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57340051d0 0x7f573400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f573400c040 con 0x7f57340039c0 2024-04-22T01:15:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f573400c1f0 con 0x7f5734004810 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f573400c3f0 con 0x7f57340051d0 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57340039c0 0x7f5734003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5743fff640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734004810 0x7f5734004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574951e640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57340051d0 0x7f573400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 -- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57340051d0 msgr2=0x7f573400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57340051d0 0x7f573400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 -- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734004810 msgr2=0x7f5734004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734004810 0x7f5734004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f574951e640 1 -- 172.21.15.70:0/731631023 reap_dead start 2024-04-22T01:15:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5748d1d640 1 -- 172.21.15.70:0/731631023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5734003530 con 0x7f57340039c0 2024-04-22T01:15:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.688+0000 7f5743fff640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734004810 0x7f5734004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f5748d1d640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57340039c0 0x7f5734003de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5738009ac0 tx=0x7f573801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8fe7a8a3268f02 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57380b2070 con 0x7f57340039c0 2024-04-22T01:15:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57380a55c0 con 0x7f57340039c0 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f57437fe640 1 -- 172.21.15.70:0/731631023 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738009690 con 0x7f57340039c0 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 -- 172.21.15.70:0/731631023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57340039c0 msgr2=0x7f5734003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57340039c0 0x7f5734003de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5738009ac0 tx=0x7f573801ca50 comp rx=0 tx=0).stop 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 -- 172.21.15.70:0/731631023 shutdown_connections 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 --2- 172.21.15.70:0/731631023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57340039c0 0x7f5734003de0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 -- 172.21.15.70:0/731631023 >> 172.21.15.70:0/731631023 conn(0x7f57440733e0 msgr2=0x7f5744099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 -- 172.21.15.70:0/731631023 shutdown_connections 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 -- 172.21.15.70:0/731631023 wait complete. 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 Processor -- start 2024-04-22T01:15:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.692+0000 7f574af81640 1 -- start start 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340039c0 0x7f57441530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57440c47b0 0x7f57440c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 0x7f574415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f574415e0b0 con 0x7f57440c5170 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57440d15e0 con 0x7f57340039c0 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f574415e230 con 0x7f57440c47b0 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 0x7f574415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 0x7f574415e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51184/0 (socket says 172.21.15.70:51184) 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 -- 172.21.15.70:0/656579269 learned_addr learned my addr 172.21.15.70:0/656579269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f5748d1d640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340039c0 0x7f57441530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f5743fff640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57440c47b0 0x7f57440c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 -- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57440c47b0 msgr2=0x7f57440c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57440c47b0 0x7f57440c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 -- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340039c0 msgr2=0x7f57441530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340039c0 0x7f57441530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 -- 172.21.15.70:0/656579269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57440c7630 con 0x7f57440c5170 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f5743fff640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57440c47b0 0x7f57440c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574951e640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 0x7f574415e3b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f57440c7ce0 tx=0x7f573c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c004d70 con 0x7f57440c5170 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573c00bb20 con 0x7f57440c5170 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c007110 con 0x7f57440c5170 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57440b5c20 con 0x7f57440c5170 2024-04-22T01:15:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57440b61e0 con 0x7f57440c5170 2024-04-22T01:15:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5744068920 con 0x7f57440c5170 2024-04-22T01:15:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f573c01b020 con 0x7f57440c5170 2024-04-22T01:15:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f57417fa640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57280678c0 0x7f5728069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f573c086240 con 0x7f57440c5170 2024-04-22T01:15:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f5748d1d640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57280678c0 0x7f5728069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.696+0000 7f5748d1d640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57280678c0 0x7f5728069d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f57380af000 tx=0x7f57380046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.700+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f573c056980 con 0x7f57440c5170 2024-04-22T01:15:49.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.848+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5744161630 con 0x7f57440c5170 2024-04-22T01:15:49.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.848+0000 7f57417fa640 1 -- 172.21.15.70:0/656579269 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f573c056280 con 0x7f57440c5170 2024-04-22T01:15:49.853 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:49.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57280678c0 msgr2=0x7f5728069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57280678c0 0x7f5728069d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f57380af000 tx=0x7f57380046d0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 msgr2=0x7f574415e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 0x7f574415e3b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f57440c7ce0 tx=0x7f573c00b4a0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 shutdown_connections 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57280678c0 0x7f5728069d80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57440c5170 0x7f574415e3b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57440c47b0 0x7f57440c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 --2- 172.21.15.70:0/656579269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340039c0 0x7f57441530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 >> 172.21.15.70:0/656579269 conn(0x7f57440733e0 msgr2=0x7f57440d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 shutdown_connections 2024-04-22T01:15:49.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.852+0000 7f574af81640 1 -- 172.21.15.70:0/656579269 wait complete. 2024-04-22T01:15:49.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:49.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 Processor -- start 2024-04-22T01:15:49.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 -- start start 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f576015e1e0 0x7f57600d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f576006d5b0 0x7f5760099c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57600d1360 con 0x7f576009a190 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f576009a5e0 con 0x7f576015e1e0 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5764d13640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f57600cc1c0 con 0x7f576006d5b0 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f576015e1e0 0x7f57600d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f576015e1e0 0x7f57600d09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51190/0 (socket says 172.21.15.70:51190) 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 -- 172.21.15.70:0/2306623689 learned_addr learned my addr 172.21.15.70:0/2306623689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f5757fff640 1 --2- 172.21.15.70:0/2306623689 >> v2:172.21.15.98:3300/0 conn(0x7f576006d5b0 0x7f5760099c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 -- 172.21.15.70:0/2306623689 >> v2:172.21.15.98:3300/0 conn(0x7f576006d5b0 msgr2=0x7f5760099c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 --2- 172.21.15.70:0/2306623689 >> v2:172.21.15.98:3300/0 conn(0x7f576006d5b0 0x7f5760099c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 -- 172.21.15.70:0/2306623689 >> v1:172.21.15.98:6789/0 conn(0x7f576009a190 legacy=0x7f576006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:49.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.988+0000 7f575e59c640 1 -- 172.21.15.70:0/2306623689 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57600d1640 con 0x7f576015e1e0 2024-04-22T01:15:49.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575e59c640 1 --2- 172.21.15.70:0/2306623689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f57600d09f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f576006e630 tx=0x7f575001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91c7a7df4ec9d14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575dd9b640 1 -- 172.21.15.70:0/2306623689 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500b2070 con 0x7f576015e1e0 2024-04-22T01:15:49.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575dd9b640 1 -- 172.21.15.70:0/2306623689 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57500a9070 con 0x7f576015e1e0 2024-04-22T01:15:49.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575dd9b640 1 -- 172.21.15.70:0/2306623689 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500ad880 con 0x7f576015e1e0 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- 172.21.15.70:0/2306623689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 msgr2=0x7f57600d09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 --2- 172.21.15.70:0/2306623689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f57600d09f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f576006e630 tx=0x7f575001c450 comp rx=0 tx=0).stop 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- 172.21.15.70:0/2306623689 shutdown_connections 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 --2- 172.21.15.70:0/2306623689 >> v2:172.21.15.98:3300/0 conn(0x7f576006d5b0 0x7f5760099c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 --2- 172.21.15.70:0/2306623689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f57600d09f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- 172.21.15.70:0/2306623689 >> 172.21.15.70:0/2306623689 conn(0x7f57600b3660 msgr2=0x7f57600b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- 172.21.15.70:0/2306623689 shutdown_connections 2024-04-22T01:15:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- 172.21.15.70:0/2306623689 wait complete. 2024-04-22T01:15:49.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 Processor -- start 2024-04-22T01:15:49.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- start start 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f576006d5b0 0x7f57600b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f576009a190 0x7f5760151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f5760151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f576016afa0 con 0x7f576015e1e0 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f576016ae20 con 0x7f576009a190 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5764d13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f576016b120 con 0x7f576006d5b0 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f5760151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f5760151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51200/0 (socket says 172.21.15.70:51200) 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 -- 172.21.15.70:0/806296980 learned_addr learned my addr 172.21.15.70:0/806296980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575e59c640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f576006d5b0 0x7f57600b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f5757fff640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f576009a190 0x7f5760151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 -- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f576006d5b0 msgr2=0x7f57600b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f576006d5b0 0x7f57600b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 -- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f576009a190 msgr2=0x7f5760151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.992+0000 7f575ed9d640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f576009a190 0x7f5760151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f575ed9d640 1 -- 172.21.15.70:0/806296980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57600afd50 con 0x7f576015e1e0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f575ed9d640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f5760151550 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f575800b970 tx=0x7f575800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758019070 con 0x7f576015e1e0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f575800cb40 con 0x7f576015e1e0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758014440 con 0x7f576015e1e0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57600b0030 con 0x7f576015e1e0 2024-04-22T01:15:49.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57600b05f0 con 0x7f576015e1e0 2024-04-22T01:15:50.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57600c8cf0 con 0x7f576015e1e0 2024-04-22T01:15:50.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f575801c070 con 0x7f576015e1e0 2024-04-22T01:15:50.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5756ffd640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57400678c0 0x7f5740069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f575e59c640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57400678c0 0x7f5740069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5758085cb0 con 0x7f576015e1e0 2024-04-22T01:15:50.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:49.996+0000 7f575e59c640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57400678c0 0x7f5740069d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f57500af000 tx=0x7f5750008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.000+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5758056480 con 0x7f576015e1e0 2024-04-22T01:15:50.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.172+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f57600b1140 con 0x7f576015e1e0 2024-04-22T01:15:50.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5756ffd640 1 -- 172.21.15.70:0/806296980 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f5758055d80 con 0x7f576015e1e0 2024-04-22T01:15:50.177 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:50.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57400678c0 msgr2=0x7f5740069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57400678c0 0x7f5740069d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f57500af000 tx=0x7f5750008040 comp rx=0 tx=0).stop 2024-04-22T01:15:50.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 msgr2=0x7f5760151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f5760151550 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f575800b970 tx=0x7f575800be30 comp rx=0 tx=0).stop 2024-04-22T01:15:50.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 shutdown_connections 2024-04-22T01:15:50.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f57400678c0 0x7f5740069d80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576015e1e0 0x7f5760151550 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f576009a190 0x7f5760151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 --2- 172.21.15.70:0/806296980 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f576006d5b0 0x7f57600b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 >> 172.21.15.70:0/806296980 conn(0x7f57600b3660 msgr2=0x7f5760071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 shutdown_connections 2024-04-22T01:15:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.176+0000 7f5764d13640 1 -- 172.21.15.70:0/806296980 wait complete. 2024-04-22T01:15:50.195 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:15:50.196 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:50.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.360+0000 7ff564564640 1 Processor -- start 2024-04-22T01:15:50.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.360+0000 7ff564564640 1 -- start start 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff564564640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff55c0d0f00 0x7ff55c15e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff564564640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff55c0cebc0 con 0x7ff55c15ac40 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff564564640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff55c15eec0 con 0x7ff55c0ce730 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff564564640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff55c0ad9b0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff562300640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff55c15ac40 0x7ff55c0d07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38796/0 (socket says 172.21.15.70:38796) 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff562300640 1 -- 172.21.15.70:0/748870953 learned_addr learned my addr 172.21.15.70:0/748870953 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:50.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff561aff640 1 --2- 172.21.15.70:0/748870953 >> v2:172.21.15.70:3300/0 conn(0x7ff55c0d0f00 0x7ff55c15e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff561aff640 1 -- 172.21.15.70:0/748870953 >> v1:172.21.15.98:6789/0 conn(0x7ff55c0ce730 legacy=0x7ff55c0ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff561aff640 1 -- 172.21.15.70:0/748870953 >> v1:172.21.15.70:6789/0 conn(0x7ff55c15ac40 legacy=0x7ff55c0d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff561aff640 1 -- 172.21.15.70:0/748870953 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c15f180 con 0x7ff55c0d0f00 2024-04-22T01:15:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff5612fe640 1 -- 172.21.15.70:0/748870953 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969112366 0 0) 0x7ff55c0cebc0 con 0x7ff55c15ac40 2024-04-22T01:15:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff561aff640 1 --2- 172.21.15.70:0/748870953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15e7b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f79e9a43bcd7b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff5612fe640 1 -- 172.21.15.70:0/748870953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b0070 con 0x7ff55c0d0f00 2024-04-22T01:15:50.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff5612fe640 1 -- 172.21.15.70:0/748870953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5480ab600 con 0x7ff55c0d0f00 2024-04-22T01:15:50.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.364+0000 7ff5612fe640 1 -- 172.21.15.70:0/748870953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480ab9d0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/748870953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 msgr2=0x7ff55c15e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 --2- 172.21.15.70:0/748870953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15e7b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801c460 comp rx=0 tx=0).stop 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/748870953 shutdown_connections 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 --2- 172.21.15.70:0/748870953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15e7b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/748870953 >> 172.21.15.70:0/748870953 conn(0x7ff55c099a60 msgr2=0x7ff55c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/748870953 shutdown_connections 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/748870953 wait complete. 2024-04-22T01:15:50.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 Processor -- start 2024-04-22T01:15:50.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- start start 2024-04-22T01:15:50.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0ce730 0x7ff55c155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c15ac40 0x7ff55c148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c0af0f0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c0aef70 con 0x7ff55c15ac40 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff55c168b00 con 0x7ff55c0ce730 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51218/0 (socket says 172.21.15.70:51218) 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 -- 172.21.15.70:0/4082790321 learned_addr learned my addr 172.21.15.70:0/4082790321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 -- 172.21.15.70:0/4082790321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0ce730 msgr2=0x7ff55c155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0ce730 0x7ff55c155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 -- 172.21.15.70:0/4082790321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c15ac40 msgr2=0x7ff55c148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c15ac40 0x7ff55c148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 -- 172.21.15.70:0/4082790321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c0cd7d0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff561aff640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15b2b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b0050 con 0x7ff55c0d0f00 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5480bc070 con 0x7ff55c0d0f00 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55c0cdab0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b5a70 con 0x7ff55c0d0f00 2024-04-22T01:15:50.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55c14abc0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.368+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff55c14a6d0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.372+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff5480ab480 con 0x7ff55c0d0f00 2024-04-22T01:15:50.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.372+0000 7ff55affd640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5400678c0 0x7ff540069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.372+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff5481209c0 con 0x7ff55c0d0f00 2024-04-22T01:15:50.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.372+0000 7ff562300640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5400678c0 0x7ff540069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.372+0000 7ff562300640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5400678c0 0x7ff540069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff550001570 tx=0x7ff550007cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.376+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff548122e20 con 0x7ff55c0d0f00 2024-04-22T01:15:50.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.544+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff55c0cdc80 con 0x7ff55c0d0f00 2024-04-22T01:15:50.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff55affd640 1 -- 172.21.15.70:0/4082790321 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7ff55c0cdc80 con 0x7ff55c0d0f00 2024-04-22T01:15:50.550 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:50.550 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5400678c0 msgr2=0x7ff540069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5400678c0 0x7ff540069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff550001570 tx=0x7ff550007cb0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 msgr2=0x7ff55c15b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15b2b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801aca0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 shutdown_connections 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5400678c0 0x7ff540069d80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c15ac40 0x7ff55c148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff55c0d0f00 0x7ff55c15b2b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 --2- 172.21.15.70:0/4082790321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0ce730 0x7ff55c155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 >> 172.21.15.70:0/4082790321 conn(0x7ff55c099a60 msgr2=0x7ff55c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 shutdown_connections 2024-04-22T01:15:50.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.548+0000 7ff564564640 1 -- 172.21.15.70:0/4082790321 wait complete. 2024-04-22T01:15:50.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:50.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:50.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff079382640 1 Processor -- start 2024-04-22T01:15:50.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff079382640 1 -- start start 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff079382640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 0x7ff0740cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff079382640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0740cff80 con 0x7ff0740b18e0 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff079382640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0740d0260 con 0x7ff0740cd320 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff079382640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0740d0540 con 0x7ff0741625a0 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff073fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff0740b18e0 0x7ff0740b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38806/0 (socket says 172.21.15.70:38806) 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff073fff640 1 -- 172.21.15.70:0/1630861897 learned_addr learned my addr 172.21.15.70:0/1630861897 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225498099 0 0) 0x7ff0740cff80 con 0x7ff0740b18e0 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff078b81640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 0x7ff0740cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff060003530 con 0x7ff0740b18e0 2024-04-22T01:15:50.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3857361399 0 0) 0x7ff060003530 con 0x7ff0740b18e0 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 msgr2=0x7ff0740cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 0x7ff0740cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 >> v1:172.21.15.98:6789/0 conn(0x7ff0740cd320 legacy=0x7ff074161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff074161970 con 0x7ff0740b18e0 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7ff064001680 con 0x7ff0740b18e0 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 >> v1:172.21.15.70:6789/0 conn(0x7ff0740b18e0 legacy=0x7ff0740b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 0x7ff0600039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff060003f70 0x7ff060004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff060004d50 0x7ff0600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff0740d0540 con 0x7ff060003f70 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff06000bab0 con 0x7ff060004d50 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff06000bcb0 con 0x7ff0741625a0 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff0737fe640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff060003f70 0x7ff060004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff0737fe640 1 -- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 msgr2=0x7ff0600039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:50.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff0737fe640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0741625a0 0x7ff0600039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff0737fe640 1 -- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff060004d50 msgr2=0x7ff0600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:50.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff0737fe640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff060004d50 0x7ff0600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.688+0000 7ff0737fe640 1 -- 172.21.15.70:0/1630861897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff060003530 con 0x7ff060003f70 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff0737fe640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff060003f70 0x7ff060004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff06c001e70 tx=0x7ff06c01c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d4ae30a29dd8b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c001a90 con 0x7ff060003f70 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff06c003b80 con 0x7ff060003f70 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff072ffd640 1 -- 172.21.15.70:0/1630861897 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c005080 con 0x7ff060003f70 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- 172.21.15.70:0/1630861897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff060003f70 msgr2=0x7ff060004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 --2- 172.21.15.70:0/1630861897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff060003f70 0x7ff060004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff06c001e70 tx=0x7ff06c01c780 comp rx=0 tx=0).stop 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff078b81640 1 -- 172.21.15.70:0/1630861897 reap_dead start 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- 172.21.15.70:0/1630861897 shutdown_connections 2024-04-22T01:15:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- 172.21.15.70:0/1630861897 >> 172.21.15.70:0/1630861897 conn(0x7ff074099a60 msgr2=0x7ff074099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:50.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- 172.21.15.70:0/1630861897 shutdown_connections 2024-04-22T01:15:50.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- 172.21.15.70:0/1630861897 wait complete. 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 Processor -- start 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- start start 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff060003f70 0x7ff0740c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 0x7ff07415f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0740aee60 0x7ff07415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff0740cfc10 con 0x7ff0740ae9e0 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0740afc80 con 0x7ff0740aee60 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff079382640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0740cdbf0 con 0x7ff060003f70 2024-04-22T01:15:50.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff0737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 0x7ff07415f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff0737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 0x7ff07415f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51238/0 (socket says 172.21.15.70:51238) 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.692+0000 7ff0737fe640 1 -- 172.21.15.70:0/3494122229 learned_addr learned my addr 172.21.15.70:0/3494122229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff073fff640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff060003f70 0x7ff0740c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff078b81640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0740aee60 0x7ff07415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff0737fe640 1 -- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff060003f70 msgr2=0x7ff0740c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff0737fe640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff060003f70 0x7ff0740c63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff0737fe640 1 -- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0740aee60 msgr2=0x7ff07415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff0737fe640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0740aee60 0x7ff07415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff0737fe640 1 -- 172.21.15.70:0/3494122229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0740c47b0 con 0x7ff0740ae9e0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff0737fe640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 0x7ff07415f560 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff06c01cc60 tx=0x7ff060005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c001a90 con 0x7ff0740ae9e0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff06c005710 con 0x7ff0740ae9e0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0740c4a90 con 0x7ff0740ae9e0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0a6680 con 0x7ff0740ae9e0 2024-04-22T01:15:50.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0740c5050 con 0x7ff0740ae9e0 2024-04-22T01:15:50.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07404f510 con 0x7ff0740ae9e0 2024-04-22T01:15:50.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff06c0a6820 con 0x7ff0740ae9e0 2024-04-22T01:15:50.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff070ff9640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0540678c0 0x7ff054069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:50.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff06c1206b0 con 0x7ff0740ae9e0 2024-04-22T01:15:50.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff073fff640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0540678c0 0x7ff054069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:50.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.696+0000 7ff073fff640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0540678c0 0x7ff054069d80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff064004880 tx=0x7ff064001540 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:50.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.700+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff06c0b6070 con 0x7ff0740ae9e0 2024-04-22T01:15:50.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.860+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff074073690 con 0x7ff0740ae9e0 2024-04-22T01:15:50.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.860+0000 7ff070ff9640 1 -- 172.21.15.70:0/3494122229 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7ff06c0ac020 con 0x7ff0740ae9e0 2024-04-22T01:15:50.865 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:50.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0540678c0 msgr2=0x7ff054069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0540678c0 0x7ff054069d80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff064004880 tx=0x7ff064001540 comp rx=0 tx=0).stop 2024-04-22T01:15:50.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 msgr2=0x7ff07415f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:50.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 0x7ff07415f560 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff06c01cc60 tx=0x7ff060005840 comp rx=0 tx=0).stop 2024-04-22T01:15:50.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 shutdown_connections 2024-04-22T01:15:50.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0540678c0 0x7ff054069d80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0740aee60 0x7ff07415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0740ae9e0 0x7ff07415f560 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 --2- 172.21.15.70:0/3494122229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff060003f70 0x7ff0740c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:50.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 >> 172.21.15.70:0/3494122229 conn(0x7ff074099a60 msgr2=0x7ff07415e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:50.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 shutdown_connections 2024-04-22T01:15:50.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:50.864+0000 7ff079382640 1 -- 172.21.15.70:0/3494122229 wait complete. 2024-04-22T01:15:50.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.004+0000 7f55cbf61640 1 Processor -- start 2024-04-22T01:15:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.004+0000 7f55cbf61640 1 -- start start 2024-04-22T01:15:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55cbf61640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c4072280 0x7f55c40ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55cbf61640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55c40cd050 0x7f55c40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55cbf61640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55c415c290 con 0x7f55c40b4450 2024-04-22T01:15:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55cbf61640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f55c415c570 con 0x7f55c40cd050 2024-04-22T01:15:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55cbf61640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f55c415c850 con 0x7f55c4072280 2024-04-22T01:15:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c9cfd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f55c40b4450 0x7f55c4071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38816/0 (socket says 172.21.15.70:38816) 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55ca4fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55c40cd050 0x7f55c40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c9cfd640 1 -- 172.21.15.70:0/3876420887 learned_addr learned my addr 172.21.15.70:0/3876420887 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c94fc640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c4072280 0x7f55c40ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55ca4fe640 1 -- 172.21.15.70:0/3876420887 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c4072280 msgr2=0x7f55c40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55ca4fe640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c4072280 0x7f55c40ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55ca4fe640 1 -- 172.21.15.70:0/3876420887 >> v1:172.21.15.70:6789/0 conn(0x7f55c40b4450 legacy=0x7f55c4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55ca4fe640 1 -- 172.21.15.70:0/3876420887 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c415cb30 con 0x7f55c40cd050 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c8cfb640 1 -- 172.21.15.70:0/3876420887 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912115349 0 0) 0x7f55c415c290 con 0x7f55c40b4450 2024-04-22T01:15:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c94fc640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c4072280 0x7f55c40ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55ca4fe640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c40cd050 0x7f55c40b2a40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009d90 tx=0x7f55bc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d4a90c49fa9f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c8cfb640 1 -- 172.21.15.70:0/3876420887 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0b2070 con 0x7f55c40cd050 2024-04-22T01:15:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c8cfb640 1 -- 172.21.15.70:0/3876420887 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55bc0a59c0 con 0x7f55c40cd050 2024-04-22T01:15:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.008+0000 7f55c8cfb640 1 -- 172.21.15.70:0/3876420887 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0a5d90 con 0x7f55c40cd050 2024-04-22T01:15:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/3876420887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c40cd050 msgr2=0x7f55c40b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c40cd050 0x7f55c40b2a40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009d90 tx=0x7f55bc01c8e0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/3876420887 shutdown_connections 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c40cd050 0x7f55c40b2a40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 --2- 172.21.15.70:0/3876420887 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c4072280 0x7f55c40ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/3876420887 >> 172.21.15.70:0/3876420887 conn(0x7f55c40b1d40 msgr2=0x7f55c40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/3876420887 shutdown_connections 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/3876420887 wait complete. 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 Processor -- start 2024-04-22T01:15:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- start start 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 0x7f55c406e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c40b4450 0x7f55c4148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55c40cd050 0x7f55c4153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55c40cd980 con 0x7f55c4072280 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55c40cd800 con 0x7f55c40cd050 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f55c40b3790 con 0x7f55c40b4450 2024-04-22T01:15:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 0x7f55c406e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 0x7f55c406e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51258/0 (socket says 172.21.15.70:51258) 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 -- 172.21.15.70:0/4189980392 learned_addr learned my addr 172.21.15.70:0/4189980392 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 -- 172.21.15.70:0/4189980392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c40b4450 msgr2=0x7f55c4148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c40b4450 0x7f55c4148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 -- 172.21.15.70:0/4189980392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55c40cd050 msgr2=0x7f55c4153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55c40cd050 0x7f55c4153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 -- 172.21.15.70:0/4189980392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c4154470 con 0x7f55c4072280 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55c9cfd640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 0x7f55c406e300 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f55c00011f0 tx=0x7f55c000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0018070 con 0x7f55c4072280 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c00074e0 con 0x7f55c4072280 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c406e8e0 con 0x7f55c4072280 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0004a10 con 0x7f55c4072280 2024-04-22T01:15:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c406eea0 con 0x7f55c4072280 2024-04-22T01:15:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.012+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c4068210 con 0x7f55c4072280 2024-04-22T01:15:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.016+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f55c001b070 con 0x7f55c4072280 2024-04-22T01:15:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.016+0000 7f55ba7fc640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55a80678c0 0x7f55a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.016+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f55c0085340 con 0x7f55c4072280 2024-04-22T01:15:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.016+0000 7f55c94fc640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55a80678c0 0x7f55a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.016+0000 7f55c94fc640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55a80678c0 0x7f55a8069d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f55c414c570 tx=0x7f55b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.020+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f55c0055b10 con 0x7f55c4072280 2024-04-22T01:15:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.168+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55c40d01e0 con 0x7f55c4072280 2024-04-22T01:15:51.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.168+0000 7f55ba7fc640 1 -- 172.21.15.70:0/4189980392 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f55c0055410 con 0x7f55c4072280 2024-04-22T01:15:51.173 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55a80678c0 msgr2=0x7f55a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55a80678c0 0x7f55a8069d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f55c414c570 tx=0x7f55b4008040 comp rx=0 tx=0).stop 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 msgr2=0x7f55c406e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 0x7f55c406e300 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f55c00011f0 tx=0x7f55c000c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 shutdown_connections 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55a80678c0 0x7f55a8069d80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55c40cd050 0x7f55c4153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55c40b4450 0x7f55c4148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 --2- 172.21.15.70:0/4189980392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55c4072280 0x7f55c406e300 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 >> 172.21.15.70:0/4189980392 conn(0x7f55c40b1d40 msgr2=0x7f55c40b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 shutdown_connections 2024-04-22T01:15:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.172+0000 7f55cbf61640 1 -- 172.21.15.70:0/4189980392 wait complete. 2024-04-22T01:15:51.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:51.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.308+0000 7f6ced41e640 1 Processor -- start 2024-04-22T01:15:51.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.308+0000 7f6ced41e640 1 -- start start 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ced41e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ce8158650 0x7f6ce815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ced41e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce806d8f0 0x7f6ce806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ced41e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ce806de60 con 0x7f6ce806f440 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ced41e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6ce806e140 con 0x7f6ce8158650 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ced41e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce806e420 con 0x7f6ce806d8f0 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ce8158650 0x7f6ce815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce806d8f0 0x7f6ce806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ce8158650 0x7f6ce815ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:34396/0 (socket says 172.21.15.70:34396) 2024-04-22T01:15:51.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/2832144293 learned_addr learned my addr 172.21.15.70:0/2832144293 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce806d8f0 msgr2=0x7f6ce806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 --2- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce806d8f0 0x7f6ce806ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/2832144293 >> v1:172.21.15.98:6789/0 conn(0x7f6ce806f440 legacy=0x7f6ce815f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/2832144293 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce806e700 con 0x7f6ce8158650 2024-04-22T01:15:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce6ffd640 1 --2- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce8158650 0x7f6ce815ee80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009870 tx=0x7f6cd801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c56f23ae7165f6f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce5ffb640 1 -- 172.21.15.70:0/2832144293 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80b0070 con 0x7f6ce8158650 2024-04-22T01:15:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce5ffb640 1 -- 172.21.15.70:0/2832144293 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd80ab600 con 0x7f6ce8158650 2024-04-22T01:15:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.312+0000 7f6ce5ffb640 1 -- 172.21.15.70:0/2832144293 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80ab9d0 con 0x7f6ce8158650 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce8158650 msgr2=0x7f6ce815ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 --2- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce8158650 0x7f6ce815ee80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009870 tx=0x7f6cd801c4d0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- 172.21.15.70:0/2832144293 shutdown_connections 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 --2- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce806d8f0 0x7f6ce806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 --2- 172.21.15.70:0/2832144293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce8158650 0x7f6ce815ee80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- 172.21.15.70:0/2832144293 >> 172.21.15.70:0/2832144293 conn(0x7f6ce80b1d40 msgr2=0x7f6ce80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- 172.21.15.70:0/2832144293 shutdown_connections 2024-04-22T01:15:51.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- 172.21.15.70:0/2832144293 wait complete. 2024-04-22T01:15:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 Processor -- start 2024-04-22T01:15:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- start start 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 0x7f6ce8147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806f440 0x7f6ce8146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce8158650 0x7f6ce8153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce816ac50 con 0x7f6ce806d8f0 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce815fee0 con 0x7f6ce806f440 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ced41e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce816add0 con 0x7f6ce8158650 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 0x7f6ce8147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 0x7f6ce8147670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51262/0 (socket says 172.21.15.70:51262) 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/3450101645 learned_addr learned my addr 172.21.15.70:0/3450101645 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce67fc640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806f440 0x7f6ce8146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce77fe640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce8158650 0x7f6ce8153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce8158650 msgr2=0x7f6ce8153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce8158650 0x7f6ce8153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806f440 msgr2=0x7f6ce8146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806f440 0x7f6ce8146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 -- 172.21.15.70:0/3450101645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce81468b0 con 0x7f6ce806d8f0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.316+0000 7f6ce6ffd640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 0x7f6ce8147670 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6cd801a100 tx=0x7f6cd801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80b0040 con 0x7f6ce806d8f0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd80bc070 con 0x7f6ce806d8f0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce8146b90 con 0x7f6ce806d8f0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80b6640 con 0x7f6ce806d8f0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce814ae00 con 0x7f6ce806d8f0 2024-04-22T01:15:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce814b090 con 0x7f6ce806d8f0 2024-04-22T01:15:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6cd80b67e0 con 0x7f6ce806d8f0 2024-04-22T01:15:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ccf7fe640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cd40678c0 0x7f6cd4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6cd8120e20 con 0x7f6ce806d8f0 2024-04-22T01:15:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ce67fc640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cd40678c0 0x7f6cd4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.320+0000 7f6ce67fc640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cd40678c0 0x7f6cd4069d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8154980 tx=0x7f6cdc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.324+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6ce814b090 con 0x7f6ce806d8f0 2024-04-22T01:15:51.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6ce8068210 con 0x7f6ce806d8f0 2024-04-22T01:15:51.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ccf7fe640 1 -- 172.21.15.70:0/3450101645 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f6cd8016150 con 0x7f6ce806d8f0 2024-04-22T01:15:51.487 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cd40678c0 msgr2=0x7f6cd4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cd40678c0 0x7f6cd4069d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8154980 tx=0x7f6cdc008040 comp rx=0 tx=0).stop 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 msgr2=0x7f6ce8147670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 0x7f6ce8147670 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6cd801a100 tx=0x7f6cd801ad10 comp rx=0 tx=0).stop 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 shutdown_connections 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cd40678c0 0x7f6cd4069d80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ce8158650 0x7f6ce8153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce806f440 0x7f6ce8146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 --2- 172.21.15.70:0/3450101645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce806d8f0 0x7f6ce8147670 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 >> 172.21.15.70:0/3450101645 conn(0x7f6ce80b1d40 msgr2=0x7f6ce80b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 shutdown_connections 2024-04-22T01:15:51.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.484+0000 7f6ced41e640 1 -- 172.21.15.70:0/3450101645 wait complete. 2024-04-22T01:15:51.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:51.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.628+0000 7f2fa03ed640 1 Processor -- start 2024-04-22T01:15:51.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.628+0000 7f2fa03ed640 1 -- start start 2024-04-22T01:15:51.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2fa03ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f9806b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2fa03ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f9806efb0 0x7f2f9806f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2fa03ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f980d1a70 con 0x7f2f980cfad0 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2fa03ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2f980cd570 con 0x7f2f9806efb0 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2fa03ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9806fd20 con 0x7f2f9806b1f0 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9e189640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f9806b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d988640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f9806efb0 0x7f2f9806f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9e189640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f9806b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:43186/0 (socket says 172.21.15.70:43186) 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9e189640 1 -- 172.21.15.70:0/1720650195 learned_addr learned my addr 172.21.15.70:0/1720650195 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d988640 1 -- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 msgr2=0x7f2f9806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d988640 1 --2- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f9806b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d988640 1 -- 172.21.15.70:0/1720650195 >> v1:172.21.15.98:6789/0 conn(0x7f2f980cfad0 legacy=0x7f2f980cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d988640 1 -- 172.21.15.70:0/1720650195 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9806bb10 con 0x7f2f9806efb0 2024-04-22T01:15:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d988640 1 --2- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f9806f410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009d90 tx=0x7f2f9001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb5b01f15a508564 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d187640 1 -- 172.21.15.70:0/1720650195 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f900b2070 con 0x7f2f9806efb0 2024-04-22T01:15:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d187640 1 -- 172.21.15.70:0/1720650195 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f900a5830 con 0x7f2f9806efb0 2024-04-22T01:15:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.632+0000 7f2f9d187640 1 -- 172.21.15.70:0/1720650195 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f900a5c00 con 0x7f2f9806efb0 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 msgr2=0x7f2f9806f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f9806f410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009d90 tx=0x7f2f9001c970 comp rx=0 tx=0).stop 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/1720650195 shutdown_connections 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f9806f410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/1720650195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f9806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/1720650195 >> 172.21.15.70:0/1720650195 conn(0x7f2f980733e0 msgr2=0x7f2f98099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/1720650195 shutdown_connections 2024-04-22T01:15:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/1720650195 wait complete. 2024-04-22T01:15:51.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 Processor -- start 2024-04-22T01:15:51.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- start start 2024-04-22T01:15:51.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f981600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f98156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 0x7f2f98157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9806f950 con 0x7f2f980cfad0 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f980d15e0 con 0x7f2f9806efb0 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9806fad0 con 0x7f2f9806b1f0 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 0x7f2f98157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 0x7f2f98157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51264/0 (socket says 172.21.15.70:51264) 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 -- 172.21.15.70:0/3578272990 learned_addr learned my addr 172.21.15.70:0/3578272990 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e189640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f981600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9d988640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f98156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 -- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 msgr2=0x7f2f981600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f981600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 -- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 msgr2=0x7f2f98156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f98156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 -- 172.21.15.70:0/3578272990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f980b6510 con 0x7f2f980cfad0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9d988640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f98156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f9e98a640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 0x7f2f98157320 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00c370 tx=0x7f2f8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c005220 con 0x7f2f980cfad0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c0053c0 con 0x7f2f980cfad0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f980b67f0 con 0x7f2f980cfad0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c007270 con 0x7f2f980cfad0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.636+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f981605f0 con 0x7f2f980cfad0 2024-04-22T01:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.640+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f98068920 con 0x7f2f980cfad0 2024-04-22T01:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.640+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2f8c019020 con 0x7f2f980cfad0 2024-04-22T01:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.640+0000 7f2f8affd640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f7c0678c0 0x7f2f7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.640+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2f8c085330 con 0x7f2f980cfad0 2024-04-22T01:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.640+0000 7f2f9e189640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f7c0678c0 0x7f2f7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.640+0000 7f2f9e189640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f7c0678c0 0x7f2f7c069d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009ae0 tx=0x7f2f94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.644+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2f8c055b00 con 0x7f2f980cfad0 2024-04-22T01:15:51.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.804+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2f981611e0 con 0x7f2f980cfad0 2024-04-22T01:15:51.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.804+0000 7f2f8affd640 1 -- 172.21.15.70:0/3578272990 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f2f8c055400 con 0x7f2f980cfad0 2024-04-22T01:15:51.807 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f7c0678c0 msgr2=0x7f2f7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f7c0678c0 0x7f2f7c069d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009ae0 tx=0x7f2f94008040 comp rx=0 tx=0).stop 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 msgr2=0x7f2f98157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 0x7f2f98157320 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00c370 tx=0x7f2f8c00c830 comp rx=0 tx=0).stop 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 shutdown_connections 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f7c0678c0 0x7f2f7c069d80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f980cfad0 0x7f2f98157320 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f9806efb0 0x7f2f98156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 --2- 172.21.15.70:0/3578272990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f9806b1f0 0x7f2f981600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 >> 172.21.15.70:0/3578272990 conn(0x7f2f980733e0 msgr2=0x7f2f9815a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 shutdown_connections 2024-04-22T01:15:51.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.808+0000 7f2fa03ed640 1 -- 172.21.15.70:0/3578272990 wait complete. 2024-04-22T01:15:51.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:51.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 Processor -- start 2024-04-22T01:15:51.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 -- start start 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4b2406c480 0x7f4b240ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4b24162510 0x7f4b2415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b240cd4a0 con 0x7f4b240ccb80 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4b24162080 con 0x7f4b2406c480 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2c078640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4b2406c8a0 con 0x7f4b24162510 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4b2406c480 0x7f4b240ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4b2406c480 0x7f4b240ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51268/0 (socket says 172.21.15.70:51268) 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 -- 172.21.15.70:0/3268024585 learned_addr learned my addr 172.21.15.70:0/3268024585 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 -- 172.21.15.70:0/3268024585 >> v2:172.21.15.98:3300/0 conn(0x7f4b24162510 msgr2=0x7f4b2415e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b2a615640 1 --2- 172.21.15.70:0/3268024585 >> v2:172.21.15.98:3300/0 conn(0x7f4b24162510 0x7f4b2415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 --2- 172.21.15.70:0/3268024585 >> v2:172.21.15.98:3300/0 conn(0x7f4b24162510 0x7f4b2415e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 -- 172.21.15.70:0/3268024585 >> v1:172.21.15.70:6789/0 conn(0x7f4b240ccb80 legacy=0x7f4b240b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b28e12640 1 -- 172.21.15.70:0/3268024585 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3358906131 0 0) 0x7f4b240cd4a0 con 0x7f4b240ccb80 2024-04-22T01:15:51.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.940+0000 7f4b29613640 1 -- 172.21.15.70:0/3268024585 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2406b320 con 0x7f4b2406c480 2024-04-22T01:15:51.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b29613640 1 --2- 172.21.15.70:0/3268024585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b2406c480 0x7f4b240ccf60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4b24161ad0 tx=0x7f4b1801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db9244a751855d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b28e12640 1 -- 172.21.15.70:0/3268024585 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b180b0070 con 0x7f4b2406c480 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b28e12640 1 -- 172.21.15.70:0/3268024585 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b180ab600 con 0x7f4b2406c480 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b28e12640 1 -- 172.21.15.70:0/3268024585 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b180ab9d0 con 0x7f4b2406c480 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- 172.21.15.70:0/3268024585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b2406c480 msgr2=0x7f4b240ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 --2- 172.21.15.70:0/3268024585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b2406c480 0x7f4b240ccf60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4b24161ad0 tx=0x7f4b1801c4f0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- 172.21.15.70:0/3268024585 shutdown_connections 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 --2- 172.21.15.70:0/3268024585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b2406c480 0x7f4b240ccf60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 --2- 172.21.15.70:0/3268024585 >> v2:172.21.15.98:3300/0 conn(0x7f4b24162510 0x7f4b2415e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- 172.21.15.70:0/3268024585 >> 172.21.15.70:0/3268024585 conn(0x7f4b240b3c00 msgr2=0x7f4b240b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- 172.21.15.70:0/3268024585 shutdown_connections 2024-04-22T01:15:51.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- 172.21.15.70:0/3268024585 wait complete. 2024-04-22T01:15:51.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 Processor -- start 2024-04-22T01:15:51.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- start start 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b2406c480 0x7f4b24156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b240ccb80 0x7f4b2415d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 0x7f4b24160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2416b050 con 0x7f4b24162510 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2416aed0 con 0x7f4b240ccb80 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2c078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4b2416b1d0 con 0x7f4b2406c480 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2a615640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 0x7f4b24160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2a615640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 0x7f4b24160800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51276/0 (socket says 172.21.15.70:51276) 2024-04-22T01:15:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.944+0000 7f4b2a615640 1 -- 172.21.15.70:0/2579079547 learned_addr learned my addr 172.21.15.70:0/2579079547 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b29613640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b240ccb80 0x7f4b2415d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2a615640 1 -- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b2406c480 msgr2=0x7f4b24156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b29e14640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b2406c480 0x7f4b24156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2a615640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b2406c480 0x7f4b24156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2a615640 1 -- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b240ccb80 msgr2=0x7f4b2415d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2a615640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b240ccb80 0x7f4b2415d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2a615640 1 -- 172.21.15.70:0/2579079547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b24155670 con 0x7f4b24162510 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b29613640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b240ccb80 0x7f4b2415d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2a615640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 0x7f4b24160800 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400cc60 tx=0x7f4b1400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b140039a0 con 0x7f4b24162510 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b14003b40 con 0x7f4b24162510 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b24155950 con 0x7f4b24162510 2024-04-22T01:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b14004020 con 0x7f4b24162510 2024-04-22T01:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b24155f10 con 0x7f4b24162510 2024-04-22T01:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b2404f510 con 0x7f4b24162510 2024-04-22T01:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b140041c0 con 0x7f4b24162510 2024-04-22T01:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b227fc640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b100678c0 0x7f4b10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4b14086500 con 0x7f4b24162510 2024-04-22T01:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b29e14640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b100678c0 0x7f4b10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.948+0000 7f4b29e14640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b100678c0 0x7f4b10069d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c007fd0 tx=0x7f4b1c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:51.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:51.952+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4b14014030 con 0x7f4b24162510 2024-04-22T01:15:52.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.124+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4b240cd830 con 0x7f4b24162510 2024-04-22T01:15:52.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.124+0000 7f4b227fc640 1 -- 172.21.15.70:0/2579079547 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f4b14056c40 con 0x7f4b24162510 2024-04-22T01:15:52.129 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:52.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b100678c0 msgr2=0x7f4b10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b100678c0 0x7f4b10069d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c007fd0 tx=0x7f4b1c009040 comp rx=0 tx=0).stop 2024-04-22T01:15:52.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 msgr2=0x7f4b24160800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 0x7f4b24160800 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400cc60 tx=0x7f4b1400a5c0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 shutdown_connections 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b100678c0 0x7f4b10069d80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b24162510 0x7f4b24160800 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b240ccb80 0x7f4b2415d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 --2- 172.21.15.70:0/2579079547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b2406c480 0x7f4b24156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 >> 172.21.15.70:0/2579079547 conn(0x7f4b240b3c00 msgr2=0x7f4b240ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 shutdown_connections 2024-04-22T01:15:52.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.128+0000 7f4b2c078640 1 -- 172.21.15.70:0/2579079547 wait complete. 2024-04-22T01:15:52.147 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:15:52.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:52.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-22T01:15:52.159 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:15:52.159 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:15:52.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:52.160 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:15:52.206 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:15:52.206 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:52.213 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:15:52.213 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:52.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:52.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.376+0000 7fe66a284640 1 Processor -- start 2024-04-22T01:15:52.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.376+0000 7fe66a284640 1 -- start start 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe6640cc150 0x7fe664071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6640cd7b0 con 0x7fe664072060 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6640cc710 con 0x7fe6641455b0 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe664072490 con 0x7fe6640cc150 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe668821640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe664072060 0x7fe66406e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38840/0 (socket says 172.21.15.70:38840) 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe6637fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe6640cc150 0x7fe664071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe668821640 1 -- 172.21.15.70:0/1215776791 learned_addr learned my addr 172.21.15.70:0/1215776791 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:52.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe662ffd640 1 -- 172.21.15.70:0/1215776791 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692017932 0 0) 0x7fe6640cd7b0 con 0x7fe664072060 2024-04-22T01:15:52.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe662ffd640 1 -- 172.21.15.70:0/1215776791 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe650003530 con 0x7fe664072060 2024-04-22T01:15:52.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe6637fe640 1 -- 172.21.15.70:0/1215776791 >> v1:172.21.15.98:6789/0 conn(0x7fe6641455b0 legacy=0x7fe6640cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:52.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe6637fe640 1 -- 172.21.15.70:0/1215776791 >> v1:172.21.15.70:6789/0 conn(0x7fe664072060 legacy=0x7fe66406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe6637fe640 1 -- 172.21.15.70:0/1215776791 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6640cc9f0 con 0x7fe6640cc150 2024-04-22T01:15:52.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe6637fe640 1 --2- 172.21.15.70:0/1215776791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6640cc150 0x7fe664071b20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0097c0 tx=0x7fe64c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd1aaa994f113cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe662ffd640 1 -- 172.21.15.70:0/1215776791 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0b0070 con 0x7fe6640cc150 2024-04-22T01:15:52.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe662ffd640 1 -- 172.21.15.70:0/1215776791 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c0ab600 con 0x7fe6640cc150 2024-04-22T01:15:52.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe662ffd640 1 -- 172.21.15.70:0/1215776791 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0ab9d0 con 0x7fe6640cc150 2024-04-22T01:15:52.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- 172.21.15.70:0/1215776791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6640cc150 msgr2=0x7fe664071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 --2- 172.21.15.70:0/1215776791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6640cc150 0x7fe664071b20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0097c0 tx=0x7fe64c01c340 comp rx=0 tx=0).stop 2024-04-22T01:15:52.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- 172.21.15.70:0/1215776791 shutdown_connections 2024-04-22T01:15:52.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 --2- 172.21.15.70:0/1215776791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6640cc150 0x7fe664071b20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- 172.21.15.70:0/1215776791 >> 172.21.15.70:0/1215776791 conn(0x7fe66406b1f0 msgr2=0x7fe66406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:52.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.380+0000 7fe66a284640 1 -- 172.21.15.70:0/1215776791 shutdown_connections 2024-04-22T01:15:52.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- 172.21.15.70:0/1215776791 wait complete. 2024-04-22T01:15:52.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 Processor -- start 2024-04-22T01:15:52.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- start start 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664072060 0x7fe664141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6640cc150 0x7fe66413f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 0x7fe6640afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe664169ee0 con 0x7fe6641455b0 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe664169d60 con 0x7fe664072060 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe66416a060 con 0x7fe6640cc150 2024-04-22T01:15:52.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 0x7fe6640afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 0x7fe6640afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51290/0 (socket says 172.21.15.70:51290) 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 -- 172.21.15.70:0/557043687 learned_addr learned my addr 172.21.15.70:0/557043687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe6637fe640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6640cc150 0x7fe66413f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe663fff640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664072060 0x7fe664141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 -- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6640cc150 msgr2=0x7fe66413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6640cc150 0x7fe66413f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 -- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664072060 msgr2=0x7fe664141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664072060 0x7fe664141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 -- 172.21.15.70:0/557043687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66415d840 con 0x7fe6641455b0 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe663fff640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664072060 0x7fe664141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe6637fe640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6640cc150 0x7fe66413f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe668821640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 0x7fe6640afe70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00c3c0 tx=0x7fe65c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c017070 con 0x7fe6641455b0 2024-04-22T01:15:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe65c005cf0 con 0x7fe6641455b0 2024-04-22T01:15:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6640b2f40 con 0x7fe6641455b0 2024-04-22T01:15:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c007310 con 0x7fe6641455b0 2024-04-22T01:15:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6640b3500 con 0x7fe6641455b0 2024-04-22T01:15:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.384+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe66413fc20 con 0x7fe6641455b0 2024-04-22T01:15:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.388+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe65c01a070 con 0x7fe6641455b0 2024-04-22T01:15:52.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.388+0000 7fe660ff9640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6440678c0 0x7fe644069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.388+0000 7fe663fff640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6440678c0 0x7fe644069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.388+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe65c0853b0 con 0x7fe6641455b0 2024-04-22T01:15:52.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.388+0000 7fe663fff640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6440678c0 0x7fe644069d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe654007e90 tx=0x7fe6540073a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.392+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe66413fc20 con 0x7fe6641455b0 2024-04-22T01:15:52.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.552+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe664068210 con 0x7fe6641455b0 2024-04-22T01:15:52.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe660ff9640 1 -- 172.21.15.70:0/557043687 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7fe65c055b80 con 0x7fe6641455b0 2024-04-22T01:15:52.557 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:52.558 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6440678c0 msgr2=0x7fe644069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6440678c0 0x7fe644069d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe654007e90 tx=0x7fe6540073a0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 msgr2=0x7fe6640afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 0x7fe6640afe70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00c3c0 tx=0x7fe65c00c880 comp rx=0 tx=0).stop 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 shutdown_connections 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6440678c0 0x7fe644069d80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6641455b0 0x7fe6640afe70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6640cc150 0x7fe66413f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 --2- 172.21.15.70:0/557043687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe664072060 0x7fe664141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 >> 172.21.15.70:0/557043687 conn(0x7fe66406b1f0 msgr2=0x7fe66406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 shutdown_connections 2024-04-22T01:15:52.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.556+0000 7fe66a284640 1 -- 172.21.15.70:0/557043687 wait complete. 2024-04-22T01:15:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:52.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:52.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.688+0000 7ff6a137c640 1 Processor -- start 2024-04-22T01:15:52.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.688+0000 7ff6a137c640 1 -- start start 2024-04-22T01:15:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff69c0ce920 0x7ff69c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff69c0cedd0 0x7ff69c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff69c0b3800 con 0x7ff69c15e6e0 2024-04-22T01:15:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff69c0b3ae0 con 0x7ff69c0cedd0 2024-04-22T01:15:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff69c0cde40 con 0x7ff69c0ce920 2024-04-22T01:15:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff69c0cedd0 0x7ff69c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff69c0cedd0 0x7ff69c1551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51304/0 (socket says 172.21.15.70:51304) 2024-04-22T01:15:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 -- 172.21.15.70:0/1424318818 learned_addr learned my addr 172.21.15.70:0/1424318818 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69a7fc640 1 --2- 172.21.15.70:0/1424318818 >> v2:172.21.15.98:3300/0 conn(0x7ff69c0ce920 0x7ff69c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 -- 172.21.15.70:0/1424318818 >> v2:172.21.15.98:3300/0 conn(0x7ff69c0ce920 msgr2=0x7ff69c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 --2- 172.21.15.70:0/1424318818 >> v2:172.21.15.98:3300/0 conn(0x7ff69c0ce920 0x7ff69c154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 -- 172.21.15.70:0/1424318818 >> v1:172.21.15.98:6789/0 conn(0x7ff69c15e6e0 legacy=0x7ff69c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 -- 172.21.15.70:0/1424318818 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69c0ce120 con 0x7ff69c0cedd0 2024-04-22T01:15:52.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff69b7fe640 1 --2- 172.21.15.70:0/1424318818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c0cedd0 0x7ff69c1551a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff69c0b26b0 tx=0x7ff69401eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73a87d60a7122b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff699ffb640 1 -- 172.21.15.70:0/1424318818 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69400b2d0 con 0x7ff69c0cedd0 2024-04-22T01:15:52.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff699ffb640 1 -- 172.21.15.70:0/1424318818 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff69400b470 con 0x7ff69c0cedd0 2024-04-22T01:15:52.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff699ffb640 1 -- 172.21.15.70:0/1424318818 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6940a7c90 con 0x7ff69c0cedd0 2024-04-22T01:15:52.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- 172.21.15.70:0/1424318818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c0cedd0 msgr2=0x7ff69c1551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 --2- 172.21.15.70:0/1424318818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c0cedd0 0x7ff69c1551a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff69c0b26b0 tx=0x7ff69401eb80 comp rx=0 tx=0).stop 2024-04-22T01:15:52.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- 172.21.15.70:0/1424318818 shutdown_connections 2024-04-22T01:15:52.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 --2- 172.21.15.70:0/1424318818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c0cedd0 0x7ff69c1551a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 --2- 172.21.15.70:0/1424318818 >> v2:172.21.15.98:3300/0 conn(0x7ff69c0ce920 0x7ff69c154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- 172.21.15.70:0/1424318818 >> 172.21.15.70:0/1424318818 conn(0x7ff69c0b1d40 msgr2=0x7ff69c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- 172.21.15.70:0/1424318818 shutdown_connections 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.692+0000 7ff6a137c640 1 -- 172.21.15.70:0/1424318818 wait complete. 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 Processor -- start 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 -- start start 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff69c0ce920 0x7ff69c15a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff69c0cedd0 0x7ff69c138590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 0x7ff69c138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c0b2e40 con 0x7ff69c15e6e0 2024-04-22T01:15:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c0b2cc0 con 0x7ff69c0cedd0 2024-04-22T01:15:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff69c0b2fc0 con 0x7ff69c0ce920 2024-04-22T01:15:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 0x7ff69c138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 0x7ff69c138ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51320/0 (socket says 172.21.15.70:51320) 2024-04-22T01:15:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 -- 172.21.15.70:0/202081739 learned_addr learned my addr 172.21.15.70:0/202081739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:52.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69affd640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff69c0ce920 0x7ff69c15a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69a7fc640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff69c0cedd0 0x7ff69c138590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 -- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff69c0ce920 msgr2=0x7ff69c15a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff69c0ce920 0x7ff69c15a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 -- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff69c0cedd0 msgr2=0x7ff69c138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff69c0cedd0 0x7ff69c138590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 -- 172.21.15.70:0/202081739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69c166040 con 0x7ff69c15e6e0 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff69b7fe640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 0x7ff69c138ad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff69401c7b0 tx=0x7ff69401b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6940be020 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff69400b2c0 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694009c50 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69c067a60 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.696+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69c1393c0 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.700+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff69c068210 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.700+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6940c1020 con 0x7ff69c15e6e0 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.700+0000 7ff6837fe640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6880678f0 0x7ff688069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.700+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff694122ed0 con 0x7ff69c15e6e0 2024-04-22T01:15:52.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.700+0000 7ff69affd640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6880678f0 0x7ff688069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.700+0000 7ff69affd640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6880678f0 0x7ff688069db0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00a790 tx=0x7ff68c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.704+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff6940c9020 con 0x7ff69c15e6e0 2024-04-22T01:15:52.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff69c0d08e0 con 0x7ff69c15e6e0 2024-04-22T01:15:52.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6837fe640 1 -- 172.21.15.70:0/202081739 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7ff6940187a0 con 0x7ff69c15e6e0 2024-04-22T01:15:52.856 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:52.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6880678f0 msgr2=0x7ff688069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6880678f0 0x7ff688069db0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00a790 tx=0x7ff68c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:52.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 msgr2=0x7ff69c138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 0x7ff69c138ad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff69401c7b0 tx=0x7ff69401b920 comp rx=0 tx=0).stop 2024-04-22T01:15:52.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 shutdown_connections 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6880678f0 0x7ff688069db0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff69c15e6e0 0x7ff69c138ad0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff69c0cedd0 0x7ff69c138590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 --2- 172.21.15.70:0/202081739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff69c0ce920 0x7ff69c15a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 >> 172.21.15.70:0/202081739 conn(0x7ff69c0b1d40 msgr2=0x7ff69c15d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 shutdown_connections 2024-04-22T01:15:52.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.852+0000 7ff6a137c640 1 -- 172.21.15.70:0/202081739 wait complete. 2024-04-22T01:15:52.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:52.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.976+0000 7fc93a200640 1 Processor -- start 2024-04-22T01:15:52.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.976+0000 7fc93a200640 1 -- start start 2024-04-22T01:15:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ce920 0x7fc934154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc9340ceda0 0x7fc9341551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9340b3800 con 0x7fc93415e6e0 2024-04-22T01:15:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc9340b3ae0 con 0x7fc9340ceda0 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9340cde40 con 0x7fc9340ce920 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc9340ceda0 0x7fc9341551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc9340ceda0 0x7fc9341551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51330/0 (socket says 172.21.15.70:51330) 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 -- 172.21.15.70:0/4025515220 learned_addr learned my addr 172.21.15.70:0/4025515220 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc9337fe640 1 --1- 172.21.15.70:0/4025515220 >> v1:172.21.15.70:6789/0 conn(0x7fc93415e6e0 0x7fc93415eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38846/0 (socket says 172.21.15.70:38846) 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc932ffd640 1 --2- 172.21.15.70:0/4025515220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ce920 0x7fc934154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 -- 172.21.15.70:0/4025515220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ce920 msgr2=0x7fc934154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 --2- 172.21.15.70:0/4025515220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ce920 0x7fc934154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc9327fc640 1 -- 172.21.15.70:0/4025515220 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432335087 0 0) 0x7fc9340b3800 con 0x7fc93415e6e0 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 -- 172.21.15.70:0/4025515220 >> v1:172.21.15.70:6789/0 conn(0x7fc93415e6e0 legacy=0x7fc93415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 -- 172.21.15.70:0/4025515220 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9340ce120 con 0x7fc9340ceda0 2024-04-22T01:15:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc933fff640 1 --2- 172.21.15.70:0/4025515220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9340ceda0 0x7fc9341551a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009e50 tx=0x7fc92c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec03421363c4d85d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc9327fc640 1 -- 172.21.15.70:0/4025515220 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0a58f0 con 0x7fc9340ceda0 2024-04-22T01:15:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc9327fc640 1 -- 172.21.15.70:0/4025515220 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92c0a5a90 con 0x7fc9340ceda0 2024-04-22T01:15:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc9327fc640 1 -- 172.21.15.70:0/4025515220 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0a5e60 con 0x7fc9340ceda0 2024-04-22T01:15:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- 172.21.15.70:0/4025515220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9340ceda0 msgr2=0x7fc9341551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 --2- 172.21.15.70:0/4025515220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9340ceda0 0x7fc9341551a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009e50 tx=0x7fc92c01ca00 comp rx=0 tx=0).stop 2024-04-22T01:15:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- 172.21.15.70:0/4025515220 shutdown_connections 2024-04-22T01:15:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 --2- 172.21.15.70:0/4025515220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9340ceda0 0x7fc9341551a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 --2- 172.21.15.70:0/4025515220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ce920 0x7fc934154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- 172.21.15.70:0/4025515220 >> 172.21.15.70:0/4025515220 conn(0x7fc9340b1d40 msgr2=0x7fc9340cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- 172.21.15.70:0/4025515220 shutdown_connections 2024-04-22T01:15:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.980+0000 7fc93a200640 1 -- 172.21.15.70:0/4025515220 wait complete. 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 Processor -- start 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- start start 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ce920 0x7fc934152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ceda0 0x7fc934147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 0x7fc934153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9340b2e40 con 0x7fc93415e6e0 2024-04-22T01:15:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9340b2cc0 con 0x7fc9340ce920 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9340b2fc0 con 0x7fc9340ceda0 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 0x7fc934153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 0x7fc934153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51332/0 (socket says 172.21.15.70:51332) 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 -- 172.21.15.70:0/1641251937 learned_addr learned my addr 172.21.15.70:0/1641251937 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 -- 172.21.15.70:0/1641251937 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ceda0 msgr2=0x7fc934147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ceda0 0x7fc934147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 -- 172.21.15.70:0/1641251937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ce920 msgr2=0x7fc934152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ce920 0x7fc934152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 -- 172.21.15.70:0/1641251937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc934147da0 con 0x7fc93415e6e0 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc933fff640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 0x7fc934153690 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009e50 tx=0x7fc92c0197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0bd020 con 0x7fc93415e6e0 2024-04-22T01:15:52.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92c0b8470 con 0x7fc93415e6e0 2024-04-22T01:15:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c009510 con 0x7fc93415e6e0 2024-04-22T01:15:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc934157010 con 0x7fc93415e6e0 2024-04-22T01:15:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9341575d0 con 0x7fc93415e6e0 2024-04-22T01:15:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc934148d10 con 0x7fc93415e6e0 2024-04-22T01:15:52.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.984+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc92c0c0070 con 0x7fc93415e6e0 2024-04-22T01:15:52.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.988+0000 7fc91bfff640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9200678f0 0x7fc920069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:52.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.988+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc92c120b40 con 0x7fc93415e6e0 2024-04-22T01:15:52.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.988+0000 7fc9337fe640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9200678f0 0x7fc920069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:52.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.988+0000 7fc9337fe640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9200678f0 0x7fc920069db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc934067c60 tx=0x7fc928007970 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:52.992+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc92c016620 con 0x7fc93415e6e0 2024-04-22T01:15:53.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.148+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc934068210 con 0x7fc93415e6e0 2024-04-22T01:15:53.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc91bfff640 1 -- 172.21.15.70:0/1641251937 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fc92c0a59b0 con 0x7fc93415e6e0 2024-04-22T01:15:53.153 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:53.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9200678f0 msgr2=0x7fc920069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9200678f0 0x7fc920069db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc934067c60 tx=0x7fc928007970 comp rx=0 tx=0).stop 2024-04-22T01:15:53.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 msgr2=0x7fc934153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 0x7fc934153690 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009e50 tx=0x7fc92c0197a0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 shutdown_connections 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9200678f0 0x7fc920069db0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc93415e6e0 0x7fc934153690 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9340ceda0 0x7fc934147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 --2- 172.21.15.70:0/1641251937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ce920 0x7fc934152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 >> 172.21.15.70:0/1641251937 conn(0x7fc9340b1d40 msgr2=0x7fc9341599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 shutdown_connections 2024-04-22T01:15:53.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.152+0000 7fc93a200640 1 -- 172.21.15.70:0/1641251937 wait complete. 2024-04-22T01:15:53.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.284+0000 7f8b94ec0640 1 Processor -- start 2024-04-22T01:15:53.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.284+0000 7f8b94ec0640 1 -- start start 2024-04-22T01:15:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b94ec0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b9015e0a0 0x7f8b900cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b94ec0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b900d0490 0x7f8b900d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b94ec0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b900ccdc0 con 0x7f8b90099da0 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b94ec0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8b900ce3a0 con 0x7f8b900d0490 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b94ec0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8b900d0e30 con 0x7f8b9015e0a0 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b900d0490 0x7f8b900d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b900d0490 0x7f8b900d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51342/0 (socket says 172.21.15.70:51342) 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 -- 172.21.15.70:0/2771124994 learned_addr learned my addr 172.21.15.70:0/2771124994 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8ed9d640 1 --1- 172.21.15.70:0/2771124994 >> v1:172.21.15.70:6789/0 conn(0x7f8b90099da0 0x7f8b9006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38850/0 (socket says 172.21.15.70:38850) 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8e59c640 1 --2- 172.21.15.70:0/2771124994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b9015e0a0 0x7f8b900cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 -- 172.21.15.70:0/2771124994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b9015e0a0 msgr2=0x7f8b900cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 --2- 172.21.15.70:0/2771124994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b9015e0a0 0x7f8b900cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 -- 172.21.15.70:0/2771124994 >> v1:172.21.15.70:6789/0 conn(0x7f8b90099da0 legacy=0x7f8b9006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 -- 172.21.15.70:0/2771124994 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b900cd0a0 con 0x7f8b900d0490 2024-04-22T01:15:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8dd9b640 1 --2- 172.21.15.70:0/2771124994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900d0490 0x7f8b900d08f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009870 tx=0x7f8b8401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787d64fdfeeeef42 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8d59a640 1 -- 172.21.15.70:0/2771124994 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b840b1070 con 0x7f8b900d0490 2024-04-22T01:15:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8d59a640 1 -- 172.21.15.70:0/2771124994 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b840ac620 con 0x7f8b900d0490 2024-04-22T01:15:53.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.288+0000 7f8b8d59a640 1 -- 172.21.15.70:0/2771124994 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b840ac9f0 con 0x7f8b900d0490 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/2771124994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900d0490 msgr2=0x7f8b900d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/2771124994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900d0490 0x7f8b900d08f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009870 tx=0x7f8b8401c4d0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/2771124994 shutdown_connections 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/2771124994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900d0490 0x7f8b900d08f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/2771124994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b9015e0a0 0x7f8b900cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/2771124994 >> 172.21.15.70:0/2771124994 conn(0x7f8b900b3770 msgr2=0x7f8b900b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/2771124994 shutdown_connections 2024-04-22T01:15:53.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/2771124994 wait complete. 2024-04-22T01:15:53.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 Processor -- start 2024-04-22T01:15:53.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- start start 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b90099da0 0x7f8b900c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b900d0490 0x7f8b9015c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 0x7f8b900b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9009a5d0 con 0x7f8b9015e0a0 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9006ce20 con 0x7f8b900d0490 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9009a750 con 0x7f8b90099da0 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 0x7f8b900b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 0x7f8b900b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51350/0 (socket says 172.21.15.70:51350) 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 -- 172.21.15.70:0/3484122513 learned_addr learned my addr 172.21.15.70:0/3484122513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8e59c640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b90099da0 0x7f8b900c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8dd9b640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b900d0490 0x7f8b9015c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 -- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b90099da0 msgr2=0x7f8b900c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b90099da0 0x7f8b900c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 -- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b900d0490 msgr2=0x7f8b9015c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b900d0490 0x7f8b9015c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 -- 172.21.15.70:0/3484122513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9015b060 con 0x7f8b9015e0a0 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8dd9b640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b900d0490 0x7f8b9015c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b8ed9d640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 0x7f8b900b48a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800c730 tx=0x7f8b8800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88017070 con 0x7f8b9015e0a0 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8800ce40 con 0x7f8b9015e0a0 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9015b340 con 0x7f8b9015e0a0 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8800a840 con 0x7f8b9015e0a0 2024-04-22T01:15:53.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9015b900 con 0x7f8b9015e0a0 2024-04-22T01:15:53.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.292+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b90068210 con 0x7f8b9015e0a0 2024-04-22T01:15:53.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.296+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b8801a070 con 0x7f8b9015e0a0 2024-04-22T01:15:53.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.296+0000 7f8b7effd640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b700678c0 0x7f8b70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.296+0000 7f8b8e59c640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b700678c0 0x7f8b70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.296+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8b88085650 con 0x7f8b9015e0a0 2024-04-22T01:15:53.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.296+0000 7f8b8e59c640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b700678c0 0x7f8b70069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8b9006e400 tx=0x7f8b80008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.300+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8b880556a0 con 0x7f8b9015e0a0 2024-04-22T01:15:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.460+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8b900b0680 con 0x7f8b9015e0a0 2024-04-22T01:15:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.460+0000 7f8b7effd640 1 -- 172.21.15.70:0/3484122513 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f8b880599a0 con 0x7f8b9015e0a0 2024-04-22T01:15:53.465 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:53.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b700678c0 msgr2=0x7f8b70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b700678c0 0x7f8b70069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8b9006e400 tx=0x7f8b80008040 comp rx=0 tx=0).stop 2024-04-22T01:15:53.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 msgr2=0x7f8b900b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 0x7f8b900b48a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800c730 tx=0x7f8b8800cbf0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 shutdown_connections 2024-04-22T01:15:53.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8b700678c0 0x7f8b70069d80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9015e0a0 0x7f8b900b48a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b900d0490 0x7f8b9015c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 --2- 172.21.15.70:0/3484122513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b90099da0 0x7f8b900c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 >> 172.21.15.70:0/3484122513 conn(0x7f8b900b3770 msgr2=0x7f8b9006fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 shutdown_connections 2024-04-22T01:15:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.464+0000 7f8b94ec0640 1 -- 172.21.15.70:0/3484122513 wait complete. 2024-04-22T01:15:53.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:53.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 Processor -- start 2024-04-22T01:15:53.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 -- start start 2024-04-22T01:15:53.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa62006f3e0 0x7fa620071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa6200720b0 0x7fa6200ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6200724c0 con 0x7fa620162350 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa62006e340 con 0x7fa62006f3e0 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.600+0000 7fa627b90640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa62006e620 con 0x7fa6200720b0 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62512b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa62006f3e0 0x7fa620071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62592c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa620162350 0x7fa62006d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38858/0 (socket says 172.21.15.70:38858) 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62592c640 1 -- 172.21.15.70:0/3524222289 learned_addr learned my addr 172.21.15.70:0/3524222289 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62612d640 1 --2- 172.21.15.70:0/3524222289 >> v2:172.21.15.98:3300/0 conn(0x7fa6200720b0 0x7fa6200ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62512b640 1 -- 172.21.15.70:0/3524222289 >> v2:172.21.15.98:3300/0 conn(0x7fa6200720b0 msgr2=0x7fa6200ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62512b640 1 --2- 172.21.15.70:0/3524222289 >> v2:172.21.15.98:3300/0 conn(0x7fa6200720b0 0x7fa6200ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62512b640 1 -- 172.21.15.70:0/3524222289 >> v1:172.21.15.70:6789/0 conn(0x7fa620162350 legacy=0x7fa62006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62492a640 1 -- 172.21.15.70:0/3524222289 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645285313 0 0) 0x7fa6100014a0 con 0x7fa620162350 2024-04-22T01:15:53.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62512b640 1 -- 172.21.15.70:0/3524222289 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62006e900 con 0x7fa62006f3e0 2024-04-22T01:15:53.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62512b640 1 --2- 172.21.15.70:0/3524222289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620071b70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa6200d0ab0 tx=0x7fa61401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78def3c187303ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62492a640 1 -- 172.21.15.70:0/3524222289 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6140ab650 con 0x7fa62006f3e0 2024-04-22T01:15:53.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62492a640 1 -- 172.21.15.70:0/3524222289 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6140b0070 con 0x7fa62006f3e0 2024-04-22T01:15:53.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa62492a640 1 -- 172.21.15.70:0/3524222289 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6140aba10 con 0x7fa62006f3e0 2024-04-22T01:15:53.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 -- 172.21.15.70:0/3524222289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 msgr2=0x7fa620071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 --2- 172.21.15.70:0/3524222289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620071b70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa6200d0ab0 tx=0x7fa61401c450 comp rx=0 tx=0).stop 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 -- 172.21.15.70:0/3524222289 shutdown_connections 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 --2- 172.21.15.70:0/3524222289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620071b70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 --2- 172.21.15.70:0/3524222289 >> v2:172.21.15.98:3300/0 conn(0x7fa6200720b0 0x7fa6200ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 -- 172.21.15.70:0/3524222289 >> 172.21.15.70:0/3524222289 conn(0x7fa6200b1d40 msgr2=0x7fa6200cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 -- 172.21.15.70:0/3524222289 shutdown_connections 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.604+0000 7fa627b90640 1 -- 172.21.15.70:0/3524222289 wait complete. 2024-04-22T01:15:53.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 Processor -- start 2024-04-22T01:15:53.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- start start 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6200720b0 0x7fa6200afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa620162350 0x7fa6200b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6200b3870 con 0x7fa62006f3e0 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6200b36f0 con 0x7fa6200720b0 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa62016f130 con 0x7fa620162350 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51372/0 (socket says 172.21.15.70:51372) 2024-04-22T01:15:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 -- 172.21.15.70:0/3935656074 learned_addr learned my addr 172.21.15.70:0/3935656074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62512b640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6200720b0 0x7fa6200afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62612d640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa620162350 0x7fa6200b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 -- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa620162350 msgr2=0x7fa6200b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa620162350 0x7fa6200b0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 -- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6200720b0 msgr2=0x7fa6200afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6200720b0 0x7fa6200afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 -- 172.21.15.70:0/3935656074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6200b07b0 con 0x7fa62006f3e0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62512b640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6200720b0 0x7fa6200afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa62592c640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620156ad0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa61000c7c0 tx=0x7fa61000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610018070 con 0x7fa62006f3e0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61000a640 con 0x7fa62006f3e0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa620160e90 con 0x7fa62006f3e0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61000aa30 con 0x7fa62006f3e0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa620161450 con 0x7fa62006f3e0 2024-04-22T01:15:53.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa620068210 con 0x7fa62006f3e0 2024-04-22T01:15:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa61001b070 con 0x7fa62006f3e0 2024-04-22T01:15:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.608+0000 7fa61e7fc640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa60c0678c0 0x7fa60c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.612+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa610085680 con 0x7fa62006f3e0 2024-04-22T01:15:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.612+0000 7fa62512b640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa60c0678c0 0x7fa60c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.612+0000 7fa62512b640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa60c0678c0 0x7fa60c069d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6140af000 tx=0x7fa614008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.616+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa6200724c0 con 0x7fa62006f3e0 2024-04-22T01:15:53.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.764+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa6200d0300 con 0x7fa62006f3e0 2024-04-22T01:15:53.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.764+0000 7fa61e7fc640 1 -- 172.21.15.70:0/3935656074 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fa610055e50 con 0x7fa62006f3e0 2024-04-22T01:15:53.768 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa60c0678c0 msgr2=0x7fa60c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa60c0678c0 0x7fa60c069d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6140af000 tx=0x7fa614008040 comp rx=0 tx=0).stop 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 msgr2=0x7fa620156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620156ad0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa61000c7c0 tx=0x7fa61000cc80 comp rx=0 tx=0).stop 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 shutdown_connections 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa60c0678c0 0x7fa60c069d80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa620162350 0x7fa6200b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6200720b0 0x7fa6200afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 --2- 172.21.15.70:0/3935656074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa62006f3e0 0x7fa620156ad0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 >> 172.21.15.70:0/3935656074 conn(0x7fa6200b1d40 msgr2=0x7fa62014e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 shutdown_connections 2024-04-22T01:15:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.768+0000 7fa627b90640 1 -- 172.21.15.70:0/3935656074 wait complete. 2024-04-22T01:15:53.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:53.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 Processor -- start 2024-04-22T01:15:53.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 -- start start 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdab815e1f0 0x7fdab809ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdab8099cc0 0x7fdab809a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab80cc370 con 0x7fdab806c510 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdab80d0c00 con 0x7fdab8099cc0 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdac0053640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdab80d0ee0 con 0x7fdab815e1f0 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdab8099cc0 0x7fdab809a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdab8099cc0 0x7fdab809a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51380/0 (socket says 172.21.15.70:51380) 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 -- 172.21.15.70:0/1840906280 learned_addr learned my addr 172.21.15.70:0/1840906280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabddef640 1 --2- 172.21.15.70:0/1840906280 >> v2:172.21.15.98:3300/0 conn(0x7fdab815e1f0 0x7fdab809ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 -- 172.21.15.70:0/1840906280 >> v2:172.21.15.98:3300/0 conn(0x7fdab815e1f0 msgr2=0x7fdab809ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 --2- 172.21.15.70:0/1840906280 >> v2:172.21.15.98:3300/0 conn(0x7fdab815e1f0 0x7fdab809ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 -- 172.21.15.70:0/1840906280 >> v1:172.21.15.98:6789/0 conn(0x7fdab806c510 legacy=0x7fdab806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.904+0000 7fdabd5ee640 1 -- 172.21.15.70:0/1840906280 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80cc630 con 0x7fdab8099cc0 2024-04-22T01:15:53.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdabd5ee640 1 --2- 172.21.15.70:0/1840906280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab8099cc0 0x7fdab809a150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdab80b4300 tx=0x7fdab401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac584ba4803dc906 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdabcded640 1 -- 172.21.15.70:0/1840906280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40aa610 con 0x7fdab8099cc0 2024-04-22T01:15:53.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdabcded640 1 -- 172.21.15.70:0/1840906280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab40af070 con 0x7fdab8099cc0 2024-04-22T01:15:53.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdabcded640 1 -- 172.21.15.70:0/1840906280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40aa9d0 con 0x7fdab8099cc0 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- 172.21.15.70:0/1840906280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab8099cc0 msgr2=0x7fdab809a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 --2- 172.21.15.70:0/1840906280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab8099cc0 0x7fdab809a150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdab80b4300 tx=0x7fdab401c490 comp rx=0 tx=0).stop 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- 172.21.15.70:0/1840906280 shutdown_connections 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 --2- 172.21.15.70:0/1840906280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab8099cc0 0x7fdab809a150 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 --2- 172.21.15.70:0/1840906280 >> v2:172.21.15.98:3300/0 conn(0x7fdab815e1f0 0x7fdab809ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- 172.21.15.70:0/1840906280 >> 172.21.15.70:0/1840906280 conn(0x7fdab80b3690 msgr2=0x7fdab80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- 172.21.15.70:0/1840906280 shutdown_connections 2024-04-22T01:15:53.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- 172.21.15.70:0/1840906280 wait complete. 2024-04-22T01:15:53.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 Processor -- start 2024-04-22T01:15:53.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- start start 2024-04-22T01:15:53.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab806c510 0x7fdab8159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdab8099cc0 0x7fdab80c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 0x7fdab8158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdab816b660 con 0x7fdab815e1f0 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdab809b3a0 con 0x7fdab806c510 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.908+0000 7fdac0053640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdab816b7e0 con 0x7fdab8099cc0 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 0x7fdab8158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 0x7fdab8158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51384/0 (socket says 172.21.15.70:51384) 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 -- 172.21.15.70:0/1065538922 learned_addr learned my addr 172.21.15.70:0/1065538922 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabd5ee640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdab8099cc0 0x7fdab80c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabddef640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab806c510 0x7fdab8159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 -- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdab8099cc0 msgr2=0x7fdab80c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdab8099cc0 0x7fdab80c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 -- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab806c510 msgr2=0x7fdab8159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab806c510 0x7fdab8159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 -- 172.21.15.70:0/1065538922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80c94c0 con 0x7fdab815e1f0 2024-04-22T01:15:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabd5ee640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdab8099cc0 0x7fdab80c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabddef640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab806c510 0x7fdab8159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabe5f0640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 0x7fdab8158d60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdab000c710 tx=0x7fdab000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0016020 con 0x7fdab815e1f0 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab0006d60 con 0x7fdab815e1f0 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab80c97a0 con 0x7fdab815e1f0 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0003bc0 con 0x7fdab815e1f0 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab80c9d60 con 0x7fdab815e1f0 2024-04-22T01:15:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab80ca230 con 0x7fdab815e1f0 2024-04-22T01:15:53.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdab000a850 con 0x7fdab815e1f0 2024-04-22T01:15:53.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdaae7fc640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda9c0678c0 0x7fda9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabddef640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda9c0678c0 0x7fda9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fdab0086a40 con 0x7fdab815e1f0 2024-04-22T01:15:53.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.912+0000 7fdabddef640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda9c0678c0 0x7fda9c069d80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdab81fabb0 tx=0x7fdaa8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:53.916+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdab0020020 con 0x7fdab815e1f0 2024-04-22T01:15:54.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.088+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdab804f510 con 0x7fdab815e1f0 2024-04-22T01:15:54.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdaae7fc640 1 -- 172.21.15.70:0/1065538922 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fdab00572f0 con 0x7fdab815e1f0 2024-04-22T01:15:54.093 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda9c0678c0 msgr2=0x7fda9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda9c0678c0 0x7fda9c069d80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdab81fabb0 tx=0x7fdaa8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 msgr2=0x7fdab8158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 0x7fdab8158d60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdab000c710 tx=0x7fdab000cbd0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 shutdown_connections 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda9c0678c0 0x7fda9c069d80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdab815e1f0 0x7fdab8158d60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdab8099cc0 0x7fdab80c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 --2- 172.21.15.70:0/1065538922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab806c510 0x7fdab8159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 >> 172.21.15.70:0/1065538922 conn(0x7fdab80b3690 msgr2=0x7fdab80b7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 shutdown_connections 2024-04-22T01:15:54.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.092+0000 7fdac0053640 1 -- 172.21.15.70:0/1065538922 wait complete. 2024-04-22T01:15:54.112 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:15:54.112 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:15:54.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:54.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.228+0000 7fc4fc4b8640 1 Processor -- start 2024-04-22T01:15:54.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.228+0000 7fc4fc4b8640 1 -- start start 2024-04-22T01:15:54.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fc4b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f415ac40 0x7fc4f415e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fc4b8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4f40ce730 0x7fc4f40d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fc4b8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f40d11c0 con 0x7fc4f40d0d30 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fc4b8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc4f40ad9b0 con 0x7fc4f40ce730 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fc4b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f415eec0 con 0x7fc4f415ac40 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4f40ce730 0x7fc4f40d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4f40ce730 0x7fc4f40d07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51390/0 (socket says 172.21.15.70:51390) 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 -- 172.21.15.70:0/2640051773 learned_addr learned my addr 172.21.15.70:0/2640051773 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4faa55640 1 --1- 172.21.15.70:0/2640051773 >> v1:172.21.15.70:6789/0 conn(0x7fc4f40d0d30 0x7fc4f406efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38860/0 (socket says 172.21.15.70:38860) 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fa254640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f415ac40 0x7fc4f415e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 -- 172.21.15.70:0/2640051773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f415ac40 msgr2=0x7fc4f415e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f415ac40 0x7fc4f415e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 -- 172.21.15.70:0/2640051773 >> v1:172.21.15.70:6789/0 conn(0x7fc4f40d0d30 legacy=0x7fc4f406efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 -- 172.21.15.70:0/2640051773 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f415f180 con 0x7fc4f40ce730 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9252640 1 -- 172.21.15.70:0/2640051773 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2652664579 0 0) 0x7fc4f40d11c0 con 0x7fc4f40d0d30 2024-04-22T01:15:54.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4fa254640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f415ac40 0x7fc4f415e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:54.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9a53640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f40ce730 0x7fc4f40d07f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009870 tx=0x7fc4e401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a4816cbbc93ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9252640 1 -- 172.21.15.70:0/2640051773 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40b2070 con 0x7fc4f40ce730 2024-04-22T01:15:54.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9252640 1 -- 172.21.15.70:0/2640051773 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e40a9070 con 0x7fc4f40ce730 2024-04-22T01:15:54.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.232+0000 7fc4f9252640 1 -- 172.21.15.70:0/2640051773 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40ad880 con 0x7fc4f40ce730 2024-04-22T01:15:54.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2640051773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f40ce730 msgr2=0x7fc4f40d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f40ce730 0x7fc4f40d07f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009870 tx=0x7fc4e401c560 comp rx=0 tx=0).stop 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2640051773 shutdown_connections 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f40ce730 0x7fc4f40d07f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2640051773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f415ac40 0x7fc4f415e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2640051773 >> 172.21.15.70:0/2640051773 conn(0x7fc4f4099a60 msgr2=0x7fc4f4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2640051773 shutdown_connections 2024-04-22T01:15:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2640051773 wait complete. 2024-04-22T01:15:54.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 Processor -- start 2024-04-22T01:15:54.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- start start 2024-04-22T01:15:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f40ce730 0x7fc4f40cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f40d0d30 0x7fc4f415d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 0x7fc4f4158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4167c70 con 0x7fc4f415ac40 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4167af0 con 0x7fc4f40d0d30 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f4167df0 con 0x7fc4f40ce730 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 0x7fc4f4158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 0x7fc4f4158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51406/0 (socket says 172.21.15.70:51406) 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 -- 172.21.15.70:0/2657672465 learned_addr learned my addr 172.21.15.70:0/2657672465 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 -- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f40ce730 msgr2=0x7fc4f40cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fa254640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f40ce730 0x7fc4f40cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f40ce730 0x7fc4f40cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 -- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f40d0d30 msgr2=0x7fc4f415d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4f9a53640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f40d0d30 0x7fc4f415d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f40d0d30 0x7fc4f415d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 -- 172.21.15.70:0/2657672465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f415d900 con 0x7fc4f415ac40 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4faa55640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 0x7fc4f4158a80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00c7a0 tx=0x7fc4ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec018070 con 0x7fc4f415ac40 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ec00a640 con 0x7fc4f415ac40 2024-04-22T01:15:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f40cb5a0 con 0x7fc4f415ac40 2024-04-22T01:15:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec00aab0 con 0x7fc4f415ac40 2024-04-22T01:15:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.236+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f40cbb60 con 0x7fc4f415ac40 2024-04-22T01:15:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.240+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f4068920 con 0x7fc4f415ac40 2024-04-22T01:15:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.240+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc4ec01b070 con 0x7fc4f415ac40 2024-04-22T01:15:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.240+0000 7fc4eaffd640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4d80678c0 0x7fc4d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.240+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc4ec0856b0 con 0x7fc4f415ac40 2024-04-22T01:15:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.240+0000 7fc4fa254640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4d80678c0 0x7fc4d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.240+0000 7fc4fa254640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4d80678c0 0x7fc4d8069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0006fd0 tx=0x7fc4f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.244+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc4ec055df0 con 0x7fc4f415ac40 2024-04-22T01:15:54.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.408+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc4f40cd450 con 0x7fc4f415ac40 2024-04-22T01:15:54.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.408+0000 7fc4eaffd640 1 -- 172.21.15.70:0/2657672465 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7fc4ec0556f0 con 0x7fc4f415ac40 2024-04-22T01:15:54.413 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:54.413 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4d80678c0 msgr2=0x7fc4d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4d80678c0 0x7fc4d8069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0006fd0 tx=0x7fc4f0008040 comp rx=0 tx=0).stop 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 msgr2=0x7fc4f4158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 0x7fc4f4158a80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00c7a0 tx=0x7fc4ec00cc60 comp rx=0 tx=0).stop 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 shutdown_connections 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4d80678c0 0x7fc4d8069d80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4f415ac40 0x7fc4f4158a80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f40d0d30 0x7fc4f415d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 --2- 172.21.15.70:0/2657672465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4f40ce730 0x7fc4f40cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 >> 172.21.15.70:0/2657672465 conn(0x7fc4f4099a60 msgr2=0x7fc4f4164920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 shutdown_connections 2024-04-22T01:15:54.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.412+0000 7fc4fc4b8640 1 -- 172.21.15.70:0/2657672465 wait complete. 2024-04-22T01:15:54.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:54.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:54.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.540+0000 7f0501198640 1 Processor -- start 2024-04-22T01:15:54.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f0501198640 1 -- start start 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f0501198640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc06f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f0501198640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04fc071b70 0x7f04fc071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f0501198640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc06da70 con 0x7f04fc0b3d60 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f0501198640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f04fc072510 con 0x7f04fc071b70 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f0501198640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f04fc06dd50 con 0x7f04fc15a320 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fa59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04fc071b70 0x7f04fc071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fb59e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f04fc0b3d60 0x7f04fc0b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38868/0 (socket says 172.21.15.70:38868) 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fb59e640 1 -- 172.21.15.70:0/3630698922 learned_addr learned my addr 172.21.15.70:0/3630698922 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fad9d640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fa59c640 1 -- 172.21.15.70:0/3630698922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 msgr2=0x7f04fc06f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fa59c640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc06f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fa59c640 1 -- 172.21.15.70:0/3630698922 >> v1:172.21.15.70:6789/0 conn(0x7f04fc0b3d60 legacy=0x7f04fc0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04f9d9b640 1 -- 172.21.15.70:0/3630698922 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391788128 0 0) 0x7f04fc06da70 con 0x7f04fc0b3d60 2024-04-22T01:15:54.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fa59c640 1 -- 172.21.15.70:0/3630698922 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc06e030 con 0x7f04fc071b70 2024-04-22T01:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fad9d640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc06f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.544+0000 7f04fa59c640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc071fd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f04f00097c0 tx=0x7f04f001d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb8029ff689015c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04f9d9b640 1 -- 172.21.15.70:0/3630698922 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f00b3070 con 0x7f04fc071b70 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04f9d9b640 1 -- 172.21.15.70:0/3630698922 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f00a6750 con 0x7f04fc071b70 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04f9d9b640 1 -- 172.21.15.70:0/3630698922 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f00a6b20 con 0x7f04fc071b70 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3630698922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 msgr2=0x7f04fc071fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc071fd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f04f00097c0 tx=0x7f04f001d6a0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3630698922 shutdown_connections 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc071fd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 --2- 172.21.15.70:0/3630698922 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc06f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3630698922 >> 172.21.15.70:0/3630698922 conn(0x7f04fc0b1d40 msgr2=0x7f04fc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3630698922 shutdown_connections 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3630698922 wait complete. 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 Processor -- start 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- start start 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc0b3d60 0x7f04fc164560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc0cf5a0 con 0x7f04fc071b70 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc0cf420 con 0x7f04fc0b3d60 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f04fc06f690 con 0x7f04fc15a320 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51436/0 (socket says 172.21.15.70:51436) 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 -- 172.21.15.70:0/3345290546 learned_addr learned my addr 172.21.15.70:0/3345290546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 -- 172.21.15.70:0/3345290546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 msgr2=0x7f04fc1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 -- 172.21.15.70:0/3345290546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc0b3d60 msgr2=0x7f04fc164560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc0b3d60 0x7f04fc164560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 -- 172.21.15.70:0/3345290546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc152a90 con 0x7f04fc071b70 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04fad9d640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc152320 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00c7c0 tx=0x7f04ec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec007a30 con 0x7f04fc071b70 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04ec007bd0 con 0x7f04fc071b70 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0113f0 con 0x7f04fc071b70 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04fc164aa0 con 0x7f04fc071b70 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.548+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04fc1568f0 con 0x7f04fc071b70 2024-04-22T01:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.552+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04fc152c60 con 0x7f04fc071b70 2024-04-22T01:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.552+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f04ec011590 con 0x7f04fc071b70 2024-04-22T01:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.552+0000 7f04e37fe640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04e80678c0 0x7f04e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.552+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f04ec086220 con 0x7f04fc071b70 2024-04-22T01:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.552+0000 7f04fa59c640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04e80678c0 0x7f04e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.552+0000 7f04fa59c640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04e80678c0 0x7f04e8069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f04f00b0000 tx=0x7f04f0004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.556+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f04ec015020 con 0x7f04fc071b70 2024-04-22T01:15:54.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.704+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04fc068210 con 0x7f04fc071b70 2024-04-22T01:15:54.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.704+0000 7f04e37fe640 1 -- 172.21.15.70:0/3345290546 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f04ec056990 con 0x7f04fc071b70 2024-04-22T01:15:54.708 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04e80678c0 msgr2=0x7f04e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04e80678c0 0x7f04e8069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f04f00b0000 tx=0x7f04f0004740 comp rx=0 tx=0).stop 2024-04-22T01:15:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 msgr2=0x7f04fc152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc152320 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00c7c0 tx=0x7f04ec00cc80 comp rx=0 tx=0).stop 2024-04-22T01:15:54.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 shutdown_connections 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04e80678c0 0x7f04e8069d80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc15a320 0x7f04fc1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc0b3d60 0x7f04fc164560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 --2- 172.21.15.70:0/3345290546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04fc071b70 0x7f04fc152320 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 >> 172.21.15.70:0/3345290546 conn(0x7f04fc0b1d40 msgr2=0x7f04fc0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 shutdown_connections 2024-04-22T01:15:54.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.708+0000 7f0501198640 1 -- 172.21.15.70:0/3345290546 wait complete. 2024-04-22T01:15:54.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:54.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c76397640 1 Processor -- start 2024-04-22T01:15:54.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c76397640 1 -- start start 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c76397640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 0x7f8c7006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c76397640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c700b3800 con 0x7f8c700ce920 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c76397640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c700b3ae0 con 0x7f8c7015e6e0 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c76397640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c700cde40 con 0x7f8c70155370 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c677fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8c700ce920 0x7f8c70154c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38872/0 (socket says 172.21.15.70:38872) 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c677fe640 1 -- 172.21.15.70:0/3997961388 learned_addr learned my addr 172.21.15.70:0/3997961388 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 988021206 0 0) 0x7f8c700b3800 con 0x7f8c700ce920 2024-04-22T01:15:54.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c5c003530 con 0x7f8c700ce920 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 0x7f8c7006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 970527115 0 0) 0x7f8c5c003530 con 0x7f8c700ce920 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 msgr2=0x7f8c7006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 0x7f8c7006d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 >> v1:172.21.15.98:6789/0 conn(0x7f8c7015e6e0 legacy=0x7f8c7015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c700ce120 con 0x7f8c700ce920 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f8c58001680 con 0x7f8c700ce920 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 >> v1:172.21.15.70:6789/0 conn(0x7f8c700ce920 legacy=0x7f8c70154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 0x7f8c5c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c004020 0x7f8c5c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c5c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700cde40 con 0x7f8c5c004d50 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c00bb90 con 0x7f8c5c004020 2024-04-22T01:15:54.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5c00bd90 con 0x7f8c70155370 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c5c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c6ffff640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 0x7f8c5c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c677fe640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c004020 0x7f8c5c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 -- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 msgr2=0x7f8c5c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c70155370 0x7f8c5c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 -- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c004020 msgr2=0x7f8c5c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c004020 0x7f8c5c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.836+0000 7f8c74934640 1 -- 172.21.15.70:0/3997961388 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5c003530 con 0x7f8c5c004d50 2024-04-22T01:15:54.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c74934640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c5c0051d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8c68007360 tx=0x7f8c6801ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3de7f827993fca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680b2070 con 0x7f8c5c004d50 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c680a5d40 con 0x7f8c5c004d50 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6f7fe640 1 -- 172.21.15.70:0/3997961388 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68009be0 con 0x7f8c5c004d50 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- 172.21.15.70:0/3997961388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 msgr2=0x7f8c5c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 --2- 172.21.15.70:0/3997961388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c5c0051d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8c68007360 tx=0x7f8c6801ce40 comp rx=0 tx=0).stop 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c74934640 1 -- 172.21.15.70:0/3997961388 reap_dead start 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- 172.21.15.70:0/3997961388 shutdown_connections 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- 172.21.15.70:0/3997961388 >> 172.21.15.70:0/3997961388 conn(0x7f8c700b1d40 msgr2=0x7f8c700cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- 172.21.15.70:0/3997961388 shutdown_connections 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- 172.21.15.70:0/3997961388 wait complete. 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 Processor -- start 2024-04-22T01:15:54.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- start start 2024-04-22T01:15:54.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c700adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7015c220 0x7f8c70157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c70157cb0 0x7f8c70158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7006f260 con 0x7f8c5c004d50 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7006f0e0 con 0x7f8c70157cb0 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c700cedd0 con 0x7f8c7015c220 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c700adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c700adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51446/0 (socket says 172.21.15.70:51446) 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 -- 172.21.15.70:0/1413473039 learned_addr learned my addr 172.21.15.70:0/1413473039 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 -- 172.21.15.70:0/1413473039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7015c220 msgr2=0x7f8c70157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7015c220 0x7f8c70157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 -- 172.21.15.70:0/1413473039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c70157cb0 msgr2=0x7f8c70158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c70157cb0 0x7f8c70158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 -- 172.21.15.70:0/1413473039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7015ca00 con 0x7f8c5c004d50 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6ffff640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c700adad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000e580 tx=0x7f8c6000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c600052e0 con 0x7f8c5c004d50 2024-04-22T01:15:54.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c60018070 con 0x7f8c5c004d50 2024-04-22T01:15:54.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.840+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7015cce0 con 0x7f8c5c004d50 2024-04-22T01:15:54.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60003820 con 0x7f8c5c004d50 2024-04-22T01:15:54.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7015d2a0 con 0x7f8c5c004d50 2024-04-22T01:15:54.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c70068210 con 0x7f8c5c004d50 2024-04-22T01:15:54.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8c600039c0 con 0x7f8c5c004d50 2024-04-22T01:15:54.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c6d7fa640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c500678c0 0x7f8c50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8c60085d50 con 0x7f8c5c004d50 2024-04-22T01:15:54.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c677fe640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c500678c0 0x7f8c50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.844+0000 7f8c677fe640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c500678c0 0x7f8c50069d80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c58001210 tx=0x7f8c58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:54.848+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8c60089030 con 0x7f8c5c004d50 2024-04-22T01:15:55.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.008+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c700b2b10 con 0x7f8c5c004d50 2024-04-22T01:15:55.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.008+0000 7f8c6d7fa640 1 -- 172.21.15.70:0/1413473039 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f8c600564a0 con 0x7f8c5c004d50 2024-04-22T01:15:55.012 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:55.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c500678c0 msgr2=0x7f8c50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c500678c0 0x7f8c50069d80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c58001210 tx=0x7f8c58009040 comp rx=0 tx=0).stop 2024-04-22T01:15:55.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 msgr2=0x7f8c700adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c700adad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000e580 tx=0x7f8c6000ea40 comp rx=0 tx=0).stop 2024-04-22T01:15:55.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 shutdown_connections 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c500678c0 0x7f8c50069d80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c70157cb0 0x7f8c70158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7015c220 0x7f8c70157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 --2- 172.21.15.70:0/1413473039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c004d50 0x7f8c700adad0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 >> 172.21.15.70:0/1413473039 conn(0x7f8c700b1d40 msgr2=0x7f8c7015eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 shutdown_connections 2024-04-22T01:15:55.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.012+0000 7f8c76397640 1 -- 172.21.15.70:0/1413473039 wait complete. 2024-04-22T01:15:55.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:55.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 Processor -- start 2024-04-22T01:15:55.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 -- start start 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d8415e6e0 0x7f7d8415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7d840ce920 0x7f7d84154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d840cedd0 0x7f7d841551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8415f000 con 0x7f7d840ce920 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7d840b3800 con 0x7f7d8415e6e0 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8aef1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7d840b3ae0 con 0x7f7d840cedd0 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7d840ce920 0x7f7d84154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7d840ce920 0x7f7d84154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51462/0 (socket says 172.21.15.70:51462) 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 -- 172.21.15.70:0/1364097581 learned_addr learned my addr 172.21.15.70:0/1364097581 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 -- 172.21.15.70:0/1364097581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d840cedd0 msgr2=0x7f7d841551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8948e640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d840cedd0 0x7f7d841551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d88c8d640 1 --2- 172.21.15.70:0/1364097581 >> v2:172.21.15.98:3300/0 conn(0x7f7d8415e6e0 0x7f7d8415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d840cedd0 0x7f7d841551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 -- 172.21.15.70:0/1364097581 >> v2:172.21.15.98:3300/0 conn(0x7f7d8415e6e0 msgr2=0x7f7d8415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 --2- 172.21.15.70:0/1364097581 >> v2:172.21.15.98:3300/0 conn(0x7f7d8415e6e0 0x7f7d8415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 -- 172.21.15.70:0/1364097581 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d840cde40 con 0x7f7d840ce920 2024-04-22T01:15:55.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d8948e640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d840cedd0 0x7f7d841551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:55.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.144+0000 7f7d83fff640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d84154c60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406fdc0 tx=0x7f7d7801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a79a76f67dadeba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d837fe640 1 -- 172.21.15.70:0/1364097581 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780aa610 con 0x7f7d840ce920 2024-04-22T01:15:55.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d837fe640 1 -- 172.21.15.70:0/1364097581 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d780af070 con 0x7f7d840ce920 2024-04-22T01:15:55.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d837fe640 1 -- 172.21.15.70:0/1364097581 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780aa9d0 con 0x7f7d840ce920 2024-04-22T01:15:55.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/1364097581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 msgr2=0x7f7d84154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d84154c60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406fdc0 tx=0x7f7d7801c4f0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/1364097581 shutdown_connections 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d840cedd0 0x7f7d841551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/1364097581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d84154c60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/1364097581 >> v2:172.21.15.98:3300/0 conn(0x7f7d8415e6e0 0x7f7d8415eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/1364097581 >> 172.21.15.70:0/1364097581 conn(0x7f7d840b1d40 msgr2=0x7f7d840cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/1364097581 shutdown_connections 2024-04-22T01:15:55.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/1364097581 wait complete. 2024-04-22T01:15:55.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 Processor -- start 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- start start 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d8413f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d840cedd0 0x7f7d84151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d8415e6e0 0x7f7d841646f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d840b2c30 con 0x7f7d840ce920 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d840b2ab0 con 0x7f7d840cedd0 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7d840b2db0 con 0x7f7d8415e6e0 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d8413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d8413f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51470/0 (socket says 172.21.15.70:51470) 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 -- 172.21.15.70:0/992825413 learned_addr learned my addr 172.21.15.70:0/992825413 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:55.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 -- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d8415e6e0 msgr2=0x7f7d841646f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d8415e6e0 0x7f7d841646f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d83fff640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d840cedd0 0x7f7d84151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 -- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d840cedd0 msgr2=0x7f7d84151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d840cedd0 0x7f7d84151a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 -- 172.21.15.70:0/992825413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d84150600 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d88c8d640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d8413f570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c7f0 tx=0x7f7d7400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74015070 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d74007820 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8414fa40 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74007c10 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.148+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d84150320 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.152+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d84068210 con 0x7f7d840ce920 2024-04-22T01:15:55.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.152+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d74018070 con 0x7f7d840ce920 2024-04-22T01:15:55.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.152+0000 7f7d817fa640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7d700678c0 0x7f7d70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.152+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7d74086120 con 0x7f7d840ce920 2024-04-22T01:15:55.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.152+0000 7f7d83fff640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7d700678c0 0x7f7d70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.152+0000 7f7d83fff640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7d700678c0 0x7f7d70069d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d780ae000 tx=0x7f7d78008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.156+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7d84150600 con 0x7f7d840ce920 2024-04-22T01:15:55.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.312+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7d84164030 con 0x7f7d840ce920 2024-04-22T01:15:55.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.312+0000 7f7d817fa640 1 -- 172.21.15.70:0/992825413 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f7d74056860 con 0x7f7d840ce920 2024-04-22T01:15:55.317 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7d700678c0 msgr2=0x7f7d70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7d700678c0 0x7f7d70069d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d780ae000 tx=0x7f7d78008040 comp rx=0 tx=0).stop 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 msgr2=0x7f7d8413f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d8413f570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c7f0 tx=0x7f7d7400ccb0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 shutdown_connections 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7d700678c0 0x7f7d70069d80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d8415e6e0 0x7f7d841646f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d840cedd0 0x7f7d84151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 --2- 172.21.15.70:0/992825413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d840ce920 0x7f7d8413f570 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 >> 172.21.15.70:0/992825413 conn(0x7f7d840b1d40 msgr2=0x7f7d84155650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 shutdown_connections 2024-04-22T01:15:55.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.316+0000 7f7d8aef1640 1 -- 172.21.15.70:0/992825413 wait complete. 2024-04-22T01:15:55.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:55.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.456+0000 7f16cb1e1640 1 Processor -- start 2024-04-22T01:15:55.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.456+0000 7f16cb1e1640 1 -- start start 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16cb1e1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f16c40b4450 0x7f16c4071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16cb1e1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c415c290 con 0x7f16c40cd220 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16cb1e1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c415c570 con 0x7f16c40720b0 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16cb1e1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f16c415c850 con 0x7f16c40b4450 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f16c40cd220 0x7f16c40b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38880/0 (socket says 172.21.15.70:38880) 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 -- 172.21.15.70:0/3053917651 learned_addr learned my addr 172.21.15.70:0/3053917651 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c8f7d640 1 --2- 172.21.15.70:0/3053917651 >> v2:172.21.15.98:3300/0 conn(0x7f16c40b4450 0x7f16c4071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361701976 0 0) 0x7f16c415c290 con 0x7f16c40cd220 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16b4003530 con 0x7f16c40cd220 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1774957456 0 0) 0x7f16b4003530 con 0x7f16c40cd220 2024-04-22T01:15:55.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 >> v2:172.21.15.98:3300/0 conn(0x7f16c40b4450 msgr2=0x7f16c4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 --2- 172.21.15.70:0/3053917651 >> v2:172.21.15.98:3300/0 conn(0x7f16c40b4450 0x7f16c4071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 >> v1:172.21.15.98:6789/0 conn(0x7f16c40720b0 legacy=0x7f16c40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c415cb30 con 0x7f16c40cd220 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f16bc001a00 con 0x7f16c40cd220 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 >> v1:172.21.15.70:6789/0 conn(0x7f16c40cd220 legacy=0x7f16c40b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16b40039c0 0x7f16b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4004810 0x7f16b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b40051d0 0x7f16b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16c415c850 con 0x7f16b40051d0 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16b400c070 con 0x7f16b4004810 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16b400c270 con 0x7f16b40039c0 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b40051d0 0x7f16b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c3fff640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4004810 0x7f16b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 -- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16b40039c0 msgr2=0x7f16b4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16b40039c0 0x7f16b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 -- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4004810 msgr2=0x7f16b4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4004810 0x7f16b4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.460+0000 7f16c977e640 1 -- 172.21.15.70:0/3053917651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b4003530 con 0x7f16b40051d0 2024-04-22T01:15:55.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 -- 172.21.15.70:0/3053917651 reap_dead start 2024-04-22T01:15:55.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b40051d0 0x7f16b400b9d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00a840 tx=0x7f16bc01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d171e167d146c764 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00a460 con 0x7f16b40051d0 2024-04-22T01:15:55.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16bc0ac640 con 0x7f16b40051d0 2024-04-22T01:15:55.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c37fe640 1 -- 172.21.15.70:0/3053917651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0aca30 con 0x7f16b40051d0 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3053917651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b40051d0 msgr2=0x7f16b400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b40051d0 0x7f16b400b9d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00a840 tx=0x7f16bc01d360 comp rx=0 tx=0).stop 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3053917651 shutdown_connections 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3053917651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b40051d0 0x7f16b400b9d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3053917651 >> 172.21.15.70:0/3053917651 conn(0x7f16c40b1d40 msgr2=0x7f16c40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3053917651 shutdown_connections 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3053917651 wait complete. 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 Processor -- start 2024-04-22T01:15:55.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- start start 2024-04-22T01:15:55.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b40051d0 0x7f16c4141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16c406e3d0 0x7f16c415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 0x7f16c4143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16c40b49b0 con 0x7f16c415b160 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16c40b4830 con 0x7f16b40051d0 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16c40cd650 con 0x7f16c406e3d0 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 0x7f16c4143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 0x7f16c4143fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51488/0 (socket says 172.21.15.70:51488) 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 -- 172.21.15.70:0/3379473911 learned_addr learned my addr 172.21.15.70:0/3379473911 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c3fff640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16c406e3d0 0x7f16c415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c8f7d640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b40051d0 0x7f16c4141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 -- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16c406e3d0 msgr2=0x7f16c415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16c406e3d0 0x7f16c415ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 -- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b40051d0 msgr2=0x7f16c4141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b40051d0 0x7f16c4141590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 -- 172.21.15.70:0/3379473911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c406f050 con 0x7f16c415b160 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c977e640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 0x7f16c4143fe0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f16bc01d190 tx=0x7f16b4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0b0030 con 0x7f16c415b160 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16bc0019c0 con 0x7f16c415b160 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c4144c10 con 0x7f16c415b160 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.464+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0ac9f0 con 0x7f16c415b160 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c406d8f0 con 0x7f16c415b160 2024-04-22T01:15:55.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c415b5e0 con 0x7f16c415b160 2024-04-22T01:15:55.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f16bc0c1430 con 0x7f16c415b160 2024-04-22T01:15:55.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16c17fa640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16a80678c0 0x7f16a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f16bc1202b0 con 0x7f16c415b160 2024-04-22T01:15:55.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16c8f7d640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16a80678c0 0x7f16a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.468+0000 7f16c8f7d640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16a80678c0 0x7f16a8069d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f16b80047c0 tx=0x7f16b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.472+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f16bc017070 con 0x7f16c415b160 2024-04-22T01:15:55.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.620+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f16c4068210 con 0x7f16c415b160 2024-04-22T01:15:55.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.620+0000 7f16c17fa640 1 -- 172.21.15.70:0/3379473911 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f16bc0f0380 con 0x7f16c415b160 2024-04-22T01:15:55.624 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:55.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16a80678c0 msgr2=0x7f16a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16a80678c0 0x7f16a8069d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f16b80047c0 tx=0x7f16b8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 msgr2=0x7f16c4143fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 0x7f16c4143fe0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f16bc01d190 tx=0x7f16b4005840 comp rx=0 tx=0).stop 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 shutdown_connections 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16a80678c0 0x7f16a8069d80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16c415b160 0x7f16c4143fe0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16c406e3d0 0x7f16c415ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 --2- 172.21.15.70:0/3379473911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b40051d0 0x7f16c4141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 >> 172.21.15.70:0/3379473911 conn(0x7f16c40b1d40 msgr2=0x7f16c415d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 shutdown_connections 2024-04-22T01:15:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.624+0000 7f16cb1e1640 1 -- 172.21.15.70:0/3379473911 wait complete. 2024-04-22T01:15:55.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:15:55.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 Processor -- start 2024-04-22T01:15:55.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 -- start start 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa290072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2900b4360 0x7fa2900b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa290144990 con 0x7fa2900cdce0 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa2900b3bd0 con 0x7fa2900b4360 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa29860f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa2900b4d00 con 0x7fa2901445b0 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2900b4360 0x7fa2900b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2900b4360 0x7fa2900b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51496/0 (socket says 172.21.15.70:51496) 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 -- 172.21.15.70:0/3596063904 learned_addr learned my addr 172.21.15.70:0/3596063904 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa2963ab640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa290072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 -- 172.21.15.70:0/3596063904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 msgr2=0x7fa290072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa290072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 -- 172.21.15.70:0/3596063904 >> v1:172.21.15.98:6789/0 conn(0x7fa2900cdce0 legacy=0x7fa2900ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa295baa640 1 -- 172.21.15.70:0/3596063904 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2900b2b50 con 0x7fa2900b4360 2024-04-22T01:15:55.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.760+0000 7fa2963ab640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa290072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:55.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa295baa640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa2900b47c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa284009870 tx=0x7fa28401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec77b00f19b45607 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2953a9640 1 -- 172.21.15.70:0/3596063904 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2840b0070 con 0x7fa2900b4360 2024-04-22T01:15:55.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2953a9640 1 -- 172.21.15.70:0/3596063904 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2840ab600 con 0x7fa2900b4360 2024-04-22T01:15:55.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2953a9640 1 -- 172.21.15.70:0/3596063904 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2840ab9d0 con 0x7fa2900b4360 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- 172.21.15.70:0/3596063904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 msgr2=0x7fa2900b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa2900b47c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa284009870 tx=0x7fa28401c310 comp rx=0 tx=0).stop 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- 172.21.15.70:0/3596063904 shutdown_connections 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa2900b47c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 --2- 172.21.15.70:0/3596063904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa290072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- 172.21.15.70:0/3596063904 >> 172.21.15.70:0/3596063904 conn(0x7fa2900cf9e0 msgr2=0x7fa2900cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- 172.21.15.70:0/3596063904 shutdown_connections 2024-04-22T01:15:55.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- 172.21.15.70:0/3596063904 wait complete. 2024-04-22T01:15:55.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 Processor -- start 2024-04-22T01:15:55.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- start start 2024-04-22T01:15:55.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa290158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2900cdce0 0x7fa290137400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa29013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa290072a40 con 0x7fa2900b4360 2024-04-22T01:15:55.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2900b36b0 con 0x7fa2900cdce0 2024-04-22T01:15:55.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa29860f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa29006edb0 con 0x7fa2901445b0 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa290158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa290158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51498/0 (socket says 172.21.15.70:51498) 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 -- 172.21.15.70:0/641009400 learned_addr learned my addr 172.21.15.70:0/641009400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa295baa640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2900cdce0 0x7fa290137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa296bac640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa29013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 -- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 msgr2=0x7fa29013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa29013ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 -- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2900cdce0 msgr2=0x7fa290137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2900cdce0 0x7fa290137400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa2963ab640 1 -- 172.21.15.70:0/641009400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa290137940 con 0x7fa2900b4360 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.764+0000 7fa295baa640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2900cdce0 0x7fa290137400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa2963ab640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa290158960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c820 tx=0x7fa28800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa288015070 con 0x7fa2900b4360 2024-04-22T01:15:55.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa288007a30 con 0x7fa2900b4360 2024-04-22T01:15:55.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2901fae80 con 0x7fa2900b4360 2024-04-22T01:15:55.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa288007e40 con 0x7fa2900b4360 2024-04-22T01:15:55.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa29013fcb0 con 0x7fa2900b4360 2024-04-22T01:15:55.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa29004f510 con 0x7fa2900b4360 2024-04-22T01:15:55.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa288018070 con 0x7fa2900b4360 2024-04-22T01:15:55.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa28effd640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa27c0678c0 0x7fa27c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa288085f40 con 0x7fa2900b4360 2024-04-22T01:15:55.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa295baa640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa27c0678c0 0x7fa27c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.768+0000 7fa295baa640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa27c0678c0 0x7fa27c069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa2840ad000 tx=0x7fa284008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.772+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa288056710 con 0x7fa2900b4360 2024-04-22T01:15:55.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.948+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa2901f7cb0 con 0x7fa2900b4360 2024-04-22T01:15:55.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.948+0000 7fa28effd640 1 -- 172.21.15.70:0/641009400 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fa288056010 con 0x7fa2900b4360 2024-04-22T01:15:55.951 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:15:55.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa27c0678c0 msgr2=0x7fa27c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa27c0678c0 0x7fa27c069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa2840ad000 tx=0x7fa284008040 comp rx=0 tx=0).stop 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 msgr2=0x7fa290158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa290158960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c820 tx=0x7fa28800cce0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 shutdown_connections 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa27c0678c0 0x7fa27c069d80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2901445b0 0x7fa29013ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2900cdce0 0x7fa290137400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 --2- 172.21.15.70:0/641009400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2900b4360 0x7fa290158960 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 >> 172.21.15.70:0/641009400 conn(0x7fa2900cf9e0 msgr2=0x7fa29015a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 shutdown_connections 2024-04-22T01:15:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:55.952+0000 7fa29860f640 1 -- 172.21.15.70:0/641009400 wait complete. 2024-04-22T01:15:55.970 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:15:55.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:55.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-22T01:15:55.986 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:15:55.986 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:15:55.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:55.986 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:15:56.035 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-22T01:15:56.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:56.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.156+0000 7f6759726640 1 Processor -- start 2024-04-22T01:15:56.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6759726640 1 -- start start 2024-04-22T01:15:56.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6759726640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67540cc150 0x7f6754071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6759726640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6754072060 0x7f675406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6759726640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67540cd7b0 con 0x7f67541455b0 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6759726640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f67540cc710 con 0x7f6754072060 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6759726640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f67540cc9f0 con 0x7f67540cc150 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f67527fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67540cc150 0x7f6754071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6752ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f67541455b0 0x7f67540cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:42310/0 (socket says 172.21.15.70:42310) 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6752ffd640 1 -- 172.21.15.70:0/435482880 learned_addr learned my addr 172.21.15.70:0/435482880 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:56.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f67537fe640 1 --2- 172.21.15.70:0/435482880 >> v2:172.21.15.98:3300/0 conn(0x7f6754072060 0x7f675406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6751ffb640 1 -- 172.21.15.70:0/435482880 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1576052831 0 0) 0x7f67540cd7b0 con 0x7f67541455b0 2024-04-22T01:15:56.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f6751ffb640 1 -- 172.21.15.70:0/435482880 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f673c003530 con 0x7f67541455b0 2024-04-22T01:15:56.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f67527fc640 1 -- 172.21.15.70:0/435482880 >> v2:172.21.15.98:3300/0 conn(0x7f6754072060 msgr2=0x7f675406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:56.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f67527fc640 1 --2- 172.21.15.70:0/435482880 >> v2:172.21.15.98:3300/0 conn(0x7f6754072060 0x7f675406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f67527fc640 1 -- 172.21.15.70:0/435482880 >> v1:172.21.15.98:6789/0 conn(0x7f67541455b0 legacy=0x7f67540cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.160+0000 7f67527fc640 1 -- 172.21.15.70:0/435482880 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67540cccd0 con 0x7f67540cc150 2024-04-22T01:15:56.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f67527fc640 1 --2- 172.21.15.70:0/435482880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67540cc150 0x7f6754071b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6744009d90 tx=0x7f674401c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2af809c558ebbc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6751ffb640 1 -- 172.21.15.70:0/435482880 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67440b2070 con 0x7f67540cc150 2024-04-22T01:15:56.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6751ffb640 1 -- 172.21.15.70:0/435482880 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67440a5710 con 0x7f67540cc150 2024-04-22T01:15:56.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6751ffb640 1 -- 172.21.15.70:0/435482880 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67440a5ac0 con 0x7f67540cc150 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- 172.21.15.70:0/435482880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67540cc150 msgr2=0x7f6754071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 --2- 172.21.15.70:0/435482880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67540cc150 0x7f6754071b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6744009d90 tx=0x7f674401c850 comp rx=0 tx=0).stop 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- 172.21.15.70:0/435482880 shutdown_connections 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 --2- 172.21.15.70:0/435482880 >> v2:172.21.15.98:3300/0 conn(0x7f6754072060 0x7f675406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 --2- 172.21.15.70:0/435482880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67540cc150 0x7f6754071b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- 172.21.15.70:0/435482880 >> 172.21.15.70:0/435482880 conn(0x7f675406b1f0 msgr2=0x7f675406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- 172.21.15.70:0/435482880 shutdown_connections 2024-04-22T01:15:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- 172.21.15.70:0/435482880 wait complete. 2024-04-22T01:15:56.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 Processor -- start 2024-04-22T01:15:56.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- start start 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6754072060 0x7f67541417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67540cc150 0x7f67541592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 0x7f675415b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f675406f2e0 con 0x7f67541455b0 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f675406f160 con 0x7f67540cc150 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6759726640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6754072470 con 0x7f6754072060 2024-04-22T01:15:56.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f67537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 0x7f675415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f67537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 0x7f675415b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34912/0 (socket says 172.21.15.70:34912) 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f67537fe640 1 -- 172.21.15.70:0/208769070 learned_addr learned my addr 172.21.15.70:0/208769070 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.164+0000 7f6752ffd640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6754072060 0x7f67541417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67527fc640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67540cc150 0x7f67541592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67537fe640 1 -- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6754072060 msgr2=0x7f67541417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67537fe640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6754072060 0x7f67541417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67537fe640 1 -- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67540cc150 msgr2=0x7f67541592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67537fe640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67540cc150 0x7f67541592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67537fe640 1 -- 172.21.15.70:0/208769070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6754144c40 con 0x7f67541455b0 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f6752ffd640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6754072060 0x7f67541417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67527fc640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67540cc150 0x7f67541592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f67537fe640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 0x7f675415b260 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f674c00e7f0 tx=0x7f674c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c017070 con 0x7f67541455b0 2024-04-22T01:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674c00c940 con 0x7f67541455b0 2024-04-22T01:15:56.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67541443a0 con 0x7f67541455b0 2024-04-22T01:15:56.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c00eea0 con 0x7f67541455b0 2024-04-22T01:15:56.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6754144960 con 0x7f67541455b0 2024-04-22T01:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6754068210 con 0x7f67541455b0 2024-04-22T01:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f674c01a070 con 0x7f67541455b0 2024-04-22T01:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f673b7fe640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67400678f0 0x7f6740069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f6752ffd640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67400678f0 0x7f6740069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f674c086320 con 0x7f67541455b0 2024-04-22T01:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.168+0000 7f6752ffd640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67400678f0 0x7f6740069db0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6748005890 tx=0x7f6748009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.172+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f674c0569e0 con 0x7f67541455b0 2024-04-22T01:15:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f675415f510 con 0x7f67541455b0 2024-04-22T01:15:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f673b7fe640 1 -- 172.21.15.70:0/208769070 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f674c0562e0 con 0x7f67541455b0 2024-04-22T01:15:56.355 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:56.355 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:56.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67400678f0 msgr2=0x7f6740069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67400678f0 0x7f6740069db0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6748005890 tx=0x7f6748009040 comp rx=0 tx=0).stop 2024-04-22T01:15:56.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 msgr2=0x7f675415b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 0x7f675415b260 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f674c00e7f0 tx=0x7f674c00ecb0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 shutdown_connections 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67400678f0 0x7f6740069db0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541455b0 0x7f675415b260 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67540cc150 0x7f67541592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 --2- 172.21.15.70:0/208769070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6754072060 0x7f67541417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.352+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 >> 172.21.15.70:0/208769070 conn(0x7f675406b1f0 msgr2=0x7f675413fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.356+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 shutdown_connections 2024-04-22T01:15:56.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.356+0000 7f6759726640 1 -- 172.21.15.70:0/208769070 wait complete. 2024-04-22T01:15:56.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:56.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 Processor -- start 2024-04-22T01:15:56.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 -- start start 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb56c0cea80 0x7fb56c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb56c0d2110 0x7fb56c0d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c15e390 con 0x7fb56c15dfb0 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb56c0cf420 con 0x7fb56c0d2110 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb571852640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb56c161e00 con 0x7fb56c0cea80 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb56c0d2110 0x7fb56c0d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb56c0d2110 0x7fb56c0d25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34920/0 (socket says 172.21.15.70:34920) 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1395374730 learned_addr learned my addr 172.21.15.70:0/1395374730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb5627fc640 1 --2- 172.21.15.70:0/1395374730 >> v2:172.21.15.98:3300/0 conn(0x7fb56c0cea80 0x7fb56c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1395374730 >> v2:172.21.15.98:3300/0 conn(0x7fb56c0cea80 msgr2=0x7fb56c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 --2- 172.21.15.70:0/1395374730 >> v2:172.21.15.98:3300/0 conn(0x7fb56c0cea80 0x7fb56c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1395374730 >> v1:172.21.15.98:6789/0 conn(0x7fb56c15dfb0 legacy=0x7fb56c0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1395374730 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c1620e0 con 0x7fb56c0d2110 2024-04-22T01:15:56.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb5627fc640 1 --2- 172.21.15.70:0/1395374730 >> v2:172.21.15.98:3300/0 conn(0x7fb56c0cea80 0x7fb56c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:56.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56b7fe640 1 --2- 172.21.15.70:0/1395374730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0d2110 0x7fb56c0d25a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0d15a0 tx=0x7fb56401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76e2933c4f102e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.492+0000 7fb56a7fc640 1 -- 172.21.15.70:0/1395374730 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5640a5780 con 0x7fb56c0d2110 2024-04-22T01:15:56.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56a7fc640 1 -- 172.21.15.70:0/1395374730 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5640a5920 con 0x7fb56c0d2110 2024-04-22T01:15:56.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56a7fc640 1 -- 172.21.15.70:0/1395374730 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5640a5cf0 con 0x7fb56c0d2110 2024-04-22T01:15:56.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/1395374730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0d2110 msgr2=0x7fb56c0d25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 --2- 172.21.15.70:0/1395374730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0d2110 0x7fb56c0d25a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0d15a0 tx=0x7fb56401c9b0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/1395374730 shutdown_connections 2024-04-22T01:15:56.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 --2- 172.21.15.70:0/1395374730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0d2110 0x7fb56c0d25a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 --2- 172.21.15.70:0/1395374730 >> v2:172.21.15.98:3300/0 conn(0x7fb56c0cea80 0x7fb56c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/1395374730 >> 172.21.15.70:0/1395374730 conn(0x7fb56c06c9b0 msgr2=0x7fb56c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/1395374730 shutdown_connections 2024-04-22T01:15:56.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/1395374730 wait complete. 2024-04-22T01:15:56.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 Processor -- start 2024-04-22T01:15:56.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- start start 2024-04-22T01:15:56.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb56c0cea80 0x7fb56c0c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb56c0d2110 0x7fb56c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 0x7fb56c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c16af50 con 0x7fb56c15dfb0 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c16add0 con 0x7fb56c0d2110 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb56c16b0d0 con 0x7fb56c0cea80 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 0x7fb56c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 0x7fb56c155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34924/0 (socket says 172.21.15.70:34924) 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 -- 172.21.15.70:0/170339608 learned_addr learned my addr 172.21.15.70:0/170339608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb5627fc640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb56c0d2110 0x7fb56c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56affd640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb56c0cea80 0x7fb56c0c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 -- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb56c0cea80 msgr2=0x7fb56c0c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb56c0cea80 0x7fb56c0c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 -- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb56c0d2110 msgr2=0x7fb56c0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb56c0d2110 0x7fb56c0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 -- 172.21.15.70:0/170339608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c158e70 con 0x7fb56c15dfb0 2024-04-22T01:15:56.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb5627fc640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb56c0d2110 0x7fb56c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb56b7fe640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 0x7fb56c155b10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb5640af040 tx=0x7fb564019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5640b8670 con 0x7fb56c15dfb0 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5640be070 con 0x7fb56c15dfb0 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c159110 con 0x7fb56c15dfb0 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5640b8a50 con 0x7fb56c15dfb0 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.496+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c1596d0 con 0x7fb56c15dfb0 2024-04-22T01:15:56.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.500+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c068210 con 0x7fb56c15dfb0 2024-04-22T01:15:56.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.500+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb56c0d32f0 con 0x7fb56c15dfb0 2024-04-22T01:15:56.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.500+0000 7fb568ff9640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb54c0678c0 0x7fb54c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.500+0000 7fb56affd640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb54c0678c0 0x7fb54c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.500+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb564120f00 con 0x7fb56c15dfb0 2024-04-22T01:15:56.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.500+0000 7fb56affd640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb54c0678c0 0x7fb54c069d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00a750 tx=0x7fb55c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.504+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb5640165a0 con 0x7fb56c15dfb0 2024-04-22T01:15:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.660+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb56c0b6210 con 0x7fb56c15dfb0 2024-04-22T01:15:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.660+0000 7fb568ff9640 1 -- 172.21.15.70:0/170339608 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7fb5640f1640 con 0x7fb56c15dfb0 2024-04-22T01:15:56.664 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:56.665 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb54c0678c0 msgr2=0x7fb54c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb54c0678c0 0x7fb54c069d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00a750 tx=0x7fb55c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 msgr2=0x7fb56c155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 0x7fb56c155b10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb5640af040 tx=0x7fb564019750 comp rx=0 tx=0).stop 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 shutdown_connections 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb54c0678c0 0x7fb54c069d80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c15dfb0 0x7fb56c155b10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb56c0d2110 0x7fb56c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 --2- 172.21.15.70:0/170339608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb56c0cea80 0x7fb56c0c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 >> 172.21.15.70:0/170339608 conn(0x7fb56c06c9b0 msgr2=0x7fb56c0b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 shutdown_connections 2024-04-22T01:15:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.664+0000 7fb571852640 1 -- 172.21.15.70:0/170339608 wait complete. 2024-04-22T01:15:56.682 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:14:44.515300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:14:45.021580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:14:48.773096+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:56.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:56.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f6991e59640 1 Processor -- start 2024-04-22T01:15:56.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f6991e59640 1 -- start start 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f6991e59640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f698c06b900 0x7f698c0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f6991e59640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c0b2440 con 0x7f698c0cc9b0 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f6991e59640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c0b0f80 con 0x7f698c0cc5d0 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f6991e59640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f698c0b18b0 con 0x7f698c06b900 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f698c0cc9b0 0x7f698c06b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53360/0 (socket says 172.21.15.70:53360) 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 -- 172.21.15.70:0/2611224191 learned_addr learned my addr 172.21.15.70:0/2611224191 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698bfff640 1 --2- 172.21.15.70:0/2611224191 >> v2:172.21.15.98:3300/0 conn(0x7f698c06b900 0x7f698c0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165872419 0 0) 0x7f698c0b2440 con 0x7f698c0cc9b0 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6978003530 con 0x7f698c0cc9b0 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1050226598 0 0) 0x7f6978003530 con 0x7f698c0cc9b0 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 >> v2:172.21.15.98:3300/0 conn(0x7f698c06b900 msgr2=0x7f698c0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 --2- 172.21.15.70:0/2611224191 >> v2:172.21.15.98:3300/0 conn(0x7f698c06b900 0x7f698c0ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 >> v1:172.21.15.98:6789/0 conn(0x7f698c0cc5d0 legacy=0x7f698c0b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c0b1260 con 0x7f698c0cc9b0 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f697c001680 con 0x7f698c0cc9b0 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 >> v1:172.21.15.70:6789/0 conn(0x7f698c0cc9b0 legacy=0x7f698c06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978003a10 0x7f6978003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6978004810 0x7f6978004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978005220 0x7f697800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f697800c140 con 0x7f6978004810 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f697800c2f0 con 0x7f6978003a10 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f697800c4f0 con 0x7f6978005220 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6978004810 0x7f6978004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698b7fe640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978003a10 0x7f6978003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698bfff640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978005220 0x7f697800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 -- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978005220 msgr2=0x7f697800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:56.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978005220 0x7f697800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 -- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978003a10 msgr2=0x7f6978003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:56.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978003a10 0x7f6978003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698bfff640 1 -- 172.21.15.70:0/2611224191 reap_dead start 2024-04-22T01:15:56.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698affd640 1 -- 172.21.15.70:0/2611224191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6978003530 con 0x7f6978004810 2024-04-22T01:15:56.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.800+0000 7f698b7fe640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978003a10 0x7f6978003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f698affd640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6978004810 0x7f6978004c90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f698c0ccdd0 tx=0x7f697c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bd660807f75961 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c007c30 con 0x7f6978004810 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697c0a9020 con 0x7f6978004810 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f698a7fc640 1 -- 172.21.15.70:0/2611224191 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c005b00 con 0x7f6978004810 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- 172.21.15.70:0/2611224191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6978004810 msgr2=0x7f6978004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6978004810 0x7f6978004c90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f698c0ccdd0 tx=0x7f697c01ca90 comp rx=0 tx=0).stop 2024-04-22T01:15:56.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- 172.21.15.70:0/2611224191 shutdown_connections 2024-04-22T01:15:56.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 --2- 172.21.15.70:0/2611224191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6978004810 0x7f6978004c90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- 172.21.15.70:0/2611224191 >> 172.21.15.70:0/2611224191 conn(0x7f698c099a60 msgr2=0x7f698c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- 172.21.15.70:0/2611224191 shutdown_connections 2024-04-22T01:15:56.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- 172.21.15.70:0/2611224191 wait complete. 2024-04-22T01:15:56.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 Processor -- start 2024-04-22T01:15:56.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- start start 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978004810 0x7f698c0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 0x7f698c152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698c1530d0 0x7f698c0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b1de0 con 0x7f698c0c8570 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b1c60 con 0x7f698c1530d0 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.804+0000 7f6991e59640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f698c0b1f60 con 0x7f6978004810 2024-04-22T01:15:56.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 0x7f698c152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 0x7f698c152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34938/0 (socket says 172.21.15.70:34938) 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 -- 172.21.15.70:0/1257449332 learned_addr learned my addr 172.21.15.70:0/1257449332 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698bfff640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698c1530d0 0x7f698c0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698b7fe640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978004810 0x7f698c0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 -- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978004810 msgr2=0x7f698c0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978004810 0x7f698c0b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 -- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698c1530d0 msgr2=0x7f698c0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698c1530d0 0x7f698c0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 -- 172.21.15.70:0/1257449332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c153650 con 0x7f698c0c8570 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698b7fe640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978004810 0x7f698c0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698affd640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 0x7f698c152b90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f697c01c8c0 tx=0x7f6978005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c0049f0 con 0x7f698c0c8570 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697c0b2070 con 0x7f698c0c8570 2024-04-22T01:15:56.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c153930 con 0x7f698c0c8570 2024-04-22T01:15:56.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c004dd0 con 0x7f698c0c8570 2024-04-22T01:15:56.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c0c8050 con 0x7f698c0c8570 2024-04-22T01:15:56.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c0c7b90 con 0x7f698c0c8570 2024-04-22T01:15:56.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f697c0a5670 con 0x7f698c0c8570 2024-04-22T01:15:56.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6983fff640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f696c0678f0 0x7f696c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698b7fe640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f696c0678f0 0x7f696c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f697c11f370 con 0x7f698c0c8570 2024-04-22T01:15:56.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.808+0000 7f698b7fe640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f696c0678f0 0x7f696c069db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f698c06a110 tx=0x7f6974009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.812+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f698c0c7b90 con 0x7f698c0c8570 2024-04-22T01:15:56.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.960+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f698c0b6a60 con 0x7f698c0c8570 2024-04-22T01:15:56.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.960+0000 7f6983fff640 1 -- 172.21.15.70:0/1257449332 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f697c0167d0 con 0x7f698c0c8570 2024-04-22T01:15:56.964 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:15:56.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f696c0678f0 msgr2=0x7f696c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f696c0678f0 0x7f696c069db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f698c06a110 tx=0x7f6974009040 comp rx=0 tx=0).stop 2024-04-22T01:15:56.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 msgr2=0x7f698c152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 0x7f698c152b90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f697c01c8c0 tx=0x7f6978005840 comp rx=0 tx=0).stop 2024-04-22T01:15:56.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 shutdown_connections 2024-04-22T01:15:56.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f696c0678f0 0x7f696c069db0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f698c1530d0 0x7f698c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f698c0c8570 0x7f698c152b90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 --2- 172.21.15.70:0/1257449332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6978004810 0x7f698c0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 >> 172.21.15.70:0/1257449332 conn(0x7f698c099a60 msgr2=0x7f698c06f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 shutdown_connections 2024-04-22T01:15:56.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:56.964+0000 7f6991e59640 1 -- 172.21.15.70:0/1257449332 wait complete. 2024-04-22T01:15:56.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9edef9640 1 Processor -- start 2024-04-22T01:15:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9edef9640 1 -- start start 2024-04-22T01:15:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9edef9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9e806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9edef9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e806b860 con 0x7fa9e80cdd70 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9edef9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e806bb40 con 0x7fa9e80b21a0 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9edef9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e8161a60 con 0x7fa9e815df10 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa9e80cdd70 0x7fa9e80b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53366/0 (socket says 172.21.15.70:53366) 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 -- 172.21.15.70:0/1380263032 learned_addr learned my addr 172.21.15.70:0/1380263032 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956083212 0 0) 0x7fa9e806b860 con 0x7fa9e80cdd70 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9d0003530 con 0x7fa9e80cdd70 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e77fe640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9e806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2928500996 0 0) 0x7fa9d0003530 con 0x7fa9e80cdd70 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 msgr2=0x7fa9e806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9e806b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 >> v1:172.21.15.98:6789/0 conn(0x7fa9e80b21a0 legacy=0x7fa9e80cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e8161d40 con 0x7fa9e80cdd70 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fa9dc001680 con 0x7fa9e80cdd70 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 >> v1:172.21.15.70:6789/0 conn(0x7fa9e80cdd70 legacy=0x7fa9e80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d00039f0 0x7fa9d0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0004810 0x7fa9d0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8161a60 con 0x7fa9d0004810 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d0005530 con 0x7fa9d00039f0 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d000bf10 con 0x7fa9e815df10 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0004810 0x7fa9d0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e77fe640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d00039f0 0x7fa9d0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 -- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 msgr2=0x7fa9d000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e7fff640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9d000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 -- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d00039f0 msgr2=0x7fa9d0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d00039f0 0x7fa9d0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e6ffd640 1 -- 172.21.15.70:0/1380263032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d0003530 con 0x7fa9d0004810 2024-04-22T01:15:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e77fe640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d00039f0 0x7fa9d0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.100+0000 7fa9e7fff640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815df10 0x7fa9d000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e6ffd640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0004810 0x7fa9d0004c90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00c240 tx=0x7fa9dc01cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0b49a65ff537c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc007d20 con 0x7fa9d0004810 2024-04-22T01:15:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9dc0a9070 con 0x7fa9d0004810 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e67fc640 1 -- 172.21.15.70:0/1380263032 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc005d60 con 0x7fa9d0004810 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- 172.21.15.70:0/1380263032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0004810 msgr2=0x7fa9d0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1380263032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0004810 0x7fa9d0004c90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00c240 tx=0x7fa9dc01cc50 comp rx=0 tx=0).stop 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 -- 172.21.15.70:0/1380263032 reap_dead start 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- 172.21.15.70:0/1380263032 shutdown_connections 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- 172.21.15.70:0/1380263032 >> 172.21.15.70:0/1380263032 conn(0x7fa9e8099a60 msgr2=0x7fa9e8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- 172.21.15.70:0/1380263032 shutdown_connections 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- 172.21.15.70:0/1380263032 wait complete. 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 Processor -- start 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- start start 2024-04-22T01:15:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d0004810 0x7fa9e814b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815a810 0x7fa9e80c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 0x7fa9e8160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8169da0 con 0x7fa9e80ca850 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8162950 con 0x7fa9d0004810 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9edef9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e8169f20 con 0x7fa9e815a810 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 0x7fa9e8160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 0x7fa9e8160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34944/0 (socket says 172.21.15.70:34944) 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 -- 172.21.15.70:0/1544075541 learned_addr learned my addr 172.21.15.70:0/1544075541 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e6ffd640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815a810 0x7fa9e80c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e77fe640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d0004810 0x7fa9e814b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 -- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815a810 msgr2=0x7fa9e80c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815a810 0x7fa9e80c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 -- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d0004810 msgr2=0x7fa9e814b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d0004810 0x7fa9e814b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 -- 172.21.15.70:0/1544075541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e80ca090 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.104+0000 7fa9e7fff640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 0x7fa9e8160730 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80739c0 tx=0x7fa9e000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0006d60 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e0003980 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e815b390 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0003d70 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e815b0b0 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e8068920 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9e0005070 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9cffff640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9d40678f0 0x7fa9d4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9e77fe640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9d40678f0 0x7fa9d4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa9e0085ba0 con 0x7fa9e80ca850 2024-04-22T01:15:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.108+0000 7fa9e77fe640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9d40678f0 0x7fa9d4069db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c890 tx=0x7fa9d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.112+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa9e00562e0 con 0x7fa9e80ca850 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.260+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa9e806c0e0 con 0x7fa9e80ca850 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.260+0000 7fa9cffff640 1 -- 172.21.15.70:0/1544075541 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fa9e001ce40 con 0x7fa9e80ca850 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9d40678f0 msgr2=0x7fa9d4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9d40678f0 0x7fa9d4069db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c890 tx=0x7fa9d8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 msgr2=0x7fa9e8160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 0x7fa9e8160730 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80739c0 tx=0x7fa9e000dc00 comp rx=0 tx=0).stop 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 shutdown_connections 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa9d40678f0 0x7fa9d4069db0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9e80ca850 0x7fa9e8160730 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9e815a810 0x7fa9e80c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 --2- 172.21.15.70:0/1544075541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d0004810 0x7fa9e814b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 >> 172.21.15.70:0/1544075541 conn(0x7fa9e8099a60 msgr2=0x7fa9e80b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 shutdown_connections 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.264+0000 7fa9edef9640 1 -- 172.21.15.70:0/1544075541 wait complete. 2024-04-22T01:15:57.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.400+0000 7fc33f0de640 1 Processor -- start 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33f0de640 1 -- start start 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33f0de640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc33815de20 0x7fc3380b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33f0de640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc3380b6db0 0x7fc3380d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33f0de640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3380b75e0 con 0x7fc3380cea10 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33f0de640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc3380b78c0 con 0x7fc3380b6db0 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33f0de640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc3380b7ba0 con 0x7fc33815de20 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc3380b6db0 0x7fc3380d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc3380b6db0 0x7fc3380d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34946/0 (socket says 172.21.15.70:34946) 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 -- 172.21.15.70:0/372052792 learned_addr learned my addr 172.21.15.70:0/372052792 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc33ce7a640 1 --2- 172.21.15.70:0/372052792 >> v2:172.21.15.98:3300/0 conn(0x7fc33815de20 0x7fc3380b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 -- 172.21.15.70:0/372052792 >> v2:172.21.15.98:3300/0 conn(0x7fc33815de20 msgr2=0x7fc3380b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 --2- 172.21.15.70:0/372052792 >> v2:172.21.15.98:3300/0 conn(0x7fc33815de20 0x7fc3380b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 -- 172.21.15.70:0/372052792 >> v1:172.21.15.98:6789/0 conn(0x7fc3380cea10 legacy=0x7fc3380cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 -- 172.21.15.70:0/372052792 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3380b7e80 con 0x7fc3380b6db0 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.404+0000 7fc337fff640 1 --2- 172.21.15.70:0/372052792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380b6db0 0x7fc3380d1d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc3380d1ac0 tx=0x7fc32401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48cdc0a655412fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc3377fe640 1 -- 172.21.15.70:0/372052792 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3240ab640 con 0x7fc3380b6db0 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc3377fe640 1 -- 172.21.15.70:0/372052792 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3240b0070 con 0x7fc3380b6db0 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc3377fe640 1 -- 172.21.15.70:0/372052792 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3240aba00 con 0x7fc3380b6db0 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- 172.21.15.70:0/372052792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380b6db0 msgr2=0x7fc3380d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 --2- 172.21.15.70:0/372052792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380b6db0 0x7fc3380d1d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc3380d1ac0 tx=0x7fc32401c4b0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- 172.21.15.70:0/372052792 shutdown_connections 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 --2- 172.21.15.70:0/372052792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380b6db0 0x7fc3380d1d80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 --2- 172.21.15.70:0/372052792 >> v2:172.21.15.98:3300/0 conn(0x7fc33815de20 0x7fc3380b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- 172.21.15.70:0/372052792 >> 172.21.15.70:0/372052792 conn(0x7fc338072d20 msgr2=0x7fc338073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- 172.21.15.70:0/372052792 shutdown_connections 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- 172.21.15.70:0/372052792 wait complete. 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 Processor -- start 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- start start 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3380b6db0 0x7fc3381567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 0x7fc3380b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc33815de20 0x7fc3380b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380b7370 con 0x7fc3380cea10 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380b71f0 con 0x7fc3380b6db0 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.408+0000 7fc33f0de640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380b8e90 con 0x7fc33815de20 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 0x7fc3380b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 0x7fc3380b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34952/0 (socket says 172.21.15.70:34952) 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 -- 172.21.15.70:0/1076937778 learned_addr learned my addr 172.21.15.70:0/1076937778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33ce7a640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3380b6db0 0x7fc3381567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 -- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc33815de20 msgr2=0x7fc3380b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc33815de20 0x7fc3380b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 -- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3380b6db0 msgr2=0x7fc3381567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3380b6db0 0x7fc3381567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 -- 172.21.15.70:0/1076937778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3380cb740 con 0x7fc3380cea10 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33ce7a640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3380b6db0 0x7fc3381567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc337fff640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 0x7fc3380b3df0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc3240ad040 tx=0x7fc32401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3240be3f0 con 0x7fc3380cea10 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3240be590 con 0x7fc3380cea10 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3380cb9e0 con 0x7fc3380cea10 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3240be940 con 0x7fc3380cea10 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3380cbfa0 con 0x7fc3380cea10 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc338068210 con 0x7fc3380cea10 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc3240b43f0 con 0x7fc3380cea10 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc3357fa640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc31c0678c0 0x7fc31c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc324120660 con 0x7fc3380cea10 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33ce7a640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc31c0678c0 0x7fc31c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.412+0000 7fc33ce7a640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc31c0678c0 0x7fc31c069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc338147ff0 tx=0x7fc32c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.416+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc3240c4060 con 0x7fc3380cea10 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc3380d10c0 con 0x7fc3380cea10 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc3357fa640 1 -- 172.21.15.70:0/1076937778 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fc3240160d0 con 0x7fc3380cea10 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc31c0678c0 msgr2=0x7fc31c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc31c0678c0 0x7fc31c069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc338147ff0 tx=0x7fc32c008040 comp rx=0 tx=0).stop 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 msgr2=0x7fc3380b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 0x7fc3380b3df0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc3240ad040 tx=0x7fc32401acf0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 shutdown_connections 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc31c0678c0 0x7fc31c069d80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc33815de20 0x7fc3380b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3380cea10 0x7fc3380b3df0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 --2- 172.21.15.70:0/1076937778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3380b6db0 0x7fc3381567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 >> 172.21.15.70:0/1076937778 conn(0x7fc338072d20 msgr2=0x7fc3380b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 shutdown_connections 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.568+0000 7fc33f0de640 1 -- 172.21.15.70:0/1076937778 wait complete. 2024-04-22T01:15:57.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 Processor -- start 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 -- start start 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9dc0cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb9dc0d1400 0x7fb9dc0d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9dc15e520 con 0x7fb9dc0cf1c0 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb9dc0b9330 con 0x7fb9dc0d1400 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9e21fa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc0d2360 con 0x7fb9dc15e140 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9d3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb9dc0cf1c0 0x7fb9dc0d0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53372/0 (socket says 172.21.15.70:53372) 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9d3fff640 1 -- 172.21.15.70:0/1008599548 learned_addr learned my addr 172.21.15.70:0/1008599548 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9dbfff640 1 --2- 172.21.15.70:0/1008599548 >> v2:172.21.15.98:3300/0 conn(0x7fb9dc0d1400 0x7fb9dc0d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9dc0cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 510170577 0 0) 0x7fb9dc15e520 con 0x7fb9dc0cf1c0 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9c8003530 con 0x7fb9dc0cf1c0 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 318973276 0 0) 0x7fb9c8003530 con 0x7fb9dc0cf1c0 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 msgr2=0x7fb9dc0cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9dc0cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 >> v2:172.21.15.98:3300/0 conn(0x7fb9dc0d1400 msgr2=0x7fb9dc0d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 --2- 172.21.15.70:0/1008599548 >> v2:172.21.15.98:3300/0 conn(0x7fb9dc0d1400 0x7fb9dc0d27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc0b8d70 con 0x7fb9dc0cf1c0 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fb9c4001680 con 0x7fb9dc0cf1c0 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 >> v1:172.21.15.70:6789/0 conn(0x7fb9dc0cf1c0 legacy=0x7fb9dc0d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9c80039c0 0x7fb9c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c8004d50 0x7fb9c80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc0b9330 con 0x7fb9c80039c0 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc0d2360 con 0x7fb9c8004d50 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c800ba80 con 0x7fb9dc15e140 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9c80039c0 0x7fb9c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9d3fff640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9dbfff640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c8004d50 0x7fb9c80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 -- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 msgr2=0x7fb9c8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9c8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 -- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c8004d50 msgr2=0x7fb9c80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c8004d50 0x7fb9c80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9db7fe640 1 -- 172.21.15.70:0/1008599548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c8003530 con 0x7fb9c80039c0 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9d3fff640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15e140 0x7fb9c8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.696+0000 7fb9dbfff640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c8004d50 0x7fb9c80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9c80039c0 0x7fb9c8003de0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00ba30 tx=0x7fb9cc01c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195453e3f94e478a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc00b600 con 0x7fb9c80039c0 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9cc0b1070 con 0x7fb9c80039c0 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9daffd640 1 -- 172.21.15.70:0/1008599548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc0ac850 con 0x7fb9c80039c0 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- 172.21.15.70:0/1008599548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9c80039c0 msgr2=0x7fb9c8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/1008599548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9c80039c0 0x7fb9c8003de0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00ba30 tx=0x7fb9cc01c670 comp rx=0 tx=0).stop 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9dbfff640 1 -- 172.21.15.70:0/1008599548 reap_dead start 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- 172.21.15.70:0/1008599548 shutdown_connections 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- 172.21.15.70:0/1008599548 >> 172.21.15.70:0/1008599548 conn(0x7fb9dc072d20 msgr2=0x7fb9dc073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- 172.21.15.70:0/1008599548 shutdown_connections 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- 172.21.15.70:0/1008599548 wait complete. 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 Processor -- start 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- start start 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c80039c0 0x7fb9dc150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 0x7fb9dc1f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15fd10 0x7fb9dc1601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc0d36c0 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc0d3540 con 0x7fb9c80039c0 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9e21fa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc0d3840 con 0x7fb9dc15fd10 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 0x7fb9dc1f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 0x7fb9dc1f6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34980/0 (socket says 172.21.15.70:34980) 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.700+0000 7fb9d3fff640 1 -- 172.21.15.70:0/355384746 learned_addr learned my addr 172.21.15.70:0/355384746 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c80039c0 0x7fb9dc150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9dbfff640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15fd10 0x7fb9dc1601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d3fff640 1 -- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15fd10 msgr2=0x7fb9dc1601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d3fff640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15fd10 0x7fb9dc1601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d3fff640 1 -- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c80039c0 msgr2=0x7fb9dc150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d3fff640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c80039c0 0x7fb9dc150890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d3fff640 1 -- 172.21.15.70:0/355384746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc1fa460 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c80039c0 0x7fb9dc150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9dbfff640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15fd10 0x7fb9dc1601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d3fff640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 0x7fb9dc1f6260 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400d810 tx=0x7fb9c400dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4019070 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c40074e0 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c40049f0 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9dc1fa740 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9dc1fad00 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9dc068920 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9c401c070 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d8ff9640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9bc0678f0 0x7fb9bc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb9c4085ad0 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9bc0678f0 0x7fb9bc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.704+0000 7fb9db7fe640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9bc0678f0 0x7fb9bc069db0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc0b0000 tx=0x7fb9cc0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.708+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb9c40562a0 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.860+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb9dc162530 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.860+0000 7fb9d8ff9640 1 -- 172.21.15.70:0/355384746 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fb9c4055ba0 con 0x7fb9dc1f9f60 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9bc0678f0 msgr2=0x7fb9bc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9bc0678f0 0x7fb9bc069db0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc0b0000 tx=0x7fb9cc0046d0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 msgr2=0x7fb9dc1f6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 0x7fb9dc1f6260 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400d810 tx=0x7fb9c400dcd0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 shutdown_connections 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9bc0678f0 0x7fb9bc069db0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9dc15fd10 0x7fb9dc1601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc1f9f60 0x7fb9dc1f6260 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 --2- 172.21.15.70:0/355384746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c80039c0 0x7fb9dc150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 >> 172.21.15.70:0/355384746 conn(0x7fb9dc072d20 msgr2=0x7fb9dc161960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 shutdown_connections 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:57.864+0000 7fb9e21fa640 1 -- 172.21.15.70:0/355384746 wait complete. 2024-04-22T01:15:57.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 Processor -- start 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 -- start start 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f164415dfb0 0x7f16440b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f16440ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164415e390 con 0x7f16440b5730 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f16440cf420 con 0x7f164415dfb0 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1649787640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1644161e00 con 0x7f16440cea80 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f16427fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f16440ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f164415dfb0 0x7f16440b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f164415dfb0 0x7f16440b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:53074/0 (socket says 172.21.15.70:53074) 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 -- 172.21.15.70:0/3371945764 learned_addr learned my addr 172.21.15.70:0/3371945764 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f16427fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f16440ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:59384/0 (socket says 172.21.15.70:59384) 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f16437fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16440b5730 0x7f16440b5b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:42340/0 (socket says 172.21.15.70:42340) 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 -- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 msgr2=0x7f16440ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 --2- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f16440ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 -- 172.21.15.70:0/3371945764 >> v1:172.21.15.98:6789/0 conn(0x7f16440b5730 legacy=0x7f16440b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.000+0000 7f1642ffd640 1 -- 172.21.15.70:0/3371945764 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16441620e0 con 0x7f164415dfb0 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1642ffd640 1 --2- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164415dfb0 0x7f16440b95d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1634009870 tx=0x7f163401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f2cc8b60da7da4c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1641ffb640 1 -- 172.21.15.70:0/3371945764 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16340a7070 con 0x7f164415dfb0 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1641ffb640 1 -- 172.21.15.70:0/3371945764 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16340b0070 con 0x7f164415dfb0 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1641ffb640 1 -- 172.21.15.70:0/3371945764 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16340ab890 con 0x7f164415dfb0 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 -- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164415dfb0 msgr2=0x7f16440b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 --2- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164415dfb0 0x7f16440b95d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1634009870 tx=0x7f163401c560 comp rx=0 tx=0).stop 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 -- 172.21.15.70:0/3371945764 shutdown_connections 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 --2- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f16440ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 --2- 172.21.15.70:0/3371945764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164415dfb0 0x7f16440b95d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 -- 172.21.15.70:0/3371945764 >> 172.21.15.70:0/3371945764 conn(0x7f164406c9b0 msgr2=0x7f164406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 -- 172.21.15.70:0/3371945764 shutdown_connections 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.004+0000 7f1649787640 1 -- 172.21.15.70:0/3371945764 wait complete. 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 Processor -- start 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- start start 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16440b5730 0x7f16440cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f1644156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 0x7f164415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f164416b5e0 con 0x7f164415dfb0 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1644161a90 con 0x7f16440b5730 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f164416b760 con 0x7f16440cea80 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 0x7f164415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 0x7f164415c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34992/0 (socket says 172.21.15.70:34992) 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 -- 172.21.15.70:0/415200719 learned_addr learned my addr 172.21.15.70:0/415200719 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1642ffd640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16440b5730 0x7f16440cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16427fc640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f1644156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 -- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 msgr2=0x7f1644156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f1644156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 -- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16440b5730 msgr2=0x7f16440cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16440b5730 0x7f16440cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 -- 172.21.15.70:0/415200719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16440cc900 con 0x7f164415dfb0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f16437fe640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 0x7f164415c150 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c700 tx=0x7f163c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c017070 con 0x7f164415dfb0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163c00ce10 con 0x7f164415dfb0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16440ccbe0 con 0x7f164415dfb0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c00a840 con 0x7f164415dfb0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16440cd1a0 con 0x7f164415dfb0 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1644068210 con 0x7f164415dfb0 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f163c01a070 con 0x7f164415dfb0 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.008+0000 7f162b7fe640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16300678f0 0x7f1630069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.012+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f163c085530 con 0x7f164415dfb0 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.012+0000 7f1642ffd640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16300678f0 0x7f1630069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.012+0000 7f1642ffd640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16300678f0 0x7f1630069db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f16340af000 tx=0x7f16340a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.016+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f163c055620 con 0x7f164415dfb0 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.208+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16440ce290 con 0x7f164415dfb0 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.208+0000 7f162b7fe640 1 -- 172.21.15.70:0/415200719 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f163c059920 con 0x7f164415dfb0 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16300678f0 msgr2=0x7f1630069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16300678f0 0x7f1630069db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f16340af000 tx=0x7f16340a7000 comp rx=0 tx=0).stop 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 msgr2=0x7f164415c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 0x7f164415c150 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c700 tx=0x7f163c00cbc0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 shutdown_connections 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f16300678f0 0x7f1630069db0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f164415dfb0 0x7f164415c150 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16440cea80 0x7f1644156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 --2- 172.21.15.70:0/415200719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16440b5730 0x7f16440cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 >> 172.21.15.70:0/415200719 conn(0x7f164406c9b0 msgr2=0x7f16440b87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 shutdown_connections 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.212+0000 7f1649787640 1 -- 172.21.15.70:0/415200719 wait complete. 2024-04-22T01:15:58.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:58.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.344+0000 7f626d855640 1 Processor -- start 2024-04-22T01:15:58.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f626d855640 1 -- start start 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f626d855640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62680cc150 0x7f6268071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f626d855640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 0x7f626806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f626d855640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62680cd7b0 con 0x7f62681455b0 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f626d855640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f62680724d0 con 0x7f62680cc150 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f626d855640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f62680cc710 con 0x7f6268072060 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f6266ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f62681455b0 0x7f62680cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53382/0 (socket says 172.21.15.70:53382) 2024-04-22T01:15:58.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f6266ffd640 1 -- 172.21.15.70:0/541363807 learned_addr learned my addr 172.21.15.70:0/541363807 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666931755 0 0) 0x7f62680cd7b0 con 0x7f62681455b0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f625e7fc640 1 --2- 172.21.15.70:0/541363807 >> v2:172.21.15.98:3300/0 conn(0x7f62680cc150 0x7f6268071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6254003530 con 0x7f62681455b0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 0x7f626806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3777002439 0 0) 0x7f6254003530 con 0x7f62681455b0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 msgr2=0x7f626806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 0x7f626806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 >> v2:172.21.15.98:3300/0 conn(0x7f62680cc150 msgr2=0x7f6268071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 --2- 172.21.15.70:0/541363807 >> v2:172.21.15.98:3300/0 conn(0x7f62680cc150 0x7f6268071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62680cc9f0 con 0x7f62681455b0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6258001680 con 0x7f62681455b0 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 >> v1:172.21.15.70:6789/0 conn(0x7f62681455b0 legacy=0x7f62680cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62540039c0 0x7f6254003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 0x7f6254004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254004d50 0x7f62540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62680724d0 con 0x7f6254004d50 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62680cc710 con 0x7f62540039c0 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f625400baa0 con 0x7f6268072060 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254004d50 0x7f62540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f6266ffd640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62540039c0 0x7f6254003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f625e7fc640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 0x7f6254004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 -- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 msgr2=0x7f6254004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6268072060 0x7f6254004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 -- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62540039c0 msgr2=0x7f6254003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62540039c0 0x7f6254003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.348+0000 7f62677fe640 1 -- 172.21.15.70:0/541363807 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6254003530 con 0x7f6254004d50 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254004d50 0x7f62540051d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6260009fc0 tx=0x7f626001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157623c528747d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62600b2070 con 0x7f6254004d50 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62600a5a70 con 0x7f6254004d50 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62667fc640 1 -- 172.21.15.70:0/541363807 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260009c10 con 0x7f6254004d50 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- 172.21.15.70:0/541363807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254004d50 msgr2=0x7f62540051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 --2- 172.21.15.70:0/541363807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254004d50 0x7f62540051d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6260009fc0 tx=0x7f626001cc80 comp rx=0 tx=0).stop 2024-04-22T01:15:58.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 -- 172.21.15.70:0/541363807 reap_dead start 2024-04-22T01:15:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- 172.21.15.70:0/541363807 shutdown_connections 2024-04-22T01:15:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- 172.21.15.70:0/541363807 >> 172.21.15.70:0/541363807 conn(0x7f626806b1f0 msgr2=0x7f626806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- 172.21.15.70:0/541363807 shutdown_connections 2024-04-22T01:15:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- 172.21.15.70:0/541363807 wait complete. 2024-04-22T01:15:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 Processor -- start 2024-04-22T01:15:58.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- start start 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6254004d50 0x7f62681f90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62681f5050 0x7f626815a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 0x7f6268143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f626806eda0 con 0x7f6268142bb0 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f626806ec20 con 0x7f62681f5050 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f626d855640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f626806ef20 con 0x7f6254004d50 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 0x7f6268143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 0x7f6268143010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35008/0 (socket says 172.21.15.70:35008) 2024-04-22T01:15:58.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 -- 172.21.15.70:0/2281233186 learned_addr learned my addr 172.21.15.70:0/2281233186 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f625e7fc640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62681f5050 0x7f626815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f6266ffd640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6254004d50 0x7f62681f90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 -- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6254004d50 msgr2=0x7f62681f90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6254004d50 0x7f62681f90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 -- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62681f5050 msgr2=0x7f626815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62681f5050 0x7f626815a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 -- 172.21.15.70:0/2281233186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62681f95f0 con 0x7f6268142bb0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.352+0000 7f62677fe640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 0x7f6268143010 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6260002240 tx=0x7f6260009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62600bd070 con 0x7f6268142bb0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62600a5870 con 0x7f6268142bb0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626815a770 con 0x7f6268142bb0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62600b2040 con 0x7f6268142bb0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62681f5850 con 0x7f6268142bb0 2024-04-22T01:15:58.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6268068210 con 0x7f6268142bb0 2024-04-22T01:15:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f62600c0070 con 0x7f6268142bb0 2024-04-22T01:15:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6264ff9640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62480678c0 0x7f6248069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6266ffd640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62480678c0 0x7f6248069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f626011fd30 con 0x7f6268142bb0 2024-04-22T01:15:58.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.356+0000 7f6266ffd640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62480678c0 0x7f6248069d80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6258001130 tx=0x7f6258009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.360+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f626815a770 con 0x7f6268142bb0 2024-04-22T01:15:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.508+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6268142970 con 0x7f6268142bb0 2024-04-22T01:15:58.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.508+0000 7f6264ff9640 1 -- 172.21.15.70:0/2281233186 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f62681f95f0 con 0x7f6268142bb0 2024-04-22T01:15:58.512 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:58.512 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:58.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62480678c0 msgr2=0x7f6248069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62480678c0 0x7f6248069d80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6258001130 tx=0x7f6258009040 comp rx=0 tx=0).stop 2024-04-22T01:15:58.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 msgr2=0x7f6268143010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 0x7f6268143010 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6260002240 tx=0x7f6260009750 comp rx=0 tx=0).stop 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 shutdown_connections 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62480678c0 0x7f6248069d80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268142bb0 0x7f6268143010 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62681f5050 0x7f626815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 --2- 172.21.15.70:0/2281233186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6254004d50 0x7f62681f90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 >> 172.21.15.70:0/2281233186 conn(0x7f626806b1f0 msgr2=0x7f626815cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 shutdown_connections 2024-04-22T01:15:58.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.512+0000 7f626d855640 1 -- 172.21.15.70:0/2281233186 wait complete. 2024-04-22T01:15:58.530 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:15:58.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.640+0000 7f3190038640 1 Processor -- start 2024-04-22T01:15:58.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.640+0000 7f3190038640 1 -- start start 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f31880cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3188072230 0x7f318806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31880d0250 con 0x7f31880cc150 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f31880cd7b0 con 0x7f3188072230 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31880cc710 con 0x7f31881455b0 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3188072230 0x7f318806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318d5d3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f31880cc150 0x7f3188071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53398/0 (socket says 172.21.15.70:53398) 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3188072230 0x7f318806e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35020/0 (socket says 172.21.15.70:35020) 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318d5d3640 1 -- 172.21.15.70:0/2372247418 learned_addr learned my addr 172.21.15.70:0/2372247418 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:58.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318ddd4640 1 --2- 172.21.15.70:0/2372247418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f31880cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 -- 172.21.15.70:0/2372247418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 msgr2=0x7f31880cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 --2- 172.21.15.70:0/2372247418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f31880cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 -- 172.21.15.70:0/2372247418 >> v1:172.21.15.70:6789/0 conn(0x7f31880cc150 legacy=0x7f3188071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 -- 172.21.15.70:0/2372247418 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31880cc9f0 con 0x7f3188072230 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318cdd2640 1 -- 172.21.15.70:0/2372247418 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940834269 0 0) 0x7f31880d0250 con 0x7f31880cc150 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318e5d5640 1 --2- 172.21.15.70:0/2372247418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3188072230 0x7f318806e200 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3180009d90 tx=0x7f318001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e2921b2c4413b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318cdd2640 1 -- 172.21.15.70:0/2372247418 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31800b2070 con 0x7f3188072230 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318cdd2640 1 -- 172.21.15.70:0/2372247418 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31800a5710 con 0x7f3188072230 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f318cdd2640 1 -- 172.21.15.70:0/2372247418 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31800a5ae0 con 0x7f3188072230 2024-04-22T01:15:58.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- 172.21.15.70:0/2372247418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3188072230 msgr2=0x7f318806e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 --2- 172.21.15.70:0/2372247418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3188072230 0x7f318806e200 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3180009d90 tx=0x7f318001c850 comp rx=0 tx=0).stop 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- 172.21.15.70:0/2372247418 shutdown_connections 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 --2- 172.21.15.70:0/2372247418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3188072230 0x7f318806e200 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 --2- 172.21.15.70:0/2372247418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f31880cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- 172.21.15.70:0/2372247418 >> 172.21.15.70:0/2372247418 conn(0x7f318806b1f0 msgr2=0x7f318806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- 172.21.15.70:0/2372247418 shutdown_connections 2024-04-22T01:15:58.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.644+0000 7f3190038640 1 -- 172.21.15.70:0/2372247418 wait complete. 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 Processor -- start 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- start start 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3188072230 0x7f31880b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 0x7f318815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f3188159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f318806f2e0 con 0x7f31880cc150 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f318806f160 con 0x7f3188072230 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3188168b40 con 0x7f31881455b0 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 0x7f318815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 0x7f318815b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35028/0 (socket says 172.21.15.70:35028) 2024-04-22T01:15:58.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 -- 172.21.15.70:0/2186866495 learned_addr learned my addr 172.21.15.70:0/2186866495 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 -- 172.21.15.70:0/2186866495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 msgr2=0x7f3188159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f3188159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 -- 172.21.15.70:0/2186866495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3188072230 msgr2=0x7f31880b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3188072230 0x7f31880b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 -- 172.21.15.70:0/2186866495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31880b0c90 con 0x7f31880cc150 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f318d5d3640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 0x7f318815b770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f317800c480 tx=0x7f317800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3178018070 con 0x7f31880cc150 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3178007500 con 0x7f31880cc150 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3188140530 con 0x7f31880cc150 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3178004a30 con 0x7f31880cc150 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3188140af0 con 0x7f31880cc150 2024-04-22T01:15:58.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3188068210 con 0x7f31880cc150 2024-04-22T01:15:58.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.648+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f317801b070 con 0x7f31880cc150 2024-04-22T01:15:58.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.652+0000 7f317e7fc640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f316c0678c0 0x7f316c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.652+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f31780852e0 con 0x7f31880cc150 2024-04-22T01:15:58.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.652+0000 7f318ddd4640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f316c0678c0 0x7f316c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.652+0000 7f318ddd4640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f316c0678c0 0x7f316c069d80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31840098f0 tx=0x7f3184008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.656+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3178055ab0 con 0x7f31880cc150 2024-04-22T01:15:58.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.836+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f318815f510 con 0x7f31880cc150 2024-04-22T01:15:58.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.836+0000 7f317e7fc640 1 -- 172.21.15.70:0/2186866495 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f31780553b0 con 0x7f31880cc150 2024-04-22T01:15:58.841 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:58.841 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:58.841 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f316c0678c0 msgr2=0x7f316c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f316c0678c0 0x7f316c069d80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31840098f0 tx=0x7f3184008040 comp rx=0 tx=0).stop 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 msgr2=0x7f318815b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 0x7f318815b770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f317800c480 tx=0x7f317800c940 comp rx=0 tx=0).stop 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 shutdown_connections 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f316c0678c0 0x7f316c069d80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31881455b0 0x7f3188159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31880cc150 0x7f318815b770 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 --2- 172.21.15.70:0/2186866495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3188072230 0x7f31880b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 >> 172.21.15.70:0/2186866495 conn(0x7f318806b1f0 msgr2=0x7f31880cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 shutdown_connections 2024-04-22T01:15:58.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.840+0000 7f3190038640 1 -- 172.21.15.70:0/2186866495 wait complete. 2024-04-22T01:15:58.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:58.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 Processor -- start 2024-04-22T01:15:58.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 -- start start 2024-04-22T01:15:58.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09201455b0 0x7f09200cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f09200cc150 0x7f0920071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09200d0250 con 0x7f0920072060 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f09200724d0 con 0x7f09200cc150 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f0925eb7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f09200cd7b0 con 0x7f09201455b0 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f09200cc150 0x7f0920071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f09200cc150 0x7f0920071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35032/0 (socket says 172.21.15.70:35032) 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0920072060 0x7f092015f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53414/0 (socket says 172.21.15.70:53414) 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 -- 172.21.15.70:0/4047899652 learned_addr learned my addr 172.21.15.70:0/4047899652 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091f7fe640 1 --2- 172.21.15.70:0/4047899652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09201455b0 0x7f09200cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 -- 172.21.15.70:0/4047899652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09201455b0 msgr2=0x7f09200cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 --2- 172.21.15.70:0/4047899652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09201455b0 0x7f09200cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 -- 172.21.15.70:0/4047899652 >> v1:172.21.15.70:6789/0 conn(0x7f0920072060 legacy=0x7f092015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091effd640 1 -- 172.21.15.70:0/4047899652 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09200cc710 con 0x7f09200cc150 2024-04-22T01:15:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.976+0000 7f091e7fc640 1 -- 172.21.15.70:0/4047899652 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994530151 0 0) 0x7f09200d0250 con 0x7f0920072060 2024-04-22T01:15:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091effd640 1 --2- 172.21.15.70:0/4047899652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09200cc150 0x7f0920071b20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0914009870 tx=0x7f091401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f2c12cbd08a3c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091e7fc640 1 -- 172.21.15.70:0/4047899652 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09140b0070 con 0x7f09200cc150 2024-04-22T01:15:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091e7fc640 1 -- 172.21.15.70:0/4047899652 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09140ab600 con 0x7f09200cc150 2024-04-22T01:15:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091e7fc640 1 -- 172.21.15.70:0/4047899652 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09140ab9d0 con 0x7f09200cc150 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- 172.21.15.70:0/4047899652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09200cc150 msgr2=0x7f0920071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 --2- 172.21.15.70:0/4047899652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09200cc150 0x7f0920071b20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0914009870 tx=0x7f091401c440 comp rx=0 tx=0).stop 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- 172.21.15.70:0/4047899652 shutdown_connections 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 --2- 172.21.15.70:0/4047899652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09200cc150 0x7f0920071b20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 --2- 172.21.15.70:0/4047899652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09201455b0 0x7f09200cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- 172.21.15.70:0/4047899652 >> 172.21.15.70:0/4047899652 conn(0x7f092006b1f0 msgr2=0x7f092006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- 172.21.15.70:0/4047899652 shutdown_connections 2024-04-22T01:15:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- 172.21.15.70:0/4047899652 wait complete. 2024-04-22T01:15:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 Processor -- start 2024-04-22T01:15:58.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- start start 2024-04-22T01:15:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 0x7f0920159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200cc150 0x7f09200b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09201455b0 0x7f092013b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0920168b40 con 0x7f0920072060 2024-04-22T01:15:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f092006f210 con 0x7f09201455b0 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f0925eb7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0920168cc0 con 0x7f09200cc150 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 0x7f0920159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 0x7f0920159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35038/0 (socket says 172.21.15.70:35038) 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091f7fe640 1 -- 172.21.15.70:0/1493272113 learned_addr learned my addr 172.21.15.70:0/1493272113 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091ffff640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09201455b0 0x7f092013b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091effd640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200cc150 0x7f09200b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.980+0000 7f091f7fe640 1 -- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200cc150 msgr2=0x7f09200b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091f7fe640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200cc150 0x7f09200b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091f7fe640 1 -- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09201455b0 msgr2=0x7f092013b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091f7fe640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09201455b0 0x7f092013b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091f7fe640 1 -- 172.21.15.70:0/1493272113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092013b9a0 con 0x7f0920072060 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091effd640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200cc150 0x7f09200b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091f7fe640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 0x7f0920159220 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0910007f20 tx=0x7f091000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910014620 con 0x7f0920072060 2024-04-22T01:15:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0910019070 con 0x7f0920072060 2024-04-22T01:15:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092015eea0 con 0x7f0920072060 2024-04-22T01:15:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910014a00 con 0x7f0920072060 2024-04-22T01:15:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09200b02d0 con 0x7f0920072060 2024-04-22T01:15:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0920068210 con 0x7f0920072060 2024-04-22T01:15:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0910014ba0 con 0x7f0920072060 2024-04-22T01:15:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0907fff640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091effd640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f09100869e0 con 0x7f0920072060 2024-04-22T01:15:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.984+0000 7f091effd640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09140ad000 tx=0x7f0914008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:58.988+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0910057120 con 0x7f0920072060 2024-04-22T01:15:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.184+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f092015f510 con 0x7f0920072060 2024-04-22T01:15:59.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.184+0000 7f0907fff640 1 -- 172.21.15.70:0/1493272113 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f0910056a20 con 0x7f0920072060 2024-04-22T01:15:59.188 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:59.188 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2387834490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2387834490},{"type":"v1","addr":"172.21.15.98:6834","nonce":2387834490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3511434659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":3511434659},{"type":"v1","addr":"172.21.15.98:6835","nonce":3511434659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/3147948381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3147948381},{"type":"v1","addr":"172.21.15.70:6836","nonce":3147948381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:14:46.030124+0000","modified":"2024-04-22T01:15:09.178266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3139539459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3139539459},{"type":"v1","addr":"172.21.15.70:6837","nonce":3139539459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-22T01:15:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-22T01:15:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 msgr2=0x7f090c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09140ad000 tx=0x7f0914008040 comp rx=0 tx=0).stop 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 msgr2=0x7f0920159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 0x7f0920159220 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0910007f20 tx=0x7f091000e570 comp rx=0 tx=0).stop 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 shutdown_connections 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09140ad000 tx=0x7f0914008040 comp rx=0 tx=0).stop 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09201455b0 0x7f092013b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200cc150 0x7f09200b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 --2- 172.21.15.70:0/1493272113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0920072060 0x7f0920159220 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 >> 172.21.15.70:0/1493272113 conn(0x7f092006b1f0 msgr2=0x7f092013f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 shutdown_connections 2024-04-22T01:15:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.188+0000 7f0925eb7640 1 -- 172.21.15.70:0/1493272113 wait complete. 2024-04-22T01:15:59.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a289a3640 1 Processor -- start 2024-04-22T01:15:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a289a3640 1 -- start start 2024-04-22T01:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a289a3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 0x7f7a2415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a289a3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a240af6e0 con 0x7f7a2406c8e0 2024-04-22T01:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a289a3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a240af9c0 con 0x7f7a2406bd70 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a289a3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2415b5e0 con 0x7f7a2415acc0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7a2406c8e0 0x7f7a240ae080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53426/0 (socket says 172.21.15.70:53426) 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 -- 172.21.15.70:0/4042650558 learned_addr learned my addr 172.21.15.70:0/4042650558 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a2259c640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 0x7f7a2415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083772987 0 0) 0x7f7a240af6e0 con 0x7f7a2406c8e0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a10003530 con 0x7f7a2406c8e0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1338547152 0 0) 0x7f7a10003530 con 0x7f7a2406c8e0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 msgr2=0x7f7a2415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 0x7f7a2415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 >> v1:172.21.15.98:6789/0 conn(0x7f7a2406bd70 legacy=0x7f7a2406c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2406b290 con 0x7f7a2406c8e0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7a1c001a00 con 0x7f7a2406c8e0 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 >> v1:172.21.15.70:6789/0 conn(0x7f7a2406c8e0 legacy=0x7f7a240ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a100039f0 0x7f7a10003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 0x7f7a10004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a10004d50 0x7f7a100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2415b5e0 con 0x7f7a10004d50 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1000ba80 con 0x7f7a100039f0 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1000bc80 con 0x7f7a2415acc0 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a10004d50 0x7f7a100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a1bfff640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 0x7f7a10004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a2259c640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a100039f0 0x7f7a10003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 -- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 msgr2=0x7f7a10004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a2415acc0 0x7f7a10004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 -- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a100039f0 msgr2=0x7f7a10003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a100039f0 0x7f7a10003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.324+0000 7f7a22d9d640 1 -- 172.21.15.70:0/4042650558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a10003530 con 0x7f7a10004d50 2024-04-22T01:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a10004d50 0x7f7a100051d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c002240 tx=0x7f7a1c01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbdbd23f0a5eed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00a4a0 con 0x7f7a10004d50 2024-04-22T01:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1c0ad650 con 0x7f7a10004d50 2024-04-22T01:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a21d9b640 1 -- 172.21.15.70:0/4042650558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0ada40 con 0x7f7a10004d50 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- 172.21.15.70:0/4042650558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a10004d50 msgr2=0x7f7a100051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 --2- 172.21.15.70:0/4042650558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a10004d50 0x7f7a100051d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c002240 tx=0x7f7a1c01d340 comp rx=0 tx=0).stop 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a22d9d640 1 -- 172.21.15.70:0/4042650558 reap_dead start 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- 172.21.15.70:0/4042650558 shutdown_connections 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- 172.21.15.70:0/4042650558 >> 172.21.15.70:0/4042650558 conn(0x7f7a240b3d90 msgr2=0x7f7a240b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- 172.21.15.70:0/4042650558 shutdown_connections 2024-04-22T01:15:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- 172.21.15.70:0/4042650558 wait complete. 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 Processor -- start 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- start start 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a241e51b0 0x7f7a241e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a241e5590 0x7f7a241f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 0x7f7a241564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2406ff60 con 0x7f7a241f3d80 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2406b850 con 0x7f7a241e5590 2024-04-22T01:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a289a3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2415b0a0 con 0x7f7a241e51b0 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a22d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 0x7f7a241564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a22d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 0x7f7a241564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35058/0 (socket says 172.21.15.70:35058) 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.328+0000 7f7a22d9d640 1 -- 172.21.15.70:0/809815170 learned_addr learned my addr 172.21.15.70:0/809815170 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a2259c640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a241e51b0 0x7f7a241e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1bfff640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a241e5590 0x7f7a241f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a22d9d640 1 -- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a241e51b0 msgr2=0x7f7a241e3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a241e51b0 0x7f7a241e3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a22d9d640 1 -- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a241e5590 msgr2=0x7f7a241f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a241e5590 0x7f7a241f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a22d9d640 1 -- 172.21.15.70:0/809815170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a241e4410 con 0x7f7a241f3d80 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a22d9d640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 0x7f7a241564b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c01dbe0 tx=0x7f7a10005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00a480 con 0x7f7a241f3d80 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1c0ad430 con 0x7f7a241f3d80 2024-04-22T01:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a241f4420 con 0x7f7a241f3d80 2024-04-22T01:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0011d0 con 0x7f7a241f3d80 2024-04-22T01:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a241e33f0 con 0x7f7a241f3d80 2024-04-22T01:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a24068920 con 0x7f7a241f3d80 2024-04-22T01:15:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7a1c0bd030 con 0x7f7a241f3d80 2024-04-22T01:15:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1affd640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a040678c0 0x7f7a04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a2259c640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a040678c0 0x7f7a04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7a1c11ff40 con 0x7f7a241f3d80 2024-04-22T01:15:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.332+0000 7f7a2259c640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a040678c0 0x7f7a04069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a2415a6c0 tx=0x7f7a14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.336+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7a1c0c4020 con 0x7f7a241f3d80 2024-04-22T01:15:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.500+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a241e4d30 con 0x7f7a241f3d80 2024-04-22T01:15:59.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.500+0000 7f7a1affd640 1 -- 172.21.15.70:0/809815170 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18864 (secure 0 0 0) 0x7f7a1c017050 con 0x7f7a241f3d80 2024-04-22T01:15:59.505 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:15:59.505 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:15:27.903748+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:14:44.515300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:14:45.021580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:14:48.773096+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.27","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:59.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a040678c0 msgr2=0x7f7a04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a040678c0 0x7f7a04069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a2415a6c0 tx=0x7f7a14008040 comp rx=0 tx=0).stop 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 msgr2=0x7f7a241564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 0x7f7a241564b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c01dbe0 tx=0x7f7a10005840 comp rx=0 tx=0).stop 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 shutdown_connections 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a040678c0 0x7f7a04069d80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a241f3d80 0x7f7a241564b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a241e5590 0x7f7a241f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 --2- 172.21.15.70:0/809815170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a241e51b0 0x7f7a241e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 >> 172.21.15.70:0/809815170 conn(0x7f7a240b3d90 msgr2=0x7f7a2415c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 shutdown_connections 2024-04-22T01:15:59.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.504+0000 7f7a289a3640 1 -- 172.21.15.70:0/809815170 wait complete. 2024-04-22T01:15:59.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:15:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 Processor -- start 2024-04-22T01:15:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 -- start start 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e000b26a0 0x7f8e000ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e00162510 0x7f8e000b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8e00162080 con 0x7f8e000b26a0 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8e000cd4a0 con 0x7f8e00162510 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8e05c9c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0006b320 con 0x7f8e000ccb80 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e000b26a0 0x7f8e000ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e000b26a0 0x7f8e000ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35062/0 (socket says 172.21.15.70:35062) 2024-04-22T01:15:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 -- 172.21.15.70:0/879250413 learned_addr learned my addr 172.21.15.70:0/879250413 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8dfffff640 1 --2- 172.21.15.70:0/879250413 >> v2:172.21.15.98:3300/0 conn(0x7f8e00162510 0x7f8e000b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8dff7fe640 1 --2- 172.21.15.70:0/879250413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 -- 172.21.15.70:0/879250413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 msgr2=0x7f8e0006c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 --2- 172.21.15.70:0/879250413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0006c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 -- 172.21.15.70:0/879250413 >> v2:172.21.15.98:3300/0 conn(0x7f8e00162510 msgr2=0x7f8e000b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 --2- 172.21.15.70:0/879250413 >> v2:172.21.15.98:3300/0 conn(0x7f8e00162510 0x7f8e000b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.648+0000 7f8df7fff640 1 -- 172.21.15.70:0/879250413 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0006b600 con 0x7f8e000b26a0 2024-04-22T01:15:59.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8df7fff640 1 --2- 172.21.15.70:0/879250413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000ccf60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8e000b3130 tx=0x7f8de801c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e109014ca8483e94 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:59.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfeffd640 1 -- 172.21.15.70:0/879250413 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de80b0070 con 0x7f8e000b26a0 2024-04-22T01:15:59.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfeffd640 1 -- 172.21.15.70:0/879250413 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de80ab600 con 0x7f8e000b26a0 2024-04-22T01:15:59.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfeffd640 1 -- 172.21.15.70:0/879250413 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de80ab9d0 con 0x7f8e000b26a0 2024-04-22T01:15:59.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/879250413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 msgr2=0x7f8e000ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/879250413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000ccf60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8e000b3130 tx=0x7f8de801c1a0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/879250413 shutdown_connections 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/879250413 >> v2:172.21.15.98:3300/0 conn(0x7f8e00162510 0x7f8e000b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/879250413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000ccf60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/879250413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0006c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/879250413 >> 172.21.15.70:0/879250413 conn(0x7f8e000b3c00 msgr2=0x7f8e000b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/879250413 shutdown_connections 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/879250413 wait complete. 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 Processor -- start 2024-04-22T01:15:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- start start 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0015a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e00162510 0x7f8e000cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0015e450 con 0x7f8e000b26a0 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0015e2d0 con 0x7f8e00162510 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0015e5d0 con 0x7f8e000ccb80 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35072/0 (socket says 172.21.15.70:35072) 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 -- 172.21.15.70:0/3971504104 learned_addr learned my addr 172.21.15.70:0/3971504104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8df7fff640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0015a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfffff640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e00162510 0x7f8e000cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 -- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 msgr2=0x7f8e0015a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0015a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 -- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e00162510 msgr2=0x7f8e000cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e00162510 0x7f8e000cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 -- 172.21.15.70:0/3971504104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e000c7280 con 0x7f8e000b26a0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8df7fff640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0015a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfffff640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e00162510 0x7f8e000cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dff7fe640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000c8a70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8df000d830 tx=0x7f8df000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0018020 con 0x7f8e000b26a0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df00096a0 con 0x7f8e000b26a0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0015b3f0 con 0x7f8e000b26a0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0009a50 con 0x7f8e000b26a0 2024-04-22T01:15:59.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.652+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e0015b9b0 con 0x7f8e000b26a0 2024-04-22T01:15:59.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.656+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0004f510 con 0x7f8e000b26a0 2024-04-22T01:15:59.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.656+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8df001b020 con 0x7f8e000b26a0 2024-04-22T01:15:59.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.656+0000 7f8dfcff9640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de00678f0 0x7f8de0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:59.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.656+0000 7f8df7fff640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de00678f0 0x7f8de0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:59.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.656+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8df0086c50 con 0x7f8e000b26a0 2024-04-22T01:15:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.656+0000 7f8df7fff640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de00678f0 0x7f8de0069db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8de80ad000 tx=0x7f8de8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.660+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8df0022020 con 0x7f8e000b26a0 2024-04-22T01:15:59.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.856+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8e000cc100 con 0x7f8e000b26a0 2024-04-22T01:15:59.861 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:15:59.860+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:15:59.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.860+0000 7f8dfcff9640 1 -- 172.21.15.70:0/3971504104 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7f8df0057500 con 0x7f8e000b26a0 2024-04-22T01:15:59.863 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:15:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de00678f0 msgr2=0x7f8de0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de00678f0 0x7f8de0069db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8de80ad000 tx=0x7f8de8008040 comp rx=0 tx=0).stop 2024-04-22T01:15:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 msgr2=0x7f8e000c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000c8a70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8df000d830 tx=0x7f8df000dcf0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 shutdown_connections 2024-04-22T01:15:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de00678f0 0x7f8de0069db0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e00162510 0x7f8e000cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e000ccb80 0x7f8e0015a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 --2- 172.21.15.70:0/3971504104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e000b26a0 0x7f8e000c8a70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 >> 172.21.15.70:0/3971504104 conn(0x7f8e000b3c00 msgr2=0x7f8e000d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 shutdown_connections 2024-04-22T01:15:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:15:59.864+0000 7f8e05c9c640 1 -- 172.21.15.70:0/3971504104 wait complete. 2024-04-22T01:15:59.876 INFO:tasks.ceph.mds.c.smithi070.stderr: -4933> 2024-04-22T01:14:50.031+0000 7f84197ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:15:59.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:15:59.942 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:16:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 Processor -- start 2024-04-22T01:16:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 -- start start 2024-04-22T01:16:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f180815f480 0x7f18080b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f180806c310 0x7f18080ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18080678d0 con 0x7f180806b400 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f18080673a0 con 0x7f180815f480 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180fcd7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f18080675f0 con 0x7f180806c310 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180da73640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f180815f480 0x7f18080b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180d272640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f180806b400 0x7f180806bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53432/0 (socket says 172.21.15.70:53432) 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180d272640 1 -- 172.21.15.70:0/2316461310 learned_addr learned my addr 172.21.15.70:0/2316461310 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180e274640 1 --2- 172.21.15.70:0/2316461310 >> v2:172.21.15.98:3300/0 conn(0x7f180806c310 0x7f18080ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180da73640 1 -- 172.21.15.70:0/2316461310 >> v2:172.21.15.98:3300/0 conn(0x7f180806c310 msgr2=0x7f18080ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180da73640 1 --2- 172.21.15.70:0/2316461310 >> v2:172.21.15.98:3300/0 conn(0x7f180806c310 0x7f18080ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180da73640 1 -- 172.21.15.70:0/2316461310 >> v1:172.21.15.70:6789/0 conn(0x7f180806b400 legacy=0x7f180806bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180da73640 1 -- 172.21.15.70:0/2316461310 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18080b5100 con 0x7f180815f480 2024-04-22T01:16:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.004+0000 7f180e274640 1 --2- 172.21.15.70:0/2316461310 >> v2:172.21.15.98:3300/0 conn(0x7f180806c310 0x7f18080ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:00.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180da73640 1 --2- 172.21.15.70:0/2316461310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f18080b5be0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f180806ec50 tx=0x7f180401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3520ce7c2525dda3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:00.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180ca71640 1 -- 172.21.15.70:0/2316461310 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18040b2070 con 0x7f180815f480 2024-04-22T01:16:00.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180ca71640 1 -- 172.21.15.70:0/2316461310 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18040a57f0 con 0x7f180815f480 2024-04-22T01:16:00.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180ca71640 1 -- 172.21.15.70:0/2316461310 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18040a5bc0 con 0x7f180815f480 2024-04-22T01:16:00.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/2316461310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 msgr2=0x7f18080b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 --2- 172.21.15.70:0/2316461310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f18080b5be0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f180806ec50 tx=0x7f180401c930 comp rx=0 tx=0).stop 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/2316461310 shutdown_connections 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 --2- 172.21.15.70:0/2316461310 >> v2:172.21.15.98:3300/0 conn(0x7f180806c310 0x7f18080ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 --2- 172.21.15.70:0/2316461310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f18080b5be0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/2316461310 >> 172.21.15.70:0/2316461310 conn(0x7f1808099a60 msgr2=0x7f18080b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/2316461310 shutdown_connections 2024-04-22T01:16:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/2316461310 wait complete. 2024-04-22T01:16:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 Processor -- start 2024-04-22T01:16:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- start start 2024-04-22T01:16:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f180806b400 0x7f1808147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f180806c310 0x7f1808147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f180815e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18080cb380 con 0x7f180815f480 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18080cb200 con 0x7f180806b400 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f18080b2c00 con 0x7f180806c310 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f180815e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f180815e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35086/0 (socket says 172.21.15.70:35086) 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 -- 172.21.15.70:0/441611737 learned_addr learned my addr 172.21.15.70:0/441611737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180d272640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f180806c310 0x7f1808147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 -- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f180806c310 msgr2=0x7f1808147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f180806c310 0x7f1808147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 -- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f180806b400 msgr2=0x7f1808147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180da73640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f180806b400 0x7f1808147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f180806b400 0x7f1808147f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 -- 172.21.15.70:0/441611737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1808148510 con 0x7f180815f480 2024-04-22T01:16:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180e274640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f180815e630 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f180000c820 tx=0x7f180000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800015020 con 0x7f180815f480 2024-04-22T01:16:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1800004240 con 0x7f180815f480 2024-04-22T01:16:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18081487f0 con 0x7f180815f480 2024-04-22T01:16:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800003620 con 0x7f180815f480 2024-04-22T01:16:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.008+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1808148db0 con 0x7f180815f480 2024-04-22T01:16:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.012+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1808068210 con 0x7f180815f480 2024-04-22T01:16:00.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.012+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1800018020 con 0x7f180815f480 2024-04-22T01:16:00.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.012+0000 7f17fe7fc640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17ec0678c0 0x7f17ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:00.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.012+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f1800086110 con 0x7f180815f480 2024-04-22T01:16:00.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.012+0000 7f180da73640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17ec0678c0 0x7f17ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:00.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.012+0000 7f180da73640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17ec0678c0 0x7f17ec069d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f18040af000 tx=0x7f18040a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:00.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.016+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f180001f020 con 0x7f180815f480 2024-04-22T01:16:00.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.200+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1808072760 con 0x7f180815f480 2024-04-22T01:16:00.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.864+0000 7f17fe7fc640 1 -- 172.21.15.70:0/441611737 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f1800056880 con 0x7f180815f480 2024-04-22T01:16:00.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.864+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17ec0678c0 msgr2=0x7f17ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.864+0000 7f180fcd7640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17ec0678c0 0x7f17ec069d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f18040af000 tx=0x7f18040a7070 comp rx=0 tx=0).stop 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.864+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 msgr2=0x7f180815e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.864+0000 7f180fcd7640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f180815e630 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f180000c820 tx=0x7f180000cce0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 shutdown_connections 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17ec0678c0 0x7f17ec069d80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f180815f480 0x7f180815e630 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f180806c310 0x7f1808147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 --2- 172.21.15.70:0/441611737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f180806b400 0x7f1808147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 >> 172.21.15.70:0/441611737 conn(0x7f1808099a60 msgr2=0x7f1808157200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 shutdown_connections 2024-04-22T01:16:00.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:00.868+0000 7f180fcd7640 1 -- 172.21.15.70:0/441611737 wait complete. 2024-04-22T01:16:00.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:16:01.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 Processor -- start 2024-04-22T01:16:01.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 -- start start 2024-04-22T01:16:01.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc66406b900 0x7fc664161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:01.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cd090 0x7fc6640cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:01.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc664162040 con 0x7fc6640cca20 2024-04-22T01:16:01.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc6640b02b0 con 0x7fc66406b900 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc669fad640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc664162320 con 0x7fc6640cd090 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc662ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc66406b900 0x7fc664161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc662ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc66406b900 0x7fc664161a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35088/0 (socket says 172.21.15.70:35088) 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.012+0000 7fc662ffd640 1 -- 172.21.15.70:0/2189711847 learned_addr learned my addr 172.21.15.70:0/2189711847 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc663fff640 1 --2- 172.21.15.70:0/2189711847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cd090 0x7fc6640cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc662ffd640 1 -- 172.21.15.70:0/2189711847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cd090 msgr2=0x7fc6640cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc662ffd640 1 --2- 172.21.15.70:0/2189711847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cd090 0x7fc6640cfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc662ffd640 1 -- 172.21.15.70:0/2189711847 >> v1:172.21.15.98:6789/0 conn(0x7fc6640cca20 legacy=0x7fc66406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc662ffd640 1 -- 172.21.15.70:0/2189711847 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc664162600 con 0x7fc66406b900 2024-04-22T01:16:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc662ffd640 1 --2- 172.21.15.70:0/2189711847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc66406b900 0x7fc664161a60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc658009870 tx=0x7fc65801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5737b3fab7679e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc6627fc640 1 -- 172.21.15.70:0/2189711847 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580ab5f0 con 0x7fc66406b900 2024-04-22T01:16:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc6627fc640 1 -- 172.21.15.70:0/2189711847 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6580b0070 con 0x7fc66406b900 2024-04-22T01:16:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc6627fc640 1 -- 172.21.15.70:0/2189711847 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580ab9b0 con 0x7fc66406b900 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 -- 172.21.15.70:0/2189711847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc66406b900 msgr2=0x7fc664161a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 --2- 172.21.15.70:0/2189711847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc66406b900 0x7fc664161a60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc658009870 tx=0x7fc65801c520 comp rx=0 tx=0).stop 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 -- 172.21.15.70:0/2189711847 shutdown_connections 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 --2- 172.21.15.70:0/2189711847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cd090 0x7fc6640cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 --2- 172.21.15.70:0/2189711847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc66406b900 0x7fc664161a60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 -- 172.21.15.70:0/2189711847 >> 172.21.15.70:0/2189711847 conn(0x7fc664099a60 msgr2=0x7fc664099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 -- 172.21.15.70:0/2189711847 shutdown_connections 2024-04-22T01:16:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.016+0000 7fc669fad640 1 -- 172.21.15.70:0/2189711847 wait complete. 2024-04-22T01:16:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 Processor -- start 2024-04-22T01:16:01.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- start start 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc66406b900 0x7fc6641528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cca20 0x7fc6640c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 0x7fc6640c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc66415f100 con 0x7fc6640cd090 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc66415ef80 con 0x7fc66406b900 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc66415f280 con 0x7fc6640cca20 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 0x7fc6640c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 0x7fc6640c6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35090/0 (socket says 172.21.15.70:35090) 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 -- 172.21.15.70:0/2861815039 learned_addr learned my addr 172.21.15.70:0/2861815039 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc662ffd640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cca20 0x7fc6640c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc6637fe640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc66406b900 0x7fc6641528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:01.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 -- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cca20 msgr2=0x7fc6640c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cca20 0x7fc6640c67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 -- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc66406b900 msgr2=0x7fc6641528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc66406b900 0x7fc6641528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 -- 172.21.15.70:0/2861815039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc664158720 con 0x7fc6640cd090 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc663fff640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 0x7fc6640c6d20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00cda0 tx=0x7fc65c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c003ba0 con 0x7fc6640cd090 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65c003d40 con 0x7fc6640cd090 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc664158a00 con 0x7fc6640cd090 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c004240 con 0x7fc6640cd090 2024-04-22T01:16:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc664158fc0 con 0x7fc6640cd090 2024-04-22T01:16:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc664068920 con 0x7fc6640cd090 2024-04-22T01:16:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.020+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc65c0043e0 con 0x7fc6640cd090 2024-04-22T01:16:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.024+0000 7fc64bfff640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6500678c0 0x7fc650069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.024+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fc65c086bd0 con 0x7fc6640cd090 2024-04-22T01:16:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.024+0000 7fc6637fe640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6500678c0 0x7fc650069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:01.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.024+0000 7fc6637fe640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6500678c0 0x7fc650069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc654007fd0 tx=0x7fc654009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.028+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc65c057340 con 0x7fc6640cd090 2024-04-22T01:16:01.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:01.188+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc6640cd6a0 con 0x7fc6640cd090 2024-04-22T01:16:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.084+0000 7fc64bfff640 1 -- 172.21.15.70:0/2861815039 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) ==== 181+0+0 (secure 0 0 0) 0x7fc65c056c40 con 0x7fc6640cd090 2024-04-22T01:16:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:16:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6500678c0 msgr2=0x7fc650069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:02.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6500678c0 0x7fc650069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc654007fd0 tx=0x7fc654009040 comp rx=0 tx=0).stop 2024-04-22T01:16:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 msgr2=0x7fc6640c6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 0x7fc6640c6d20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00cda0 tx=0x7fc65c00a660 comp rx=0 tx=0).stop 2024-04-22T01:16:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 shutdown_connections 2024-04-22T01:16:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6500678c0 0x7fc650069d80 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc654007fd0 tx=0x7fc654009040 comp rx=0 tx=0).stop 2024-04-22T01:16:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6640cd090 0x7fc6640c6d20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6640cca20 0x7fc6640c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 --2- 172.21.15.70:0/2861815039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc66406b900 0x7fc6641528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 >> 172.21.15.70:0/2861815039 conn(0x7fc664099a60 msgr2=0x7fc6640ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:02.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 shutdown_connections 2024-04-22T01:16:02.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.088+0000 7fc669fad640 1 -- 172.21.15.70:0/2861815039 wait complete. 2024-04-22T01:16:02.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:16:02.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 Processor -- start 2024-04-22T01:16:02.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 -- start start 2024-04-22T01:16:02.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c5415a320 0x7f3c5406f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 0x7f3c54072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5406da70 con 0x7f3c54071b70 2024-04-22T01:16:02.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5406dd50 con 0x7f3c5415a320 2024-04-22T01:16:02.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5c3c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3c5406e030 con 0x7f3c540ce420 2024-04-22T01:16:02.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59961640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3c54071b70 0x7f3c540b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53440/0 (socket says 172.21.15.70:53440) 2024-04-22T01:16:02.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59961640 1 -- 172.21.15.70:0/2141596323 learned_addr learned my addr 172.21.15.70:0/2141596323 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:02.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a162640 1 --2- 172.21.15.70:0/2141596323 >> v2:172.21.15.98:3300/0 conn(0x7f3c5415a320 0x7f3c5406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 0x7f3c54072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353429037 0 0) 0x7f3c5406da70 con 0x7f3c54071b70 2024-04-22T01:16:02.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c40003530 con 0x7f3c54071b70 2024-04-22T01:16:02.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1091684241 0 0) 0x7f3c40003530 con 0x7f3c54071b70 2024-04-22T01:16:02.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 msgr2=0x7f3c54072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:02.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 0x7f3c54072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 >> v2:172.21.15.98:3300/0 conn(0x7f3c5415a320 msgr2=0x7f3c5406f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 --2- 172.21.15.70:0/2141596323 >> v2:172.21.15.98:3300/0 conn(0x7f3c5415a320 0x7f3c5406f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5406e310 con 0x7f3c54071b70 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3c44001680 con 0x7f3c54071b70 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 >> v1:172.21.15.70:6789/0 conn(0x7f3c54071b70 legacy=0x7f3c540b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 0x7f3c400039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40003f90 0x7f3c40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c40004d50 0x7f3c400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5406dd50 con 0x7f3c40004d50 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5406e030 con 0x7f3c40003f90 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4000b9d0 con 0x7f3c540ce420 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c40004d50 0x7f3c400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a162640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 0x7f3c400039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c59961640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40003f90 0x7f3c40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 -- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 msgr2=0x7f3c400039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:02.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c540ce420 0x7f3c400039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 -- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40003f90 msgr2=0x7f3c40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:02.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40003f90 0x7f3c40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.236+0000 7f3c5a963640 1 -- 172.21.15.70:0/2141596323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c40003530 con 0x7f3c40004d50 2024-04-22T01:16:02.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5a963640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c40004d50 0x7f3c400051d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00e170 tx=0x7f3c4c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c711d7a0ac82d294 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:02.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c0b2070 con 0x7f3c40004d50 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c0a5c70 con 0x7f3c40004d50 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c59160640 1 -- 172.21.15.70:0/2141596323 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c009c10 con 0x7f3c40004d50 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/2141596323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c40004d50 msgr2=0x7f3c400051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/2141596323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c40004d50 0x7f3c400051d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00e170 tx=0x7f3c4c01cc60 comp rx=0 tx=0).stop 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5a963640 1 -- 172.21.15.70:0/2141596323 reap_dead start 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/2141596323 shutdown_connections 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/2141596323 >> 172.21.15.70:0/2141596323 conn(0x7f3c540b1d40 msgr2=0x7f3c540cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/2141596323 shutdown_connections 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/2141596323 wait complete. 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 Processor -- start 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.240+0000 7f3c5c3c6640 1 -- start start 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40004d50 0x7f3c54144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c54154090 0x7f3c541511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 0x7f3c54151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540cf420 con 0x7f3c541527d0 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5415a700 con 0x7f3c40004d50 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3c540cf5a0 con 0x7f3c54154090 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 0x7f3c54151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 0x7f3c54151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35110/0 (socket says 172.21.15.70:35110) 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 -- 172.21.15.70:0/302586469 learned_addr learned my addr 172.21.15.70:0/302586469 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a162640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40004d50 0x7f3c54144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c59961640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c54154090 0x7f3c541511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 -- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c54154090 msgr2=0x7f3c541511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c54154090 0x7f3c541511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 -- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40004d50 msgr2=0x7f3c54144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40004d50 0x7f3c54144430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:02.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 -- 172.21.15.70:0/302586469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c54152ff0 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c59961640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c54154090 0x7f3c541511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a963640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 0x7f3c54151730 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00e170 tx=0x7f3c4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c0b2040 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c0a5a70 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c0aead0 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c541532d0 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c54153890 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c54068210 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3c4c0be020 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c4affd640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3c380678c0 0x7f3c38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6383+0+0 (secure 0 0 0) 0x7f3c4c1200e0 con 0x7f3c541527d0 2024-04-22T01:16:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.244+0000 7f3c5a162640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3c380678c0 0x7f3c38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:02.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.248+0000 7f3c5a162640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3c380678c0 0x7f3c38069d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c5000ab30 tx=0x7f3c50008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:02.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.252+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3c4c0c5020 con 0x7f3c541527d0 2024-04-22T01:16:02.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:02.396+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3c540b4050 con 0x7f3c541527d0 2024-04-22T01:16:03.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.108+0000 7f3c4affd640 1 -- 172.21.15.70:0/302586469 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) ==== 169+0+0 (secure 0 0 0) 0x7f3c4c0a5e20 con 0x7f3c541527d0 2024-04-22T01:16:03.110 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:16:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3c380678c0 msgr2=0x7f3c38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3c380678c0 0x7f3c38069d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c5000ab30 tx=0x7f3c50008040 comp rx=0 tx=0).stop 2024-04-22T01:16:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 msgr2=0x7f3c54151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 0x7f3c54151730 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00e170 tx=0x7f3c4c009750 comp rx=0 tx=0).stop 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 shutdown_connections 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3c380678c0 0x7f3c38069d80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c541527d0 0x7f3c54151730 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c54154090 0x7f3c541511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 --2- 172.21.15.70:0/302586469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c40004d50 0x7f3c54144430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 >> 172.21.15.70:0/302586469 conn(0x7f3c540b1d40 msgr2=0x7f3c5406f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.112+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 shutdown_connections 2024-04-22T01:16:03.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.116+0000 7f3c5c3c6640 1 -- 172.21.15.70:0/302586469 wait complete. 2024-04-22T01:16:03.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-22T01:16:03.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.260+0000 7fd1e772f640 1 Processor -- start 2024-04-22T01:16:03.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e772f640 1 -- start start 2024-04-22T01:16:03.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e772f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e00b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:03.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e772f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1e00cea10 0x7fd1e00ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:03.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e772f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00cf3e0 con 0x7fd1e00b6db0 2024-04-22T01:16:03.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e772f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e00b75e0 con 0x7fd1e00cea10 2024-04-22T01:16:03.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e772f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00b78c0 con 0x7fd1e015de20 2024-04-22T01:16:03.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1e00cea10 0x7fd1e00ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:03.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1e00cea10 0x7fd1e00ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35124/0 (socket says 172.21.15.70:35124) 2024-04-22T01:16:03.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 -- 172.21.15.70:0/2354293225 learned_addr learned my addr 172.21.15.70:0/2354293225 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:03.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e54cb640 1 --2- 172.21.15.70:0/2354293225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:03.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 -- 172.21.15.70:0/2354293225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 msgr2=0x7fd1e00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:03.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 --2- 172.21.15.70:0/2354293225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 -- 172.21.15.70:0/2354293225 >> v1:172.21.15.98:6789/0 conn(0x7fd1e00b6db0 legacy=0x7fd1e00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:03.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.264+0000 7fd1e5ccc640 1 -- 172.21.15.70:0/2354293225 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00b7ba0 con 0x7fd1e00cea10 2024-04-22T01:16:03.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e5ccc640 1 --2- 172.21.15.70:0/2354293225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cea10 0x7fd1e00ceea0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009e80 tx=0x7fd1d801f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49146e781470a83f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:03.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1d7fff640 1 -- 172.21.15.70:0/2354293225 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80b4070 con 0x7fd1e00cea10 2024-04-22T01:16:03.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1d7fff640 1 -- 172.21.15.70:0/2354293225 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d80af400 con 0x7fd1e00cea10 2024-04-22T01:16:03.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1d7fff640 1 -- 172.21.15.70:0/2354293225 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80af7d0 con 0x7fd1e00cea10 2024-04-22T01:16:03.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 -- 172.21.15.70:0/2354293225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cea10 msgr2=0x7fd1e00ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:03.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 --2- 172.21.15.70:0/2354293225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cea10 0x7fd1e00ceea0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009e80 tx=0x7fd1d801f200 comp rx=0 tx=0).stop 2024-04-22T01:16:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 -- 172.21.15.70:0/2354293225 shutdown_connections 2024-04-22T01:16:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 --2- 172.21.15.70:0/2354293225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cea10 0x7fd1e00ceea0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 --2- 172.21.15.70:0/2354293225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e00b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 -- 172.21.15.70:0/2354293225 >> 172.21.15.70:0/2354293225 conn(0x7fd1e0072d20 msgr2=0x7fd1e0073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 -- 172.21.15.70:0/2354293225 shutdown_connections 2024-04-22T01:16:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.268+0000 7fd1e772f640 1 -- 172.21.15.70:0/2354293225 wait complete. 2024-04-22T01:16:03.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 Processor -- start 2024-04-22T01:16:03.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 -- start start 2024-04-22T01:16:03.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 0x7fd1e01584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:03.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e00cea10 0x7fd1e0158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:03.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e015e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:03.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 0x7fd1e01584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:03.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e015e200 con 0x7fd1e00b6db0 2024-04-22T01:16:03.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b8160 con 0x7fd1e00cea10 2024-04-22T01:16:03.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00b71f0 con 0x7fd1e015de20 2024-04-22T01:16:03.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 0x7fd1e01584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35134/0 (socket says 172.21.15.70:35134) 2024-04-22T01:16:03.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 -- 172.21.15.70:0/1086734777 learned_addr learned my addr 172.21.15.70:0/1086734777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:03.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e5ccc640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:03.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e4cca640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e00cea10 0x7fd1e0158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:03.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 -- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 msgr2=0x7fd1e015e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:03.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e015e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 -- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e00cea10 msgr2=0x7fd1e0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:03.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e00cea10 0x7fd1e0158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:03.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 -- 172.21.15.70:0/1086734777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00c8f70 con 0x7fd1e00b6db0 2024-04-22T01:16:03.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e54cb640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 0x7fd1e01584a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d5f0 tx=0x7fd1dc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:03.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0125f0 con 0x7fd1e00b6db0 2024-04-22T01:16:03.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1dc019070 con 0x7fd1e00b6db0 2024-04-22T01:16:03.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e00c91f0 con 0x7fd1e00b6db0 2024-04-22T01:16:03.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0129b0 con 0x7fd1e00b6db0 2024-04-22T01:16:03.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.272+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e00c97b0 con 0x7fd1e00b6db0 2024-04-22T01:16:03.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.276+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e0068210 con 0x7fd1e00b6db0 2024-04-22T01:16:03.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.276+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd1dc012b50 con 0x7fd1e00b6db0 2024-04-22T01:16:03.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.276+0000 7fd1d5ffb640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1c40678f0 0x7fd1c4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:03.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.276+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5853+0+0 (secure 0 0 0) 0x7fd1dc0860f0 con 0x7fd1e00b6db0 2024-04-22T01:16:03.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.276+0000 7fd1e4cca640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1c40678f0 0x7fd1c4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:03.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.276+0000 7fd1e4cca640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1c40678f0 0x7fd1c4069db0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00cf080 tx=0x7fd1d0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:03.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.280+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd1e00c8f70 con 0x7fd1e00b6db0 2024-04-22T01:16:03.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:03.440+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd1e00b6960 con 0x7fd1e00b6db0 2024-04-22T01:16:04.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.108+0000 7fd1d5ffb640 1 -- 172.21.15.70:0/1086734777 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) ==== 178+0+0 (secure 0 0 0) 0x7fd1dc056ba0 con 0x7fd1e00b6db0 2024-04-22T01:16:04.113 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data_ec' removed 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.112+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1c40678f0 msgr2=0x7fd1c4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.112+0000 7fd1e772f640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1c40678f0 0x7fd1c4069db0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00cf080 tx=0x7fd1d0009040 comp rx=0 tx=0).stop 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.112+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 msgr2=0x7fd1e01584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.112+0000 7fd1e772f640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 0x7fd1e01584a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d5f0 tx=0x7fd1dc00dab0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 shutdown_connections 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1c40678f0 0x7fd1c4069db0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e015de20 0x7fd1e015e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e00cea10 0x7fd1e0158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 --2- 172.21.15.70:0/1086734777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00b6db0 0x7fd1e01584a0 secure :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d5f0 tx=0x7fd1dc00dab0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 >> 172.21.15.70:0/1086734777 conn(0x7fd1e0072d20 msgr2=0x7fd1e00b84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 shutdown_connections 2024-04-22T01:16:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.116+0000 7fd1e772f640 1 -- 172.21.15.70:0/1086734777 wait complete. 2024-04-22T01:16:04.140 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:16:04.140 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-22T01:16:04.140 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:16:04.140 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:04.140 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:16:04.141 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-22T01:16:04.141 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:16:04.141 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:04.141 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:16:04.141 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-22T01:16:04.141 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:16:04.141 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:04.142 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:16:04.142 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-22T01:16:04.142 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:16:04.142 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:04.151 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:16:04.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:16:04.152 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:16:04.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:16:04.157 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:16:04.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:16:04.161 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:16:04.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:16:04.195 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:16:04.195 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:04.196 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:16:04.196 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:04.206 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:16:04.207 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:04.208 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:16:04.208 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:04.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-22T01:16:04.258 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:16:04.266 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:16:04.276 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:16:04.282 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:16:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 Processor -- start 2024-04-22T01:16:04.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 -- start start 2024-04-22T01:16:04.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c161b90 0x7f343c162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:04.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f343c0cecd0 0x7f343c0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:04.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f343c06b450 con 0x7f343c1615a0 2024-04-22T01:16:04.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f343c162550 con 0x7f343c0cecd0 2024-04-22T01:16:04.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f3441a2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0cf6a0 con 0x7f343c161b90 2024-04-22T01:16:04.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f343c0cecd0 0x7f343c0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f343c0cecd0 0x7f343c0cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35182/0 (socket says 172.21.15.70:35182) 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 -- 172.21.15.70:0/3475896481 learned_addr learned my addr 172.21.15.70:0/3475896481 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343a7fc640 1 --2- 172.21.15.70:0/3475896481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c161b90 0x7f343c162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 -- 172.21.15.70:0/3475896481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c161b90 msgr2=0x7f343c162010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 --2- 172.21.15.70:0/3475896481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c161b90 0x7f343c162010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 -- 172.21.15.70:0/3475896481 >> v1:172.21.15.98:6789/0 conn(0x7f343c1615a0 legacy=0x7f343c0b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:04.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 -- 172.21.15.70:0/3475896481 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c0b77d0 con 0x7f343c0cecd0 2024-04-22T01:16:04.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.328+0000 7f343b7fe640 1 --2- 172.21.15.70:0/3475896481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c0cecd0 0x7f343c0cf160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e80 tx=0x7f343401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9dab7d1254d297c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:04.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3439ffb640 1 -- 172.21.15.70:0/3475896481 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340a56e0 con 0x7f343c0cecd0 2024-04-22T01:16:04.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3439ffb640 1 -- 172.21.15.70:0/3475896481 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34340a5880 con 0x7f343c0cecd0 2024-04-22T01:16:04.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3439ffb640 1 -- 172.21.15.70:0/3475896481 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340a5c50 con 0x7f343c0cecd0 2024-04-22T01:16:04.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/3475896481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c0cecd0 msgr2=0x7f343c0cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:04.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 --2- 172.21.15.70:0/3475896481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c0cecd0 0x7f343c0cf160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e80 tx=0x7f343401ca10 comp rx=0 tx=0).stop 2024-04-22T01:16:04.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/3475896481 shutdown_connections 2024-04-22T01:16:04.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 --2- 172.21.15.70:0/3475896481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c0cecd0 0x7f343c0cf160 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 --2- 172.21.15.70:0/3475896481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c161b90 0x7f343c162010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/3475896481 >> 172.21.15.70:0/3475896481 conn(0x7f343c06c910 msgr2=0x7f343c06ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:04.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/3475896481 shutdown_connections 2024-04-22T01:16:04.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/3475896481 wait complete. 2024-04-22T01:16:04.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 Processor -- start 2024-04-22T01:16:04.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- start start 2024-04-22T01:16:04.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c0cecd0 0x7f343c159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:04.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f343c1615a0 0x7f343c15a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:04.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 0x7f343c0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:04.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f343c0d2130 con 0x7f343c161b90 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f343c0ce830 con 0x7f343c1615a0 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0d3120 con 0x7f343c0cecd0 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 0x7f343c0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c0cecd0 0x7f343c159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 0x7f343c0c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35188/0 (socket says 172.21.15.70:35188) 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 -- 172.21.15.70:0/2953484362 learned_addr learned my addr 172.21.15.70:0/2953484362 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343a7fc640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f343c1615a0 0x7f343c15a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 -- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c0cecd0 msgr2=0x7f343c159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c0cecd0 0x7f343c159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 -- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f343c1615a0 msgr2=0x7f343c15a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f343c1615a0 0x7f343c15a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 -- 172.21.15.70:0/2953484362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c0b4760 con 0x7f343c161b90 2024-04-22T01:16:04.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f343b7fe640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 0x7f343c0c8fc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f343401a640 tx=0x7f34340197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:04.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434009510 con 0x7f343c161b90 2024-04-22T01:16:04.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34340be070 con 0x7f343c161b90 2024-04-22T01:16:04.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c0b4a40 con 0x7f343c161b90 2024-04-22T01:16:04.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340b88b0 con 0x7f343c161b90 2024-04-22T01:16:04.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.332+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c0b5000 con 0x7f343c161b90 2024-04-22T01:16:04.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.336+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f343c04f510 con 0x7f343c161b90 2024-04-22T01:16:04.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.336+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f34340b8a50 con 0x7f343c161b90 2024-04-22T01:16:04.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.336+0000 7f3423fff640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34280678c0 0x7f3428069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:04.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.336+0000 7f343affd640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34280678c0 0x7f3428069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:04.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.336+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5335+0+0 (secure 0 0 0) 0x7f3434120a70 con 0x7f343c161b90 2024-04-22T01:16:04.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.336+0000 7f343affd640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34280678c0 0x7f3428069d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f342c001810 tx=0x7f342c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:04.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.340+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f343c0b77d0 con 0x7f343c161b90 2024-04-22T01:16:04.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:04.532+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f343c0d1b90 con 0x7f343c161b90 2024-04-22T01:16:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.140+0000 7f3423fff640 1 -- 172.21.15.70:0/2953484362 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) ==== 97+0+0 (secure 0 0 0) 0x7f3434016600 con 0x7f343c161b90 2024-04-22T01:16:05.142 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34280678c0 msgr2=0x7f3428069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34280678c0 0x7f3428069d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f342c001810 tx=0x7f342c009040 comp rx=0 tx=0).stop 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 msgr2=0x7f343c0c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 0x7f343c0c8fc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f343401a640 tx=0x7f34340197b0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 shutdown_connections 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34280678c0 0x7f3428069d80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f343c161b90 0x7f343c0c8fc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f343c1615a0 0x7f343c15a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 --2- 172.21.15.70:0/2953484362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f343c0cecd0 0x7f343c159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 >> 172.21.15.70:0/2953484362 conn(0x7f343c06c910 msgr2=0x7f343c0cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 shutdown_connections 2024-04-22T01:16:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.144+0000 7f3441a2b640 1 -- 172.21.15.70:0/2953484362 wait complete. 2024-04-22T01:16:05.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-22T01:16:05.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.284+0000 7ff49c2fb640 1 Processor -- start 2024-04-22T01:16:05.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.284+0000 7ff49c2fb640 1 -- start start 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49c2fb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff49415dfb0 0x7ff4940b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49c2fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4940cea80 0x7ff4940ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49c2fb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff4940b5730 0x7ff4940b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49c2fb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff4940cf420 con 0x7ff4940b5730 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49c2fb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff49415e390 con 0x7ff49415dfb0 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49c2fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff494161e00 con 0x7ff4940cea80 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff4940b5730 0x7ff4940b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff4940b5730 0x7ff4940b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35198/0 (socket says 172.21.15.70:35198) 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 -- 172.21.15.70:0/3843842700 learned_addr learned my addr 172.21.15.70:0/3843842700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff499896640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4940cea80 0x7ff4940ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a097640 1 --2- 172.21.15.70:0/3843842700 >> v2:172.21.15.98:3300/0 conn(0x7ff49415dfb0 0x7ff4940b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 -- 172.21.15.70:0/3843842700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4940cea80 msgr2=0x7ff4940ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4940cea80 0x7ff4940ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 -- 172.21.15.70:0/3843842700 >> v2:172.21.15.98:3300/0 conn(0x7ff49415dfb0 msgr2=0x7ff4940b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 --2- 172.21.15.70:0/3843842700 >> v2:172.21.15.98:3300/0 conn(0x7ff49415dfb0 0x7ff4940b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 -- 172.21.15.70:0/3843842700 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4941620e0 con 0x7ff4940b5730 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff499896640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4940cea80 0x7ff4940ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:05.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a097640 1 --2- 172.21.15.70:0/3843842700 >> v2:172.21.15.98:3300/0 conn(0x7ff49415dfb0 0x7ff4940b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:05.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.288+0000 7ff49a898640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff4940b5b90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff4940d4d80 tx=0x7ff48c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e09b93fa8c058c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:05.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff499095640 1 -- 172.21.15.70:0/3843842700 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c0a5780 con 0x7ff4940b5730 2024-04-22T01:16:05.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff499095640 1 -- 172.21.15.70:0/3843842700 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48c0a5920 con 0x7ff4940b5730 2024-04-22T01:16:05.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff499095640 1 -- 172.21.15.70:0/3843842700 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c0a5cf0 con 0x7ff4940b5730 2024-04-22T01:16:05.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/3843842700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 msgr2=0x7ff4940b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff4940b5b90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff4940d4d80 tx=0x7ff48c01c9f0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/3843842700 shutdown_connections 2024-04-22T01:16:05.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff4940b5b90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/3843842700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4940cea80 0x7ff4940ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/3843842700 >> v2:172.21.15.98:3300/0 conn(0x7ff49415dfb0 0x7ff4940b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/3843842700 >> 172.21.15.70:0/3843842700 conn(0x7ff49406c9b0 msgr2=0x7ff49406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:05.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/3843842700 shutdown_connections 2024-04-22T01:16:05.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/3843842700 wait complete. 2024-04-22T01:16:05.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 Processor -- start 2024-04-22T01:16:05.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- start start 2024-04-22T01:16:05.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff494152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4940cea80 0x7ff49415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff49415dfb0 0x7ff4940cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff494161a90 con 0x7ff4940b5730 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4940b9b10 con 0x7ff4940cea80 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff4940d2900 con 0x7ff49415dfb0 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff494152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff494152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35200/0 (socket says 172.21.15.70:35200) 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 -- 172.21.15.70:0/1001686485 learned_addr learned my addr 172.21.15.70:0/1001686485 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff499896640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4940cea80 0x7ff49415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a898640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff49415dfb0 0x7ff4940cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 -- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff49415dfb0 msgr2=0x7ff4940cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff49415dfb0 0x7ff4940cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 -- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4940cea80 msgr2=0x7ff49415bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4940cea80 0x7ff49415bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 -- 172.21.15.70:0/1001686485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff494154350 con 0x7ff4940b5730 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff499896640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4940cea80 0x7ff49415bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49a097640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff494152700 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff48400c7f0 tx=0x7ff48400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484015070 con 0x7ff4940b5730 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff484007820 con 0x7ff4940b5730 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff494154a20 con 0x7ff4940b5730 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484007c10 con 0x7ff4940b5730 2024-04-22T01:16:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.292+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff494154070 con 0x7ff4940b5730 2024-04-22T01:16:05.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.296+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff494068210 con 0x7ff4940b5730 2024-04-22T01:16:05.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.296+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff484018070 con 0x7ff4940b5730 2024-04-22T01:16:05.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.296+0000 7ff48affd640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff4780678f0 0x7ff478069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.296+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7ff484085b30 con 0x7ff4940b5730 2024-04-22T01:16:05.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.296+0000 7ff499896640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff4780678f0 0x7ff478069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.296+0000 7ff499896640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff4780678f0 0x7ff478069db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff490006fd0 tx=0x7ff490008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:05.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.300+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff4840568c0 con 0x7ff4940b5730 2024-04-22T01:16:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.508+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff4940d1950 con 0x7ff4940b5730 2024-04-22T01:16:05.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff48affd640 1 -- 172.21.15.70:0/1001686485 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) ==== 80+0+6645 (secure 0 0 0) 0x7ff4840561c0 con 0x7ff4940b5730 2024-04-22T01:16:05.514 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:05.514 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw==", 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.515 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w==", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ==", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA==", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA==", 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA==", 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg==", 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.519 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg==", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong==", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ==", 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA==", 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ==", 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw==", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ==", 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-22T01:16:05.524 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA==", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw==", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:05.525 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm2O5GBRAA/JxaogwD4Hjmtr9eQvMY0g==", 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-mds", 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDuuSVmom8OAhAAyQqDLWZIALMiU/yVfTXcpg==", 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-mds" 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-mgr", 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDuuSVmi4kOAhAA53VYzs3Ej60OSee1swNX5A==", 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-osd", 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDuuSVmjKoOAhAAc0Ni46SRgOerMCpsMOq3uw==", 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-osd" 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-rbd", 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDuuSVmhsAOAhAAyZXPRWboHJ2im7MWlQ/AUw==", 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDuuSVmXtYOAhAAbaARsKebf/dbQ2BKlcDG8A==", 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-rgw", 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDuuSVmeu0OAhAAopakf/BnoL1+bOzTpoM1RA==", 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw==", 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg==", 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:05.530 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:16:05.531 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:16:05.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:05.531 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.531 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:05.531 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-22T01:16:05.531 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:16:05.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff4780678f0 msgr2=0x7ff478069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff4780678f0 0x7ff478069db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff490006fd0 tx=0x7ff490008040 comp rx=0 tx=0).stop 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 msgr2=0x7ff494152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff494152700 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff48400c7f0 tx=0x7ff48400ccb0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 shutdown_connections 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff4780678f0 0x7ff478069db0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff49415dfb0 0x7ff4940cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4940cea80 0x7ff49415bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 --2- 172.21.15.70:0/1001686485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4940b5730 0x7ff494152700 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 >> 172.21.15.70:0/1001686485 conn(0x7ff49406c9b0 msgr2=0x7ff4940b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 shutdown_connections 2024-04-22T01:16:05.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.512+0000 7ff49c2fb640 1 -- 172.21.15.70:0/1001686485 wait complete. 2024-04-22T01:16:05.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-22T01:16:05.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 Processor -- start 2024-04-22T01:16:05.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 -- start start 2024-04-22T01:16:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53840d2130 0x7f5384161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 0x7f5384161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53841624e0 con 0x7f538415dd90 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5384162770 con 0x7f53840d2130 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538bed6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f53840ce380 con 0x7f53840cf140 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5389c72640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f538415dd90 0x7f53840d1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:53458/0 (socket says 172.21.15.70:53458) 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5389c72640 1 -- 172.21.15.70:0/1695756386 learned_addr learned my addr 172.21.15.70:0/1695756386 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5389471640 1 --2- 172.21.15.70:0/1695756386 >> v2:172.21.15.98:3300/0 conn(0x7f53840d2130 0x7f5384161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f538a473640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 0x7f5384161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560520932 0 0) 0x7f53841624e0 con 0x7f538415dd90 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5370003530 con 0x7f538415dd90 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2389927858 0 0) 0x7f5370003530 con 0x7f538415dd90 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 msgr2=0x7f5384161fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 0x7f5384161fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 >> v2:172.21.15.98:3300/0 conn(0x7f53840d2130 msgr2=0x7f5384161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 --2- 172.21.15.70:0/1695756386 >> v2:172.21.15.98:3300/0 conn(0x7f53840d2130 0x7f5384161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53840ce660 con 0x7f538415dd90 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.652+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f5380001680 con 0x7f538415dd90 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 >> v1:172.21.15.70:6789/0 conn(0x7f538415dd90 legacy=0x7f53840d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 0x7f5370003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5370003fb0 0x7f5370004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370004d50 0x7f53700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5384162770 con 0x7f5370003fb0 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53840ce380 con 0x7f5370004d50 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f537000ba00 con 0x7f53840cf140 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5370003fb0 0x7f5370004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389c72640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 0x7f5370003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 -- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 msgr2=0x7f5370003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53840cf140 0x7f5370003a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 -- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370004d50 msgr2=0x7f53700051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370004d50 0x7f53700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 -- 172.21.15.70:0/1695756386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5370003530 con 0x7f5370003fb0 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5389471640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5370003fb0 0x7f5370004810 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5374009ae0 tx=0x7f537401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67bfca72220a8d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740b2070 con 0x7f5370003fb0 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53740a56a0 con 0x7f5370003fb0 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f5388c70640 1 -- 172.21.15.70:0/1695756386 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740096b0 con 0x7f5370003fb0 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538bed6640 1 -- 172.21.15.70:0/1695756386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5370003fb0 msgr2=0x7f5370004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538bed6640 1 --2- 172.21.15.70:0/1695756386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5370003fb0 0x7f5370004810 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5374009ae0 tx=0x7f537401cab0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538a473640 1 -- 172.21.15.70:0/1695756386 reap_dead start 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538bed6640 1 -- 172.21.15.70:0/1695756386 shutdown_connections 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538bed6640 1 -- 172.21.15.70:0/1695756386 >> 172.21.15.70:0/1695756386 conn(0x7f538408b1b0 msgr2=0x7f538408b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538bed6640 1 -- 172.21.15.70:0/1695756386 shutdown_connections 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.656+0000 7f538bed6640 1 -- 172.21.15.70:0/1695756386 wait complete. 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 Processor -- start 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- start start 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370003fb0 0x7f5384150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 0x7f538414cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538414d420 0x7f538414d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f538415e170 con 0x7f538414a590 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53840cec20 con 0x7f538414d420 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f538415e2f0 con 0x7f5370003fb0 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 0x7f538414cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 0x7f538414cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35218/0 (socket says 172.21.15.70:35218) 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 -- 172.21.15.70:0/2207032509 learned_addr learned my addr 172.21.15.70:0/2207032509 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 -- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370003fb0 msgr2=0x7f5384150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370003fb0 0x7f5384150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 -- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538414d420 msgr2=0x7f538414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538a473640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538414d420 0x7f538414d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538414d420 0x7f538414d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 -- 172.21.15.70:0/2207032509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53840b6ad0 con 0x7f538414a590 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f5389471640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 0x7f538414cee0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5374009ae0 tx=0x7f5370005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740bc030 con 0x7f538414a590 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53740a5dd0 con 0x7f538414a590 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53840b6db0 con 0x7f538414a590 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740b2040 con 0x7f538414a590 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53840b7370 con 0x7f538414a590 2024-04-22T01:16:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f538404f510 con 0x7f538414a590 2024-04-22T01:16:05.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.660+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f53740bf070 con 0x7f538414a590 2024-04-22T01:16:05.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.664+0000 7f537a7fc640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53680678c0 0x7f5368069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:05.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.664+0000 7f5389c72640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53680678c0 0x7f5368069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:05.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.664+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f537411f580 con 0x7f538414a590 2024-04-22T01:16:05.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.664+0000 7f5389c72640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53680678c0 0x7f5368069d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5380001210 tx=0x7f5380000ee0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:05.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.668+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f53740167c0 con 0x7f538414a590 2024-04-22T01:16:05.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.884+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f53840b1740 con 0x7f538414a590 2024-04-22T01:16:05.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.888+0000 7f537a7fc640 1 -- 172.21.15.70:0/2207032509 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f53740efcc0 con 0x7f538414a590 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53680678c0 msgr2=0x7f5368069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53680678c0 0x7f5368069d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5380001210 tx=0x7f5380000ee0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 msgr2=0x7f538414cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 0x7f538414cee0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5374009ae0 tx=0x7f5370005840 comp rx=0 tx=0).stop 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 shutdown_connections 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53680678c0 0x7f5368069d80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538414d420 0x7f538414d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538414a590 0x7f538414cee0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 --2- 172.21.15.70:0/2207032509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370003fb0 0x7f5384150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 >> 172.21.15.70:0/2207032509 conn(0x7f538408b1b0 msgr2=0x7f53840cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:05.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 shutdown_connections 2024-04-22T01:16:05.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:05.892+0000 7f538bed6640 1 -- 172.21.15.70:0/2207032509 wait complete. 2024-04-22T01:16:05.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-22T01:16:06.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 Processor -- start 2024-04-22T01:16:06.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 -- start start 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb7c1445b0 0x7fdb7c072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdb7c0b4360 0x7fdb7c0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0cdce0 0x7fdb7c0ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7c0b3bd0 con 0x7fdb7c0b4360 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7c144990 con 0x7fdb7c1445b0 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb8435c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c0b2b50 con 0x7fdb7c0cdce0 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdb7c0b4360 0x7fdb7c0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdb7c0b4360 0x7fdb7c0b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35224/0 (socket says 172.21.15.70:35224) 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 -- 172.21.15.70:0/2346804281 learned_addr learned my addr 172.21.15.70:0/2346804281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb820f8640 1 --2- 172.21.15.70:0/2346804281 >> v2:172.21.15.98:3300/0 conn(0x7fdb7c1445b0 0x7fdb7c072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb828f9640 1 --2- 172.21.15.70:0/2346804281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0cdce0 0x7fdb7c0ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 -- 172.21.15.70:0/2346804281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0cdce0 msgr2=0x7fdb7c0ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 --2- 172.21.15.70:0/2346804281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0cdce0 0x7fdb7c0ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 -- 172.21.15.70:0/2346804281 >> v2:172.21.15.98:3300/0 conn(0x7fdb7c1445b0 msgr2=0x7fdb7c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:06.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 --2- 172.21.15.70:0/2346804281 >> v2:172.21.15.98:3300/0 conn(0x7fdb7c1445b0 0x7fdb7c072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.024+0000 7fdb818f7640 1 -- 172.21.15.70:0/2346804281 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c0b2e10 con 0x7fdb7c0b4360 2024-04-22T01:16:06.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb818f7640 1 --2- 172.21.15.70:0/2346804281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c0b4360 0x7fdb7c0b47f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c073480 tx=0x7fdb7801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4555c76e6fda7f29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb810f6640 1 -- 172.21.15.70:0/2346804281 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb780b1070 con 0x7fdb7c0b4360 2024-04-22T01:16:06.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb810f6640 1 -- 172.21.15.70:0/2346804281 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb780ac630 con 0x7fdb7c0b4360 2024-04-22T01:16:06.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb810f6640 1 -- 172.21.15.70:0/2346804281 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb780a8050 con 0x7fdb7c0b4360 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- 172.21.15.70:0/2346804281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c0b4360 msgr2=0x7fdb7c0b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- 172.21.15.70:0/2346804281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c0b4360 0x7fdb7c0b47f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c073480 tx=0x7fdb7801c520 comp rx=0 tx=0).stop 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- 172.21.15.70:0/2346804281 shutdown_connections 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- 172.21.15.70:0/2346804281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0cdce0 0x7fdb7c0ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- 172.21.15.70:0/2346804281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c0b4360 0x7fdb7c0b47f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- 172.21.15.70:0/2346804281 >> v2:172.21.15.98:3300/0 conn(0x7fdb7c1445b0 0x7fdb7c072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- 172.21.15.70:0/2346804281 >> 172.21.15.70:0/2346804281 conn(0x7fdb7c0cf9e0 msgr2=0x7fdb7c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:06.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- 172.21.15.70:0/2346804281 shutdown_connections 2024-04-22T01:16:06.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- 172.21.15.70:0/2346804281 wait complete. 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 Processor -- start 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- start start 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0b4360 0x7fdb7c136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c0cdce0 0x7fdb7c155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 0x7fdb7c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c06edb0 con 0x7fdb7c1445b0 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c072a40 con 0x7fdb7c0cdce0 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb8435c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c06ef30 con 0x7fdb7c0b4360 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 0x7fdb7c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 0x7fdb7c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35234/0 (socket says 172.21.15.70:35234) 2024-04-22T01:16:06.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 -- 172.21.15.70:0/3354433349 learned_addr learned my addr 172.21.15.70:0/3354433349 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 -- 172.21.15.70:0/3354433349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0b4360 msgr2=0x7fdb7c136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0b4360 0x7fdb7c136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 -- 172.21.15.70:0/3354433349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c0cdce0 msgr2=0x7fdb7c155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c0cdce0 0x7fdb7c155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.028+0000 7fdb828f9640 1 -- 172.21.15.70:0/3354433349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c163550 con 0x7fdb7c1445b0 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb828f9640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 0x7fdb7c13ed90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009dd0 tx=0x7fdb7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74018020 con 0x7fdb7c1445b0 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb74014480 con 0x7fdb7c1445b0 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c163830 con 0x7fdb7c1445b0 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74014870 con 0x7fdb7c1445b0 2024-04-22T01:16:06.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c163df0 con 0x7fdb7c1445b0 2024-04-22T01:16:06.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c04f510 con 0x7fdb7c1445b0 2024-04-22T01:16:06.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb7401b020 con 0x7fdb7c1445b0 2024-04-22T01:16:06.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb6effd640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb600678c0 0x7fdb60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fdb74086610 con 0x7fdb7c1445b0 2024-04-22T01:16:06.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb820f8640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb600678c0 0x7fdb60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.032+0000 7fdb820f8640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb600678c0 0x7fdb60069d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdb700099c0 tx=0x7fdb70009480 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.036+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdb74022070 con 0x7fdb7c1445b0 2024-04-22T01:16:06.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.244+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fdb7c15aa90 con 0x7fdb7c1445b0 2024-04-22T01:16:06.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.244+0000 7fdb6effd640 1 -- 172.21.15.70:0/3354433349 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7fdb74010040 con 0x7fdb7c1445b0 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb600678c0 msgr2=0x7fdb60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb600678c0 0x7fdb60069d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdb700099c0 tx=0x7fdb70009480 comp rx=0 tx=0).stop 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 msgr2=0x7fdb7c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 0x7fdb7c13ed90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009dd0 tx=0x7fdb7400c490 comp rx=0 tx=0).stop 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 shutdown_connections 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb600678c0 0x7fdb60069d80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c1445b0 0x7fdb7c13ed90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c0cdce0 0x7fdb7c155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 --2- 172.21.15.70:0/3354433349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c0b4360 0x7fdb7c136bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 >> 172.21.15.70:0/3354433349 conn(0x7fdb7c0cf9e0 msgr2=0x7fdb7c1422e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:06.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 shutdown_connections 2024-04-22T01:16:06.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.248+0000 7fdb8435c640 1 -- 172.21.15.70:0/3354433349 wait complete. 2024-04-22T01:16:06.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-22T01:16:06.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 Processor -- start 2024-04-22T01:16:06.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 -- start start 2024-04-22T01:16:06.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd84c0cecf0 0x7fd84c0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd84c0b48a0 0x7fd84c1646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd84c144990 con 0x7fd84c1445b0 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd84c0b3bd0 con 0x7fd84c0b48a0 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd852d83640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd84c0b4cd0 con 0x7fd84c0cecf0 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd84c0b48a0 0x7fd84c1646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd84c0b48a0 0x7fd84c1646a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38076/0 (socket says 172.21.15.70:38076) 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 -- 172.21.15.70:0/3366132467 learned_addr learned my addr 172.21.15.70:0/3366132467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd84bfff640 1 --2- 172.21.15.70:0/3366132467 >> v2:172.21.15.98:3300/0 conn(0x7fd84c0cecf0 0x7fd84c0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 -- 172.21.15.70:0/3366132467 >> v2:172.21.15.98:3300/0 conn(0x7fd84c0cecf0 msgr2=0x7fd84c0b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 --2- 172.21.15.70:0/3366132467 >> v2:172.21.15.98:3300/0 conn(0x7fd84c0cecf0 0x7fd84c0b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 -- 172.21.15.70:0/3366132467 >> v1:172.21.15.98:6789/0 conn(0x7fd84c1445b0 legacy=0x7fd84c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd851320640 1 -- 172.21.15.70:0/3366132467 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84c0b2b50 con 0x7fd84c0b48a0 2024-04-22T01:16:06.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.372+0000 7fd84bfff640 1 --2- 172.21.15.70:0/3366132467 >> v2:172.21.15.98:3300/0 conn(0x7fd84c0cecf0 0x7fd84c0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd851320640 1 --2- 172.21.15.70:0/3366132467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0b48a0 0x7fd84c1646a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd84c071a80 tx=0x7fd83c01c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bb13904d17f8d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd84b7fe640 1 -- 172.21.15.70:0/3366132467 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0a5580 con 0x7fd84c0b48a0 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd84b7fe640 1 -- 172.21.15.70:0/3366132467 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd83c0a5720 con 0x7fd84c0b48a0 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd84b7fe640 1 -- 172.21.15.70:0/3366132467 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0a5af0 con 0x7fd84c0b48a0 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 -- 172.21.15.70:0/3366132467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0b48a0 msgr2=0x7fd84c1646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 --2- 172.21.15.70:0/3366132467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0b48a0 0x7fd84c1646a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd84c071a80 tx=0x7fd83c01c7d0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 -- 172.21.15.70:0/3366132467 shutdown_connections 2024-04-22T01:16:06.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 --2- 172.21.15.70:0/3366132467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0b48a0 0x7fd84c1646a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 --2- 172.21.15.70:0/3366132467 >> v2:172.21.15.98:3300/0 conn(0x7fd84c0cecf0 0x7fd84c0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 -- 172.21.15.70:0/3366132467 >> 172.21.15.70:0/3366132467 conn(0x7fd84c0cf9e0 msgr2=0x7fd84c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:06.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 -- 172.21.15.70:0/3366132467 shutdown_connections 2024-04-22T01:16:06.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 -- 172.21.15.70:0/3366132467 wait complete. 2024-04-22T01:16:06.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 Processor -- start 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.376+0000 7fd852d83640 1 -- start start 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd84c0b48a0 0x7fd84c140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 0x7fd84c138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd84c1445b0 0x7fd84c153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c06edb0 con 0x7fd84c0cecf0 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c0b36b0 con 0x7fd84c1445b0 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd84c06ef30 con 0x7fd84c0b48a0 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 0x7fd84c138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 0x7fd84c138a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38078/0 (socket says 172.21.15.70:38078) 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 -- 172.21.15.70:0/1307169354 learned_addr learned my addr 172.21.15.70:0/1307169354 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd850b1f640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd84c0b48a0 0x7fd84c140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd851320640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd84c1445b0 0x7fd84c153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 -- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd84c0b48a0 msgr2=0x7fd84c140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd84c0b48a0 0x7fd84c140dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 -- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd84c1445b0 msgr2=0x7fd84c153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd84c1445b0 0x7fd84c153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 -- 172.21.15.70:0/1307169354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84c139490 con 0x7fd84c0cecf0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd84bfff640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 0x7fd84c138a20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd83400c7f0 tx=0x7fd83400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834015070 con 0x7fd84c0cecf0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8340079d0 con 0x7fd84c0cecf0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84c139770 con 0x7fd84c0cecf0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834007da0 con 0x7fd84c0cecf0 2024-04-22T01:16:06.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84c139d30 con 0x7fd84c0cecf0 2024-04-22T01:16:06.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd84c04f510 con 0x7fd84c0cecf0 2024-04-22T01:16:06.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd834018070 con 0x7fd84c0cecf0 2024-04-22T01:16:06.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd8497fa640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd82c0678c0 0x7fd82c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fd834085a10 con 0x7fd84c0cecf0 2024-04-22T01:16:06.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd850b1f640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd82c0678c0 0x7fd82c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.380+0000 7fd850b1f640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd82c0678c0 0x7fd82c069d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd84400a790 tx=0x7fd844008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.384+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd834056710 con 0x7fd84c0cecf0 2024-04-22T01:16:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.584+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fd84c06de30 con 0x7fd84c0cecf0 2024-04-22T01:16:06.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.588+0000 7fd8497fa640 1 -- 172.21.15.70:0/1307169354 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) ==== 90+0+0 (secure 0 0 0) 0x7fd834056010 con 0x7fd84c0cecf0 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd82c0678c0 msgr2=0x7fd82c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd82c0678c0 0x7fd82c069d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd84400a790 tx=0x7fd844008040 comp rx=0 tx=0).stop 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 msgr2=0x7fd84c138a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 0x7fd84c138a20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd83400c7f0 tx=0x7fd83400ccb0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 shutdown_connections 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd82c0678c0 0x7fd82c069d80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd84c1445b0 0x7fd84c153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd84c0cecf0 0x7fd84c138a20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 --2- 172.21.15.70:0/1307169354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd84c0b48a0 0x7fd84c140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 >> 172.21.15.70:0/1307169354 conn(0x7fd84c0cf9e0 msgr2=0x7fd84c1428a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:06.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 shutdown_connections 2024-04-22T01:16:06.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.592+0000 7fd852d83640 1 -- 172.21.15.70:0/1307169354 wait complete. 2024-04-22T01:16:06.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-22T01:16:06.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.724+0000 7f9017c5d640 1 Processor -- start 2024-04-22T01:16:06.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.724+0000 7f9017c5d640 1 -- start start 2024-04-22T01:16:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f9017c5d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f90100b4ee0 0x7f90100b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f9017c5d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f901015d030 0x7f90100b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f9017c5d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015fb70 0x7f9010160000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f9017c5d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f90100b2cd0 con 0x7f90100b4ee0 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f9017c5d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f90100b5800 con 0x7f901015d030 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f9017c5d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f90100ccbd0 con 0x7f901015fb70 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f90100b4ee0 0x7f90100b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f90100b4ee0 0x7f90100b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38090/0 (socket says 172.21.15.70:38090) 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 -- 172.21.15.70:0/2843906688 learned_addr learned my addr 172.21.15.70:0/2843906688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90151f8640 1 --2- 172.21.15.70:0/2843906688 >> v2:172.21.15.98:3300/0 conn(0x7f901015d030 0x7f90100b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90161fa640 1 --2- 172.21.15.70:0/2843906688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015fb70 0x7f9010160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 -- 172.21.15.70:0/2843906688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015fb70 msgr2=0x7f9010160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 --2- 172.21.15.70:0/2843906688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015fb70 0x7f9010160000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 -- 172.21.15.70:0/2843906688 >> v2:172.21.15.98:3300/0 conn(0x7f901015d030 msgr2=0x7f90100b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 --2- 172.21.15.70:0/2843906688 >> v2:172.21.15.98:3300/0 conn(0x7f901015d030 0x7f90100b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 -- 172.21.15.70:0/2843906688 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90100cceb0 con 0x7f90100b4ee0 2024-04-22T01:16:06.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90159f9640 1 --2- 172.21.15.70:0/2843906688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90100b4ee0 0x7f90100b26a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9010071fb0 tx=0x7f900c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d8146b02ea58a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90149f7640 1 -- 172.21.15.70:0/2843906688 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c0ab620 con 0x7f90100b4ee0 2024-04-22T01:16:06.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90149f7640 1 -- 172.21.15.70:0/2843906688 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f900c0b0070 con 0x7f90100b4ee0 2024-04-22T01:16:06.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.728+0000 7f90149f7640 1 -- 172.21.15.70:0/2843906688 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c0ab9e0 con 0x7f90100b4ee0 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/2843906688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90100b4ee0 msgr2=0x7f90100b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- 172.21.15.70:0/2843906688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90100b4ee0 0x7f90100b26a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9010071fb0 tx=0x7f900c01c460 comp rx=0 tx=0).stop 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/2843906688 shutdown_connections 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- 172.21.15.70:0/2843906688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015fb70 0x7f9010160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- 172.21.15.70:0/2843906688 >> v2:172.21.15.98:3300/0 conn(0x7f901015d030 0x7f90100b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- 172.21.15.70:0/2843906688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90100b4ee0 0x7f90100b26a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/2843906688 >> 172.21.15.70:0/2843906688 conn(0x7f90100b3c00 msgr2=0x7f90100b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/2843906688 shutdown_connections 2024-04-22T01:16:06.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/2843906688 wait complete. 2024-04-22T01:16:06.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 Processor -- start 2024-04-22T01:16:06.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- start start 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90100b4ee0 0x7f90100c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015d030 0x7f9010157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 0x7f90100c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f901015d590 con 0x7f901015fb70 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f901015d410 con 0x7f90100b4ee0 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f901015c2b0 con 0x7f901015d030 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 0x7f90100c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 0x7f90100c6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38102/0 (socket says 172.21.15.70:38102) 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 -- 172.21.15.70:0/1504044566 learned_addr learned my addr 172.21.15.70:0/1504044566 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90151f8640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015d030 0x7f9010157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90159f9640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90100b4ee0 0x7f90100c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 -- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015d030 msgr2=0x7f9010157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015d030 0x7f9010157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 -- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90100b4ee0 msgr2=0x7f90100c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90100b4ee0 0x7f90100c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 -- 172.21.15.70:0/1504044566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90100c9b30 con 0x7f901015fb70 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90161fa640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 0x7f90100c6170 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9008007e90 tx=0x7f900800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9008014420 con 0x7f901015fb70 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9008019070 con 0x7f901015fb70 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9008014800 con 0x7f901015fb70 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90100c9e10 con 0x7f901015fb70 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.732+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90100c3ce0 con 0x7f901015fb70 2024-04-22T01:16:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.736+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f901004f510 con 0x7f901015fb70 2024-04-22T01:16:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.736+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f90080149a0 con 0x7f901015fb70 2024-04-22T01:16:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.736+0000 7f90067fc640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ff40678c0 0x7f8ff4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.736+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f9008087210 con 0x7f901015fb70 2024-04-22T01:16:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.736+0000 7f90159f9640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ff40678c0 0x7f8ff4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.736+0000 7f90159f9640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ff40678c0 0x7f8ff4069d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f900c0af000 tx=0x7f900c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:06.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.740+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9008057ef0 con 0x7f901015fb70 2024-04-22T01:16:06.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.964+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f9010072f70 con 0x7f901015fb70 2024-04-22T01:16:06.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f90067fc640 1 -- 172.21.15.70:0/1504044566 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7f90080577f0 con 0x7f901015fb70 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ff40678c0 msgr2=0x7f8ff4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ff40678c0 0x7f8ff4069d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f900c0af000 tx=0x7f900c0a7000 comp rx=0 tx=0).stop 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 msgr2=0x7f90100c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 0x7f90100c6170 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9008007e90 tx=0x7f900800e4b0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 shutdown_connections 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ff40678c0 0x7f8ff4069d80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f901015fb70 0x7f90100c6170 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901015d030 0x7f9010157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 --2- 172.21.15.70:0/1504044566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90100b4ee0 0x7f90100c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 >> 172.21.15.70:0/1504044566 conn(0x7f90100b3c00 msgr2=0x7f90100b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 shutdown_connections 2024-04-22T01:16:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:06.968+0000 7f9017c5d640 1 -- 172.21.15.70:0/1504044566 wait complete. 2024-04-22T01:16:06.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-22T01:16:07.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 Processor -- start 2024-04-22T01:16:07.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 -- start start 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a80cecf0 0x7f59a80b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f59a80b48a0 0x7f59a81646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a8144990 con 0x7f59a81445b0 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f59a80b3bd0 con 0x7f59a80b48a0 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59acc6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f59a80b2b50 con 0x7f59a80cecf0 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f59a80b48a0 0x7f59a81646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f59a80b48a0 0x7f59a81646a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38118/0 (socket says 172.21.15.70:38118) 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 -- 172.21.15.70:0/1402740339 learned_addr learned my addr 172.21.15.70:0/1402740339 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a5d9b640 1 --2- 172.21.15.70:0/1402740339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a80cecf0 0x7f59a80b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 -- 172.21.15.70:0/1402740339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a80cecf0 msgr2=0x7f59a80b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 --2- 172.21.15.70:0/1402740339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a80cecf0 0x7f59a80b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 -- 172.21.15.70:0/1402740339 >> v1:172.21.15.98:6789/0 conn(0x7f59a81445b0 legacy=0x7f59a8072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:07.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.096+0000 7f59a6d9d640 1 -- 172.21.15.70:0/1402740339 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a80b2e10 con 0x7f59a80b48a0 2024-04-22T01:16:07.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59a6d9d640 1 --2- 172.21.15.70:0/1402740339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a81646a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5998009e50 tx=0x7f599801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570acf6d31465e09 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59a559a640 1 -- 172.21.15.70:0/1402740339 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980ac020 con 0x7f59a80b48a0 2024-04-22T01:16:07.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59a559a640 1 -- 172.21.15.70:0/1402740339 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59980a64a0 con 0x7f59a80b48a0 2024-04-22T01:16:07.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59a559a640 1 -- 172.21.15.70:0/1402740339 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980a6870 con 0x7f59a80b48a0 2024-04-22T01:16:07.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 -- 172.21.15.70:0/1402740339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 msgr2=0x7f59a81646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 --2- 172.21.15.70:0/1402740339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a81646a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5998009e50 tx=0x7f599801c810 comp rx=0 tx=0).stop 2024-04-22T01:16:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 -- 172.21.15.70:0/1402740339 shutdown_connections 2024-04-22T01:16:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 --2- 172.21.15.70:0/1402740339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a81646a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 --2- 172.21.15.70:0/1402740339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a80cecf0 0x7f59a80b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 -- 172.21.15.70:0/1402740339 >> 172.21.15.70:0/1402740339 conn(0x7f59a80cf9e0 msgr2=0x7f59a80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 -- 172.21.15.70:0/1402740339 shutdown_connections 2024-04-22T01:16:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 -- 172.21.15.70:0/1402740339 wait complete. 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 Processor -- start 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.100+0000 7f59acc6b640 1 -- start start 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a813d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a80cecf0 0x7f59a813b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a81445b0 0x7f59a8158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59a806ef30 con 0x7f59a80b48a0 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59a806edb0 con 0x7f59a80cecf0 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f59a8167400 con 0x7f59a81445b0 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a813d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a813d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38120/0 (socket says 172.21.15.70:38120) 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 -- 172.21.15.70:0/585045855 learned_addr learned my addr 172.21.15.70:0/585045855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a6d9d640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a81445b0 0x7f59a8158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a5d9b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a80cecf0 0x7f59a813b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 -- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a81445b0 msgr2=0x7f59a8158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a81445b0 0x7f59a8158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 -- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a80cecf0 msgr2=0x7f59a813b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a80cecf0 0x7f59a813b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 -- 172.21.15.70:0/585045855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a8159660 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a659c640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a813d040 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f59a000b440 tx=0x7f59a000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0015020 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a0011890 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a8162ab0 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0011c80 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a8163070 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59a804f510 con 0x7f59a80b48a0 2024-04-22T01:16:07.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59a0018020 con 0x7f59a80b48a0 2024-04-22T01:16:07.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f599effd640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59880678c0 0x7f5988069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a5d9b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59880678c0 0x7f5988069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f59a0085830 con 0x7f59a80b48a0 2024-04-22T01:16:07.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.104+0000 7f59a5d9b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59880678c0 0x7f5988069d80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f59a815e260 tx=0x7f5990008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.107+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f59a001f030 con 0x7f59a80b48a0 2024-04-22T01:16:07.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.311+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f59a8141910 con 0x7f59a80b48a0 2024-04-22T01:16:07.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f599effd640 1 -- 172.21.15.70:0/585045855 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) ==== 97+0+0 (secure 0 0 0) 0x7f59a000d040 con 0x7f59a80b48a0 2024-04-22T01:16:07.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59880678c0 msgr2=0x7f5988069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59880678c0 0x7f5988069d80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f59a815e260 tx=0x7f5990008040 comp rx=0 tx=0).stop 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 msgr2=0x7f59a813d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a813d040 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f59a000b440 tx=0x7f59a000b900 comp rx=0 tx=0).stop 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 shutdown_connections 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f59880678c0 0x7f5988069d80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59a81445b0 0x7f59a8158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59a80cecf0 0x7f59a813b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 --2- 172.21.15.70:0/585045855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a80b48a0 0x7f59a813d040 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 >> 172.21.15.70:0/585045855 conn(0x7f59a80cf9e0 msgr2=0x7f59a8157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:07.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 shutdown_connections 2024-04-22T01:16:07.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.315+0000 7f59acc6b640 1 -- 172.21.15.70:0/585045855 wait complete. 2024-04-22T01:16:07.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-22T01:16:07.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f427ab09640 1 Processor -- start 2024-04-22T01:16:07.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f427ab09640 1 -- start start 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f427ab09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f427415e040 0x7f42740b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f427ab09640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740b8330 con 0x7f42740cf0b0 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f427ab09640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740b8610 con 0x7f42740b7db0 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f427ab09640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f42740b88f0 con 0x7f427415e040 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42790a6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f42740cf0b0 0x7f42740d1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33046/0 (socket says 172.21.15.70:33046) 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42790a6640 1 -- 172.21.15.70:0/817047958 learned_addr learned my addr 172.21.15.70:0/817047958 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42788a5640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f427415e040 0x7f42740b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239643204 0 0) 0x7f42740b8330 con 0x7f42740cf0b0 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f425c003530 con 0x7f42740cf0b0 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4204660509 0 0) 0x7f425c003530 con 0x7f42740cf0b0 2024-04-22T01:16:07.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f427415e040 msgr2=0x7f42740b7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f427415e040 0x7f42740b7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 >> v1:172.21.15.98:6789/0 conn(0x7f42740b7db0 legacy=0x7f42740ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42740b8bd0 con 0x7f42740cf0b0 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f426c001680 con 0x7f42740cf0b0 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 >> v1:172.21.15.70:6789/0 conn(0x7f42740cf0b0 legacy=0x7f42740d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c003a60 0x7f425c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f425c004810 0x7f425c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f425c0052f0 0x7f425c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42740b88f0 con 0x7f425c004810 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00bff0 con 0x7f425c003a60 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f425c00c1f0 con 0x7f425c0052f0 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.451+0000 7f4273fff640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f425c004810 0x7f425c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42788a5640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c003a60 0x7f425c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42790a6640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f425c0052f0 0x7f425c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f4273fff640 1 -- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f425c0052f0 msgr2=0x7f425c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:07.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f4273fff640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f425c0052f0 0x7f425c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f4273fff640 1 -- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c003a60 msgr2=0x7f425c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:07.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f4273fff640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c003a60 0x7f425c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f4273fff640 1 -- 172.21.15.70:0/817047958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425c003530 con 0x7f425c004810 2024-04-22T01:16:07.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42788a5640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c003a60 0x7f425c003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f4273fff640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f425c004810 0x7f425c004c90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4268001780 tx=0x7f426801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf7e575c6163106 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42680013a0 con 0x7f425c004810 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42680052f0 con 0x7f425c004810 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42737fe640 1 -- 172.21.15.70:0/817047958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268003840 con 0x7f425c004810 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f427ab09640 1 -- 172.21.15.70:0/817047958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f425c004810 msgr2=0x7f425c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f427ab09640 1 --2- 172.21.15.70:0/817047958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f425c004810 0x7f425c004c90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4268001780 tx=0x7f426801c3f0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f42790a6640 1 -- 172.21.15.70:0/817047958 reap_dead start 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f427ab09640 1 -- 172.21.15.70:0/817047958 shutdown_connections 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f427ab09640 1 -- 172.21.15.70:0/817047958 >> 172.21.15.70:0/817047958 conn(0x7f4274072d20 msgr2=0x7f4274073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:07.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f427ab09640 1 -- 172.21.15.70:0/817047958 shutdown_connections 2024-04-22T01:16:07.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.455+0000 7f427ab09640 1 -- 172.21.15.70:0/817047958 wait complete. 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 Processor -- start 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- start start 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c004810 0x7f42740cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4274155d10 0x7f427415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 0x7f4274160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42741699f0 con 0x7f4274160170 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42740d2fb0 con 0x7f425c004810 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4274169b70 con 0x7f4274155d10 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 0x7f4274160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 0x7f4274160790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38146/0 (socket says 172.21.15.70:38146) 2024-04-22T01:16:07.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 -- 172.21.15.70:0/2189900851 learned_addr learned my addr 172.21.15.70:0/2189900851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42788a5640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c004810 0x7f42740cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f4273fff640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4274155d10 0x7f427415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 -- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4274155d10 msgr2=0x7f427415fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4274155d10 0x7f427415fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 -- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c004810 msgr2=0x7f42740cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c004810 0x7f42740cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 -- 172.21.15.70:0/2189900851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4274158bf0 con 0x7f4274160170 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42790a6640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 0x7f4274160790 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4274161900 tx=0x7f426c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c019070 con 0x7f4274160170 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c010510 con 0x7f4274160170 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42740d3ab0 con 0x7f4274160170 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c010900 con 0x7f4274160170 2024-04-22T01:16:07.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427415f430 con 0x7f4274160170 2024-04-22T01:16:07.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f427404f510 con 0x7f4274160170 2024-04-22T01:16:07.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.459+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f426c01c070 con 0x7f4274160170 2024-04-22T01:16:07.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.463+0000 7f42717fa640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f42600678c0 0x7f4260069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.463+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f426c084ef0 con 0x7f4274160170 2024-04-22T01:16:07.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.463+0000 7f42788a5640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f42600678c0 0x7f4260069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.463+0000 7f42788a5640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f42600678c0 0x7f4260069d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f42740d1db0 tx=0x7f426400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.467+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4274158bf0 con 0x7f4274160170 2024-04-22T01:16:07.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.679+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f42740b6250 con 0x7f4274160170 2024-04-22T01:16:07.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.683+0000 7f42717fa640 1 -- 172.21.15.70:0/2189900851 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) ==== 90+0+0 (secure 0 0 0) 0x7f426c055c80 con 0x7f4274160170 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f42600678c0 msgr2=0x7f4260069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f42600678c0 0x7f4260069d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f42740d1db0 tx=0x7f426400b040 comp rx=0 tx=0).stop 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 msgr2=0x7f4274160790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 0x7f4274160790 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4274161900 tx=0x7f426c00e960 comp rx=0 tx=0).stop 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 shutdown_connections 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f42600678c0 0x7f4260069d80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4274160170 0x7f4274160790 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4274155d10 0x7f427415fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 --2- 172.21.15.70:0/2189900851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f425c004810 0x7f42740cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 >> 172.21.15.70:0/2189900851 conn(0x7f4274072d20 msgr2=0x7f4274073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:07.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 shutdown_connections 2024-04-22T01:16:07.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.687+0000 7f427ab09640 1 -- 172.21.15.70:0/2189900851 wait complete. 2024-04-22T01:16:07.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:07.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.815+0000 7fb319740640 1 Processor -- start 2024-04-22T01:16:07.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.815+0000 7fb319740640 1 -- start start 2024-04-22T01:16:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb319740640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb3140cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb319740640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3140705d0 0x7fb3140b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb319740640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3140715f0 con 0x7fb3140b2660 2024-04-22T01:16:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb319740640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb3140b3f10 con 0x7fb3140705d0 2024-04-22T01:16:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb319740640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb3140d05f0 con 0x7fb3141445a0 2024-04-22T01:16:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb3140cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb3127fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3140705d0 0x7fb3140b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb3140cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:48038/0 (socket says 172.21.15.70:48038) 2024-04-22T01:16:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 -- 172.21.15.70:0/2081516216 learned_addr learned my addr 172.21.15.70:0/2081516216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 -- 172.21.15.70:0/2081516216 >> v2:172.21.15.98:3300/0 conn(0x7fb3140705d0 msgr2=0x7fb3140b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 --2- 172.21.15.70:0/2081516216 >> v2:172.21.15.98:3300/0 conn(0x7fb3140705d0 0x7fb3140b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 -- 172.21.15.70:0/2081516216 >> v1:172.21.15.98:6789/0 conn(0x7fb3140b2660 legacy=0x7fb314161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.819+0000 7fb312ffd640 1 -- 172.21.15.70:0/2081516216 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb314071030 con 0x7fb3141445a0 2024-04-22T01:16:07.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb312ffd640 1 --2- 172.21.15.70:0/2081516216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb3140cdec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb304009870 tx=0x7fb30401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f7d8de739e991e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb311ffb640 1 -- 172.21.15.70:0/2081516216 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b1070 con 0x7fb3141445a0 2024-04-22T01:16:07.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb311ffb640 1 -- 172.21.15.70:0/2081516216 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040a8050 con 0x7fb3141445a0 2024-04-22T01:16:07.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb311ffb640 1 -- 172.21.15.70:0/2081516216 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040ac860 con 0x7fb3141445a0 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- 172.21.15.70:0/2081516216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 msgr2=0x7fb3140cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 --2- 172.21.15.70:0/2081516216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb3140cdec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb304009870 tx=0x7fb30401c4d0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- 172.21.15.70:0/2081516216 shutdown_connections 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 --2- 172.21.15.70:0/2081516216 >> v2:172.21.15.98:3300/0 conn(0x7fb3140705d0 0x7fb3140b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 --2- 172.21.15.70:0/2081516216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb3140cdec0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- 172.21.15.70:0/2081516216 >> 172.21.15.70:0/2081516216 conn(0x7fb31406c300 msgr2=0x7fb31406c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- 172.21.15.70:0/2081516216 shutdown_connections 2024-04-22T01:16:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- 172.21.15.70:0/2081516216 wait complete. 2024-04-22T01:16:07.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 Processor -- start 2024-04-22T01:16:07.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- start start 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140705d0 0x7fb31415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 0x7fb314146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb314142db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140b2e80 con 0x7fb3140b2660 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140d0210 con 0x7fb3140705d0 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb319740640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb3141f8440 con 0x7fb3141445a0 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 0x7fb314146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 0x7fb314146870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38154/0 (socket says 172.21.15.70:38154) 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 -- 172.21.15.70:0/1001568920 learned_addr learned my addr 172.21.15.70:0/1001568920 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb312ffd640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140705d0 0x7fb31415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3137fe640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb314142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 -- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 msgr2=0x7fb314142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb314142db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 -- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140705d0 msgr2=0x7fb31415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140705d0 0x7fb31415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb3127fc640 1 -- 172.21.15.70:0/1001568920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3141f8720 con 0x7fb3140b2660 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.823+0000 7fb312ffd640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140705d0 0x7fb31415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb3127fc640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 0x7fb314146870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb30800e930 tx=0x7fb30800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308017070 con 0x7fb3140b2660 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30800ca00 con 0x7fb3140b2660 2024-04-22T01:16:07.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3141f8a00 con 0x7fb3140b2660 2024-04-22T01:16:07.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30800cdb0 con 0x7fb3140b2660 2024-04-22T01:16:07.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3141f8fc0 con 0x7fb3140b2660 2024-04-22T01:16:07.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb314068210 con 0x7fb3140b2660 2024-04-22T01:16:07.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb30801a070 con 0x7fb3140b2660 2024-04-22T01:16:07.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb2fb7fe640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb3000678c0 0x7fb300069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:07.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fb308085ee0 con 0x7fb3140b2660 2024-04-22T01:16:07.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb312ffd640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb3000678c0 0x7fb300069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:07.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.827+0000 7fb312ffd640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb3000678c0 0x7fb300069d80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb3040ae000 tx=0x7fb3040a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:07.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.831+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb308056ca0 con 0x7fb3140b2660 2024-04-22T01:16:07.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.979+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3140d12a0 con 0x7fb3140b2660 2024-04-22T01:16:07.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.979+0000 7fb2fb7fe640 1 -- 172.21.15.70:0/1001568920 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+13128 (secure 0 0 0) 0x7fb3080565a0 con 0x7fb3140b2660 2024-04-22T01:16:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:07.985 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":34,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:05.155838+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:07.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb3000678c0 msgr2=0x7fb300069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb3000678c0 0x7fb300069d80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb3040ae000 tx=0x7fb3040a7000 comp rx=0 tx=0).stop 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 msgr2=0x7fb314146870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 0x7fb314146870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb30800e930 tx=0x7fb30800edf0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 shutdown_connections 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb3000678c0 0x7fb300069d80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb3141445a0 0x7fb314142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3140b2660 0x7fb314146870 secure :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb30800e930 tx=0x7fb30800edf0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 --2- 172.21.15.70:0/1001568920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140705d0 0x7fb31415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 >> 172.21.15.70:0/1001568920 conn(0x7fb31406c300 msgr2=0x7fb314160860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 shutdown_connections 2024-04-22T01:16:07.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:07.983+0000 7fb319740640 1 -- 172.21.15.70:0/1001568920 wait complete. 2024-04-22T01:16:08.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:08.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:08.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.111+0000 7f64925ee640 1 Processor -- start 2024-04-22T01:16:08.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.111+0000 7f64925ee640 1 -- start start 2024-04-22T01:16:08.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f64925ee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 0x7f648c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f64925ee640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f648c0cd220 0x7f648c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f64925ee640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c15c290 con 0x7f648c0720b0 2024-04-22T01:16:08.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f64925ee640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f648c072510 con 0x7f648c0cd220 2024-04-22T01:16:08.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f64925ee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f648c15c570 con 0x7f648c0b4450 2024-04-22T01:16:08.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f648c0720b0 0x7f648c0ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33056/0 (socket says 172.21.15.70:33056) 2024-04-22T01:16:08.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 -- 172.21.15.70:0/447088238 learned_addr learned my addr 172.21.15.70:0/447088238 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648bfff640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 0x7f648c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f6490b8b640 1 --2- 172.21.15.70:0/447088238 >> v2:172.21.15.98:3300/0 conn(0x7f648c0cd220 0x7f648c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1063808017 0 0) 0x7f648c15c290 con 0x7f648c0720b0 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6474003530 con 0x7f648c0720b0 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 436341354 0 0) 0x7f6474003530 con 0x7f648c0720b0 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 msgr2=0x7f648c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 0x7f648c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 >> v2:172.21.15.98:3300/0 conn(0x7f648c0cd220 msgr2=0x7f648c0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 --2- 172.21.15.70:0/447088238 >> v2:172.21.15.98:3300/0 conn(0x7f648c0cd220 0x7f648c0b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c15c850 con 0x7f648c0720b0 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6480001680 con 0x7f648c0720b0 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 >> v1:172.21.15.70:6789/0 conn(0x7f648c0720b0 legacy=0x7f648c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64740039c0 0x7f6474003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6474004810 0x7f6474004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 0x7f64740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f648c15c570 con 0x7f6474004810 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f647400b9d0 con 0x7f64740039c0 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f647400bb50 con 0x7f648c0b4450 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6474004810 0x7f6474004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648bfff640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64740039c0 0x7f6474003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f6490b8b640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 0x7f64740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 -- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 msgr2=0x7f64740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c0b4450 0x7f64740051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 -- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64740039c0 msgr2=0x7f6474003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64740039c0 0x7f6474003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 -- 172.21.15.70:0/447088238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6474003530 con 0x7f6474004810 2024-04-22T01:16:08.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648bfff640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64740039c0 0x7f6474003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:08.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.115+0000 7f648b7fe640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6474004810 0x7f6474004c90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f648000c240 tx=0x7f648001f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f885ac8a3e0e42a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480007d40 con 0x7f6474004810 2024-04-22T01:16:08.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6480005b00 con 0x7f6474004810 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648affd640 1 -- 172.21.15.70:0/447088238 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480007070 con 0x7f6474004810 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/447088238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6474004810 msgr2=0x7f6474004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 --2- 172.21.15.70:0/447088238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6474004810 0x7f6474004c90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f648000c240 tx=0x7f648001f410 comp rx=0 tx=0).stop 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f6490b8b640 1 -- 172.21.15.70:0/447088238 reap_dead start 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/447088238 shutdown_connections 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/447088238 >> 172.21.15.70:0/447088238 conn(0x7f648c0b1d40 msgr2=0x7f648c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/447088238 shutdown_connections 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/447088238 wait complete. 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 Processor -- start 2024-04-22T01:16:08.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- start start 2024-04-22T01:16:08.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6474004810 0x7f648c14d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 0x7f648c1471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c14c190 0x7f648c14da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f648c0b4830 con 0x7f648c1f5d80 2024-04-22T01:16:08.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f648c15d050 con 0x7f6474004810 2024-04-22T01:16:08.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f648c0b49b0 con 0x7f648c14c190 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 0x7f648c1471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 0x7f648c1471c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38176/0 (socket says 172.21.15.70:38176) 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 -- 172.21.15.70:0/3462579131 learned_addr learned my addr 172.21.15.70:0/3462579131 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648bfff640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6474004810 0x7f648c14d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f6490b8b640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c14c190 0x7f648c14da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 -- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c14c190 msgr2=0x7f648c14da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c14c190 0x7f648c14da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 -- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6474004810 msgr2=0x7f648c14d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6474004810 0x7f648c14d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 -- 172.21.15.70:0/3462579131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c1f6200 con 0x7f648c1f5d80 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f6490b8b640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c14c190 0x7f648c14da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f648b7fe640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 0x7f648c1471c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f648000c240 tx=0x7f6474005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480005d70 con 0x7f648c1f5d80 2024-04-22T01:16:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64800047f0 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c14b3d0 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480004bc0 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.119+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c14b990 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.123+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c068210 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.123+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f64800ba030 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.123+0000 7f6488ff9640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64780678f0 0x7f6478069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.123+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f64801224b0 con 0x7f648c1f5d80 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.123+0000 7f648bfff640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64780678f0 0x7f6478069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.123+0000 7f648bfff640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64780678f0 0x7f6478069db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f647c006f90 tx=0x7f647c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.127+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f64800c1030 con 0x7f648c1f5d80 2024-04-22T01:16:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f648c074750 con 0x7f648c1f5d80 2024-04-22T01:16:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f6488ff9640 1 -- 172.21.15.70:0/3462579131 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f6480019120 con 0x7f648c1f5d80 2024-04-22T01:16:08.287 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64780678f0 msgr2=0x7f6478069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64780678f0 0x7f6478069db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f647c006f90 tx=0x7f647c008040 comp rx=0 tx=0).stop 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 msgr2=0x7f648c1471c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 0x7f648c1471c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f648000c240 tx=0x7f6474005840 comp rx=0 tx=0).stop 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 shutdown_connections 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64780678f0 0x7f6478069db0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f648c14c190 0x7f648c14da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f648c1f5d80 0x7f648c1471c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 --2- 172.21.15.70:0/3462579131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6474004810 0x7f648c14d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 >> 172.21.15.70:0/3462579131 conn(0x7f648c0b1d40 msgr2=0x7f648c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 shutdown_connections 2024-04-22T01:16:08.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.283+0000 7f64925ee640 1 -- 172.21.15.70:0/3462579131 wait complete. 2024-04-22T01:16:08.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:16:08.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 Processor -- start 2024-04-22T01:16:08.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 -- start start 2024-04-22T01:16:08.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fad740cc150 0x7fad74071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74072060 0x7fad7406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad740cd7b0 con 0x7fad741455b0 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fad740724d0 con 0x7fad740cc150 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad78fb4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fad740cc710 con 0x7fad74072060 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad69d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fad740cc150 0x7fad74071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad7259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fad741455b0 0x7fad740cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33058/0 (socket says 172.21.15.70:33058) 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad7259c640 1 -- 172.21.15.70:0/6838888 learned_addr learned my addr 172.21.15.70:0/6838888 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad72d9d640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74072060 0x7fad7406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad69d9b640 1 -- 172.21.15.70:0/6838888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74072060 msgr2=0x7fad7406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad69d9b640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74072060 0x7fad7406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad69d9b640 1 -- 172.21.15.70:0/6838888 >> v1:172.21.15.70:6789/0 conn(0x7fad741455b0 legacy=0x7fad740cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad71d9b640 1 -- 172.21.15.70:0/6838888 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118125084 0 0) 0x7fad740cd7b0 con 0x7fad741455b0 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad69d9b640 1 -- 172.21.15.70:0/6838888 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad740cc9f0 con 0x7fad740cc150 2024-04-22T01:16:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.419+0000 7fad72d9d640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74072060 0x7fad7406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad69d9b640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad740cc150 0x7fad74071b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fad640097c0 tx=0x7fad6401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47eb72f6c5e4af1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad71d9b640 1 -- 172.21.15.70:0/6838888 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640b0070 con 0x7fad740cc150 2024-04-22T01:16:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad71d9b640 1 -- 172.21.15.70:0/6838888 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad640ab600 con 0x7fad740cc150 2024-04-22T01:16:08.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad71d9b640 1 -- 172.21.15.70:0/6838888 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640ab9d0 con 0x7fad740cc150 2024-04-22T01:16:08.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/6838888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad740cc150 msgr2=0x7fad74071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad740cc150 0x7fad74071b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fad640097c0 tx=0x7fad6401c380 comp rx=0 tx=0).stop 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/6838888 shutdown_connections 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74072060 0x7fad7406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 --2- 172.21.15.70:0/6838888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad740cc150 0x7fad74071b20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/6838888 >> 172.21.15.70:0/6838888 conn(0x7fad7406b1f0 msgr2=0x7fad7406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/6838888 shutdown_connections 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/6838888 wait complete. 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 Processor -- start 2024-04-22T01:16:08.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- start start 2024-04-22T01:16:08.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 0x7fad74160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad740cc150 0x7fad7415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad741455b0 0x7fad741416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad7406f2e0 con 0x7fad74072060 2024-04-22T01:16:08.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad7406f160 con 0x7fad740cc150 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fad74168b40 con 0x7fad741455b0 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 0x7fad74160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 0x7fad74160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38198/0 (socket says 172.21.15.70:38198) 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 -- 172.21.15.70:0/2990734198 learned_addr learned my addr 172.21.15.70:0/2990734198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 -- 172.21.15.70:0/2990734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad741455b0 msgr2=0x7fad741416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad741455b0 0x7fad741416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 -- 172.21.15.70:0/2990734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad740cc150 msgr2=0x7fad7415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad740cc150 0x7fad7415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 -- 172.21.15.70:0/2990734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7415ea00 con 0x7fad74072060 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad7259c640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 0x7fad74160860 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c4b0 tx=0x7fad5c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c018070 con 0x7fad74072060 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5c007500 con 0x7fad74072060 2024-04-22T01:16:08.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7415ece0 con 0x7fad74072060 2024-04-22T01:16:08.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c004a30 con 0x7fad74072060 2024-04-22T01:16:08.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.423+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7415efc0 con 0x7fad74072060 2024-04-22T01:16:08.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.427+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad74068210 con 0x7fad74072060 2024-04-22T01:16:08.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.427+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad5c01b070 con 0x7fad74072060 2024-04-22T01:16:08.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.427+0000 7fad6b7fe640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fad540678c0 0x7fad54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.427+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fad5c084da0 con 0x7fad74072060 2024-04-22T01:16:08.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.427+0000 7fad69d9b640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fad540678c0 0x7fad54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.427+0000 7fad69d9b640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fad540678c0 0x7fad54069d80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad640ad000 tx=0x7fad64008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.431+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fad5c055aa0 con 0x7fad74072060 2024-04-22T01:16:08.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.591+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fad7415f510 con 0x7fad74072060 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.591+0000 7fad6b7fe640 1 -- 172.21.15.70:0/2990734198 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) ==== 84+0+65 (secure 0 0 0) 0x7fad5c0553a0 con 0x7fad74072060 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:08.597 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fad540678c0 msgr2=0x7fad54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fad540678c0 0x7fad54069d80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad640ad000 tx=0x7fad64008040 comp rx=0 tx=0).stop 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 msgr2=0x7fad74160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 0x7fad74160860 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c4b0 tx=0x7fad5c00c970 comp rx=0 tx=0).stop 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 shutdown_connections 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fad540678c0 0x7fad54069d80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad741455b0 0x7fad741416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad740cc150 0x7fad7415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 --2- 172.21.15.70:0/2990734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad74072060 0x7fad74160860 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 >> 172.21.15.70:0/2990734198 conn(0x7fad7406b1f0 msgr2=0x7fad7406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 shutdown_connections 2024-04-22T01:16:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.595+0000 7fad78fb4640 1 -- 172.21.15.70:0/2990734198 wait complete. 2024-04-22T01:16:08.616 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:16:08.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:16:08.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 Processor -- start 2024-04-22T01:16:08.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 -- start start 2024-04-22T01:16:08.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efccc15de20 0x7efccc0b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efccc0b6db0 0x7efccc0d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0cea10 0x7efccc0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efccc0cf3c0 con 0x7efccc0b6db0 2024-04-22T01:16:08.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efccc0b75e0 con 0x7efccc15de20 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efcd2243640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efccc0b78c0 con 0x7efccc0cea10 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efccc0b6db0 0x7efccc0d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efccc0b6db0 0x7efccc0d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38212/0 (socket says 172.21.15.70:38212) 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 -- 172.21.15.70:0/3264014670 learned_addr learned my addr 172.21.15.70:0/3264014670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccb7fe640 1 --2- 172.21.15.70:0/3264014670 >> v2:172.21.15.98:3300/0 conn(0x7efccc15de20 0x7efccc0b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccbfff640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0cea10 0x7efccc0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 -- 172.21.15.70:0/3264014670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0cea10 msgr2=0x7efccc0cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0cea10 0x7efccc0cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 -- 172.21.15.70:0/3264014670 >> v2:172.21.15.98:3300/0 conn(0x7efccc15de20 msgr2=0x7efccc0b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 --2- 172.21.15.70:0/3264014670 >> v2:172.21.15.98:3300/0 conn(0x7efccc15de20 0x7efccc0b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccaffd640 1 -- 172.21.15.70:0/3264014670 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc0b7ba0 con 0x7efccc0b6db0 2024-04-22T01:16:08.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.739+0000 7efccbfff640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0cea10 0x7efccc0cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efccaffd640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc0b6db0 0x7efccc0d1d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efccc0b77f0 tx=0x7efcc001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c2457841b824f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcca7fc640 1 -- 172.21.15.70:0/3264014670 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc00b5070 con 0x7efccc0b6db0 2024-04-22T01:16:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcca7fc640 1 -- 172.21.15.70:0/3264014670 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc00a89d0 con 0x7efccc0b6db0 2024-04-22T01:16:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcca7fc640 1 -- 172.21.15.70:0/3264014670 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc00a8da0 con 0x7efccc0b6db0 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 -- 172.21.15.70:0/3264014670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc0b6db0 msgr2=0x7efccc0d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc0b6db0 0x7efccc0d1d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efccc0b77f0 tx=0x7efcc001ed10 comp rx=0 tx=0).stop 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 -- 172.21.15.70:0/3264014670 shutdown_connections 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0cea10 0x7efccc0cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 --2- 172.21.15.70:0/3264014670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc0b6db0 0x7efccc0d1d80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 --2- 172.21.15.70:0/3264014670 >> v2:172.21.15.98:3300/0 conn(0x7efccc15de20 0x7efccc0b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 -- 172.21.15.70:0/3264014670 >> 172.21.15.70:0/3264014670 conn(0x7efccc072d20 msgr2=0x7efccc073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 -- 172.21.15.70:0/3264014670 shutdown_connections 2024-04-22T01:16:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 -- 172.21.15.70:0/3264014670 wait complete. 2024-04-22T01:16:08.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 Processor -- start 2024-04-22T01:16:08.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.743+0000 7efcd2243640 1 -- start start 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0b6db0 0x7efccc15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efccc0cea10 0x7efccc0cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 0x7efccc0cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efccc15e200 con 0x7efccc15de20 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efccc0b8160 con 0x7efccc0cea10 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efccc0d22c0 con 0x7efccc0b6db0 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 0x7efccc0cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 0x7efccc0cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38224/0 (socket says 172.21.15.70:38224) 2024-04-22T01:16:08.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 -- 172.21.15.70:0/2855772030 learned_addr learned my addr 172.21.15.70:0/2855772030 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccaffd640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efccc0cea10 0x7efccc0cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccb7fe640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0b6db0 0x7efccc15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 -- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0b6db0 msgr2=0x7efccc15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0b6db0 0x7efccc15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 -- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efccc0cea10 msgr2=0x7efccc0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efccc0cea10 0x7efccc0cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 -- 172.21.15.70:0/2855772030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc0cd7d0 con 0x7efccc15de20 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccb7fe640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0b6db0 0x7efccc15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccbfff640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 0x7efccc0cc050 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efcc400cef0 tx=0x7efcc400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc401a070 con 0x7efccc15de20 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc400d840 con 0x7efccc15de20 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efccc0d30c0 con 0x7efccc15de20 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc400dc30 con 0x7efccc15de20 2024-04-22T01:16:08.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efccc0d3680 con 0x7efccc15de20 2024-04-22T01:16:08.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efccc0d2ee0 con 0x7efccc15de20 2024-04-22T01:16:08.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efcc401d070 con 0x7efccc15de20 2024-04-22T01:16:08.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcb3fff640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efcb80678c0 0x7efcb8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:08.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccb7fe640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efcb80678c0 0x7efcb8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7efcc4086280 con 0x7efccc15de20 2024-04-22T01:16:08.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.747+0000 7efccb7fe640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efcb80678c0 0x7efcb8069d80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009840 tx=0x7efcbc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:08.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.751+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7efccc0d2ee0 con 0x7efccc15de20 2024-04-22T01:16:08.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:08.911+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efccc068210 con 0x7efccc15de20 2024-04-22T01:16:09.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.151+0000 7efcb3fff640 1 -- 172.21.15.70:0/2855772030 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) ==== 138+0+0 (secure 0 0 0) 0x7efcc4057010 con 0x7efccc15de20 2024-04-22T01:16:09.156 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efcb80678c0 msgr2=0x7efcb8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efcb80678c0 0x7efcb8069d80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009840 tx=0x7efcbc008040 comp rx=0 tx=0).stop 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 msgr2=0x7efccc0cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 0x7efccc0cc050 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efcc400cef0 tx=0x7efcc400a710 comp rx=0 tx=0).stop 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 shutdown_connections 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efcb80678c0 0x7efcb8069d80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15de20 0x7efccc0cc050 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efccc0cea10 0x7efccc0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 --2- 172.21.15.70:0/2855772030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efccc0b6db0 0x7efccc15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 >> 172.21.15.70:0/2855772030 conn(0x7efccc072d20 msgr2=0x7efccc15c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 shutdown_connections 2024-04-22T01:16:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.155+0000 7efcd2243640 1 -- 172.21.15.70:0/2855772030 wait complete. 2024-04-22T01:16:09.187 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:16:09.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 Processor -- start 2024-04-22T01:16:09.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 -- start start 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99dc09a440 0x7f99dc0d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 0x7f99dc0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99dc0ce010 con 0x7f99dc15bcf0 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f99dc0ce2f0 con 0x7f99dc09a440 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99e262c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99dc0b10b0 con 0x7f99dc0b2ba0 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99dbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f99dc15bcf0 0x7f99dc099d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33068/0 (socket says 172.21.15.70:33068) 2024-04-22T01:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.315+0000 7f99dbfff640 1 -- 172.21.15.70:0/2023499673 learned_addr learned my addr 172.21.15.70:0/2023499673 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:09.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99e0bc9640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 0x7f99dc0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 --2- 172.21.15.70:0/2023499673 >> v2:172.21.15.98:3300/0 conn(0x7f99dc09a440 0x7f99dc0d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29403012 0 0) 0x7f99dc0ce010 con 0x7f99dc15bcf0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c4003530 con 0x7f99dc15bcf0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2653870495 0 0) 0x7f99c4003530 con 0x7f99dc15bcf0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 msgr2=0x7f99dc0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 0x7f99dc0cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 >> v2:172.21.15.98:3300/0 conn(0x7f99dc09a440 msgr2=0x7f99dc0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 --2- 172.21.15.70:0/2023499673 >> v2:172.21.15.98:3300/0 conn(0x7f99dc09a440 0x7f99dc0d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99dc0b1390 con 0x7f99dc15bcf0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f99cc001680 con 0x7f99dc15bcf0 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 >> v1:172.21.15.70:6789/0 conn(0x7f99dc15bcf0 legacy=0x7f99dc099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 0x7f99c40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99c4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c4004d50 0x7f99c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc0ce2f0 con 0x7f99c4003f40 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc0b10b0 con 0x7f99c4004d50 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99c400b9d0 con 0x7f99dc0b2ba0 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99c4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99e0bc9640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c4004d50 0x7f99c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99dbfff640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 0x7f99c40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 -- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 msgr2=0x7f99c40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0b2ba0 0x7f99c40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 -- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c4004d50 msgr2=0x7f99c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c4004d50 0x7f99c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 -- 172.21.15.70:0/2023499673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c4003530 con 0x7f99c4003f40 2024-04-22T01:16:09.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.319+0000 7f99db7fe640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99c4004810 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009ae0 tx=0x7f99d001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383aa7acc4c2bbfc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:09.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00b2070 con 0x7f99c4003f40 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d00a56a0 con 0x7f99c4003f40 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99daffd640 1 -- 172.21.15.70:0/2023499673 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00096b0 con 0x7f99c4003f40 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/2023499673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 msgr2=0x7f99c4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 --2- 172.21.15.70:0/2023499673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99c4004810 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009ae0 tx=0x7f99d001ca50 comp rx=0 tx=0).stop 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e0bc9640 1 -- 172.21.15.70:0/2023499673 reap_dead start 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/2023499673 shutdown_connections 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/2023499673 >> 172.21.15.70:0/2023499673 conn(0x7f99dc0b3700 msgr2=0x7f99dc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/2023499673 shutdown_connections 2024-04-22T01:16:09.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/2023499673 wait complete. 2024-04-22T01:16:09.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 Processor -- start 2024-04-22T01:16:09.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- start start 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99dc1f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0cac00 0x7f99dc1f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99dc0cb080 0x7f99dc1f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc15c250 con 0x7f99c4003f40 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc15c0d0 con 0x7f99dc0cb080 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99dc15c3d0 con 0x7f99dc0cac00 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99dc1f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99dc1f17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38236/0 (socket says 172.21.15.70:38236) 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 -- 172.21.15.70:0/1718491305 learned_addr learned my addr 172.21.15.70:0/1718491305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99db7fe640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0cac00 0x7f99dc1f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e0bc9640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99dc0cb080 0x7f99dc1f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 -- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0cac00 msgr2=0x7f99dc1f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0cac00 0x7f99dc1f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 -- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99dc0cb080 msgr2=0x7f99dc1f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99dc0cb080 0x7f99dc1f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 -- 172.21.15.70:0/1718491305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99dc1f4ae0 con 0x7f99c4003f40 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99dbfff640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99dc1f17b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99cc0118c0 tx=0x7f99cc011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc010040 con 0x7f99c4003f40 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99cc01a070 con 0x7f99c4003f40 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99dc0c9b30 con 0x7f99c4003f40 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00fdc0 con 0x7f99c4003f40 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.323+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99dc0c5bc0 con 0x7f99c4003f40 2024-04-22T01:16:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.327+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99dc068920 con 0x7f99c4003f40 2024-04-22T01:16:09.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.327+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f99cc01f460 con 0x7f99c4003f40 2024-04-22T01:16:09.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.327+0000 7f99d8ff9640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99c80678c0 0x7f99c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:09.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.327+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5619+0+0 (secure 0 0 0) 0x7f99cc089760 con 0x7f99c4003f40 2024-04-22T01:16:09.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.327+0000 7f99db7fe640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99c80678c0 0x7f99c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:09.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.327+0000 7f99db7fe640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99c80678c0 0x7f99c8069d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99d00af000 tx=0x7f99d00046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:09.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.331+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f99cc05a460 con 0x7f99c4003f40 2024-04-22T01:16:09.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:09.487+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f99dc09b430 con 0x7f99c4003f40 2024-04-22T01:16:10.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.155+0000 7f99d8ff9640 1 -- 172.21.15.70:0/1718491305 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) ==== 170+0+0 (secure 0 0 0) 0x7f99cc059d60 con 0x7f99c4003f40 2024-04-22T01:16:10.160 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:16:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99c80678c0 msgr2=0x7f99c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99c80678c0 0x7f99c8069d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99d00af000 tx=0x7f99d00046d0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 msgr2=0x7f99dc1f17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99dc1f17b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99cc0118c0 tx=0x7f99cc011d80 comp rx=0 tx=0).stop 2024-04-22T01:16:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 shutdown_connections 2024-04-22T01:16:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f99c80678c0 0x7f99c8069d80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99dc0cb080 0x7f99dc1f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99dc0cac00 0x7f99dc1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 --2- 172.21.15.70:0/1718491305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99c4003f40 0x7f99dc1f17b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 >> 172.21.15.70:0/1718491305 conn(0x7f99dc0b3700 msgr2=0x7f99dc099d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 shutdown_connections 2024-04-22T01:16:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.159+0000 7f99e262c640 1 -- 172.21.15.70:0/1718491305 wait complete. 2024-04-22T01:16:10.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:16:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 Processor -- start 2024-04-22T01:16:10.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 -- start start 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b415bdc0 0x7f54b40d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54b40d0e80 0x7f54b406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f54b40b5e30 0x7f54b406b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f54b40674c0 con 0x7f54b40d0e80 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f54b40677a0 con 0x7f54b40b5e30 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54bb7db640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f54b40cf930 con 0x7f54b415bdc0 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54b40d0e80 0x7f54b406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54b40d0e80 0x7f54b406b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38242/0 (socket says 172.21.15.70:38242) 2024-04-22T01:16:10.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 -- 172.21.15.70:0/128795756 learned_addr learned my addr 172.21.15.70:0/128795756 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b9577640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b415bdc0 0x7f54b40d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b9d78640 1 --2- 172.21.15.70:0/128795756 >> v2:172.21.15.98:3300/0 conn(0x7f54b40b5e30 0x7f54b406b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 -- 172.21.15.70:0/128795756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b415bdc0 msgr2=0x7f54b40d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b415bdc0 0x7f54b40d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 -- 172.21.15.70:0/128795756 >> v2:172.21.15.98:3300/0 conn(0x7f54b40b5e30 msgr2=0x7f54b406b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 --2- 172.21.15.70:0/128795756 >> v2:172.21.15.98:3300/0 conn(0x7f54b40b5e30 0x7f54b406b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b8d76640 1 -- 172.21.15.70:0/128795756 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b40cfc10 con 0x7f54b40d0e80 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b9577640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b415bdc0 0x7f54b40d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.311+0000 7f54b9d78640 1 --2- 172.21.15.70:0/128795756 >> v2:172.21.15.98:3300/0 conn(0x7f54b40b5e30 0x7f54b406b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54b8d76640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40d0e80 0x7f54b406b1f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f54b409a030 tx=0x7f54a401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d24214e497ee6dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54abfff640 1 -- 172.21.15.70:0/128795756 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a40b2070 con 0x7f54b40d0e80 2024-04-22T01:16:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54abfff640 1 -- 172.21.15.70:0/128795756 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54a40ad680 con 0x7f54b40d0e80 2024-04-22T01:16:10.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54abfff640 1 -- 172.21.15.70:0/128795756 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a40ada50 con 0x7f54b40d0e80 2024-04-22T01:16:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- 172.21.15.70:0/128795756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40d0e80 msgr2=0x7f54b406b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40d0e80 0x7f54b406b1f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f54b409a030 tx=0x7f54a401c460 comp rx=0 tx=0).stop 2024-04-22T01:16:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- 172.21.15.70:0/128795756 shutdown_connections 2024-04-22T01:16:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- 172.21.15.70:0/128795756 >> v2:172.21.15.98:3300/0 conn(0x7f54b40b5e30 0x7f54b406b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40d0e80 0x7f54b406b1f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- 172.21.15.70:0/128795756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b415bdc0 0x7f54b40d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- 172.21.15.70:0/128795756 >> 172.21.15.70:0/128795756 conn(0x7f54b4099a60 msgr2=0x7f54b40b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:10.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- 172.21.15.70:0/128795756 shutdown_connections 2024-04-22T01:16:10.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- 172.21.15.70:0/128795756 wait complete. 2024-04-22T01:16:10.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 Processor -- start 2024-04-22T01:16:10.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- start start 2024-04-22T01:16:10.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 0x7f54b4155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b40d0e80 0x7f54b415a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b415bdc0 0x7f54b415a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54b4167b60 con 0x7f54b40b5e30 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54b41679e0 con 0x7f54b415bdc0 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54bb7db640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f54b4167ce0 con 0x7f54b40d0e80 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54b9577640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 0x7f54b4155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54b9577640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 0x7f54b4155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38258/0 (socket says 172.21.15.70:38258) 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.315+0000 7f54b9577640 1 -- 172.21.15.70:0/4100158284 learned_addr learned my addr 172.21.15.70:0/4100158284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b8d76640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b40d0e80 0x7f54b415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9d78640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b415bdc0 0x7f54b415a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9577640 1 -- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b40d0e80 msgr2=0x7f54b415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9577640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b40d0e80 0x7f54b415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9577640 1 -- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b415bdc0 msgr2=0x7f54b415a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9577640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b415bdc0 0x7f54b415a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9577640 1 -- 172.21.15.70:0/4100158284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b414f350 con 0x7f54b40b5e30 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b9577640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 0x7f54b4155b90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f54b000c7c0 tx=0x7f54b000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0007a30 con 0x7f54b40b5e30 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b0007bd0 con 0x7f54b40b5e30 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b414ed90 con 0x7f54b40b5e30 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00113f0 con 0x7f54b40b5e30 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b40add90 con 0x7f54b40b5e30 2024-04-22T01:16:10.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b4068920 con 0x7f54b40b5e30 2024-04-22T01:16:10.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f54b0011590 con 0x7f54b40b5e30 2024-04-22T01:16:10.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54a9ffb640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54980678c0 0x7f5498069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:10.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6001+0+0 (secure 0 0 0) 0x7f54b0085f60 con 0x7f54b40b5e30 2024-04-22T01:16:10.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b8d76640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54980678c0 0x7f5498069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:10.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.319+0000 7f54b8d76640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54980678c0 0x7f5498069d80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f54a40af000 tx=0x7f54a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:10.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.323+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f54b0015020 con 0x7f54b40b5e30 2024-04-22T01:16:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:10.515+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f54b41595d0 con 0x7f54b40b5e30 2024-04-22T01:16:11.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.183+0000 7f54a9ffb640 1 -- 172.21.15.70:0/4100158284 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f54b00569e0 con 0x7f54b40b5e30 2024-04-22T01:16:11.186 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 5 and data pool 6 2024-04-22T01:16:11.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54980678c0 msgr2=0x7f5498069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54980678c0 0x7f5498069d80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f54a40af000 tx=0x7f54a4008040 comp rx=0 tx=0).stop 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 msgr2=0x7f54b4155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 0x7f54b4155b90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f54b000c7c0 tx=0x7f54b000cc80 comp rx=0 tx=0).stop 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 shutdown_connections 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f54980678c0 0x7f5498069d80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54b415bdc0 0x7f54b415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54b40d0e80 0x7f54b415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 --2- 172.21.15.70:0/4100158284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40b5e30 0x7f54b4155b90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 >> 172.21.15.70:0/4100158284 conn(0x7f54b4099a60 msgr2=0x7f54b40b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 shutdown_connections 2024-04-22T01:16:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.187+0000 7f54bb7db640 1 -- 172.21.15.70:0/4100158284 wait complete. 2024-04-22T01:16:11.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:11.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f9716c10640 1 Processor -- start 2024-04-22T01:16:11.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f9716c10640 1 -- start start 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f9716c10640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f97100ceee0 0x7f97100d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f9716c10640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971015e390 con 0x7f97100cea80 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f9716c10640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9710161e00 con 0x7f971015dfb0 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f9716c10640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f97101620e0 con 0x7f97100ceee0 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f97100cea80 0x7f97100d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33070/0 (socket says 172.21.15.70:33070) 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970ffff640 1 -- 172.21.15.70:0/684934845 learned_addr learned my addr 172.21.15.70:0/684934845 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f97151ad640 1 --2- 172.21.15.70:0/684934845 >> v2:172.21.15.98:3300/0 conn(0x7f97100ceee0 0x7f97100d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052185518 0 0) 0x7f971015e390 con 0x7f97100cea80 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96f8003530 con 0x7f97100cea80 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1978372523 0 0) 0x7f96f8003530 con 0x7f97100cea80 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 >> v2:172.21.15.98:3300/0 conn(0x7f97100ceee0 msgr2=0x7f97100d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 --2- 172.21.15.70:0/684934845 >> v2:172.21.15.98:3300/0 conn(0x7f97100ceee0 0x7f97100d2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 >> v1:172.21.15.98:6789/0 conn(0x7f971015dfb0 legacy=0x7f97100b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97101623c0 con 0x7f97100cea80 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f9704001680 con 0x7f97100cea80 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f97151ad640 1 --2- 172.21.15.70:0/684934845 >> v2:172.21.15.98:3300/0 conn(0x7f97100ceee0 0x7f97100d2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 >> v1:172.21.15.70:6789/0 conn(0x7f97100cea80 legacy=0x7f97100d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f80039c0 0x7f96f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8004810 0x7f96f8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80051d0 0x7f96f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97101620e0 con 0x7f96f8004810 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96f800c120 con 0x7f96f80039c0 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f96f800c320 con 0x7f96f80051d0 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.339+0000 7f970ffff640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8004810 0x7f96f8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f97149ac640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f80039c0 0x7f96f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970ffff640 1 -- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80051d0 msgr2=0x7f96f800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970ffff640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80051d0 0x7f96f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970ffff640 1 -- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f80039c0 msgr2=0x7f96f8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970ffff640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f80039c0 0x7f96f8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970ffff640 1 -- 172.21.15.70:0/684934845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f8003530 con 0x7f96f8004810 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f97149ac640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f80039c0 0x7f96f8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f97151ad640 1 -- 172.21.15.70:0/684934845 reap_dead start 2024-04-22T01:16:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970ffff640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8004810 0x7f96f8004c90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f970400c210 tx=0x7f970401f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d5f5814a7cb4a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704007d40 con 0x7f96f8004810 2024-04-22T01:16:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704005b00 con 0x7f96f8004810 2024-04-22T01:16:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f970f7fe640 1 -- 172.21.15.70:0/684934845 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704007070 con 0x7f96f8004810 2024-04-22T01:16:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 -- 172.21.15.70:0/684934845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8004810 msgr2=0x7f96f8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8004810 0x7f96f8004c90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f970400c210 tx=0x7f970401f3f0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 -- 172.21.15.70:0/684934845 shutdown_connections 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 --2- 172.21.15.70:0/684934845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8004810 0x7f96f8004c90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 -- 172.21.15.70:0/684934845 >> 172.21.15.70:0/684934845 conn(0x7f971006c9b0 msgr2=0x7f971006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 -- 172.21.15.70:0/684934845 shutdown_connections 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.343+0000 7f9716c10640 1 -- 172.21.15.70:0/684934845 wait complete. 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 Processor -- start 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- start start 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8004810 0x7f97100cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97100d3350 0x7f97100cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 0x7f97101f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9710161a90 con 0x7f97101f6a00 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97100cf370 con 0x7f96f8004810 2024-04-22T01:16:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f97100b95d0 con 0x7f97100d3350 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 0x7f97101f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 0x7f97101f6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38278/0 (socket says 172.21.15.70:38278) 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 -- 172.21.15.70:0/3185201056 learned_addr learned my addr 172.21.15.70:0/3185201056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970ffff640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97100d3350 0x7f97100cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97149ac640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8004810 0x7f97100cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 -- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97100d3350 msgr2=0x7f97100cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97100d3350 0x7f97100cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 -- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8004810 msgr2=0x7f97100cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8004810 0x7f97100cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 -- 172.21.15.70:0/3185201056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971015b690 con 0x7f97101f6a00 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97151ad640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 0x7f97101f6e80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f97100ce2b0 tx=0x7f970800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970801a070 con 0x7f97101f6a00 2024-04-22T01:16:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f970800dc50 con 0x7f97101f6a00 2024-04-22T01:16:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97101f5c40 con 0x7f97101f6a00 2024-04-22T01:16:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97080039a0 con 0x7f97101f6a00 2024-04-22T01:16:11.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97101f6200 con 0x7f97101f6a00 2024-04-22T01:16:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97100d3730 con 0x7f97101f6a00 2024-04-22T01:16:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f970801d070 con 0x7f97101f6a00 2024-04-22T01:16:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970d7fa640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96fc0678c0 0x7f96fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f9708085e30 con 0x7f97101f6a00 2024-04-22T01:16:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.347+0000 7f97149ac640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96fc0678c0 0x7f96fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.351+0000 7f97149ac640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96fc0678c0 0x7f96fc069d80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9700000f70 tx=0x7f9700009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.355+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f97100d3730 con 0x7f97101f6a00 2024-04-22T01:16:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.511+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9710068210 con 0x7f97101f6a00 2024-04-22T01:16:11.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.515+0000 7f970d7fa640 1 -- 172.21.15.70:0/3185201056 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16919 (secure 0 0 0) 0x7f971015b690 con 0x7f97101f6a00 2024-04-22T01:16:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:11.520 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:10.523681+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96fc0678c0 msgr2=0x7f96fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96fc0678c0 0x7f96fc069d80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9700000f70 tx=0x7f9700009040 comp rx=0 tx=0).stop 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 msgr2=0x7f97101f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 0x7f97101f6e80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f97100ce2b0 tx=0x7f970800d4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 shutdown_connections 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96fc0678c0 0x7f96fc069d80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97101f6a00 0x7f97101f6e80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97100d3350 0x7f97100cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 --2- 172.21.15.70:0/3185201056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8004810 0x7f97100cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 >> 172.21.15.70:0/3185201056 conn(0x7f971006c9b0 msgr2=0x7f971015be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 shutdown_connections 2024-04-22T01:16:11.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.519+0000 7f9716c10640 1 -- 172.21.15.70:0/3185201056 wait complete. 2024-04-22T01:16:11.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.651+0000 7efe2737f640 1 Processor -- start 2024-04-22T01:16:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.651+0000 7efe2737f640 1 -- start start 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2737f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe200b3cd0 0x7efe20072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2737f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe2015bf20 con 0x7efe2015bb40 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2737f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe200aec30 con 0x7efe2006ed40 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2737f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efe2006e8b0 con 0x7efe200b3cd0 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2511b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efe2015bb40 0x7efe200b35c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33082/0 (socket says 172.21.15.70:33082) 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2511b640 1 -- 172.21.15.70:0/2743736962 learned_addr learned my addr 172.21.15.70:0/2743736962 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2491a640 1 --2- 172.21.15.70:0/2743736962 >> v2:172.21.15.98:3300/0 conn(0x7efe200b3cd0 0x7efe20072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762592148 0 0) 0x7efe2015bf20 con 0x7efe2015bb40 2024-04-22T01:16:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0c003530 con 0x7efe2015bb40 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1959926668 0 0) 0x7efe0c003530 con 0x7efe2015bb40 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 >> v2:172.21.15.98:3300/0 conn(0x7efe200b3cd0 msgr2=0x7efe20072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 --2- 172.21.15.70:0/2743736962 >> v2:172.21.15.98:3300/0 conn(0x7efe200b3cd0 0x7efe20072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 >> v1:172.21.15.98:6789/0 conn(0x7efe2006ed40 legacy=0x7efe20161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe200aef10 con 0x7efe2015bb40 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7efe1c001680 con 0x7efe2015bb40 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 >> v1:172.21.15.70:6789/0 conn(0x7efe2015bb40 legacy=0x7efe200b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c0039c0 0x7efe0c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c0043c0 0x7efe0c004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c005090 0x7efe0c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe2006e8b0 con 0x7efe0c005090 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c00bff0 con 0x7efe0c0043c0 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efe0c00c1f0 con 0x7efe0c0039c0 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2591c640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c005090 0x7efe0c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2491a640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c0043c0 0x7efe0c004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2511b640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c0039c0 0x7efe0c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2591c640 1 -- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c0039c0 msgr2=0x7efe0c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2591c640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c0039c0 0x7efe0c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2591c640 1 -- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c0043c0 msgr2=0x7efe0c004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2591c640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c0043c0 0x7efe0c004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.655+0000 7efe2591c640 1 -- 172.21.15.70:0/2743736962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c003530 con 0x7efe0c005090 2024-04-22T01:16:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 -- 172.21.15.70:0/2743736962 reap_dead start 2024-04-22T01:16:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c005090 0x7efe0c00b9d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7efe18001f60 tx=0x7efe1801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860d6f0a1fb68bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18001b60 con 0x7efe0c005090 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe18003b80 con 0x7efe0c005090 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe17fff640 1 -- 172.21.15.70:0/2743736962 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe180050d0 con 0x7efe0c005090 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/2743736962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c005090 msgr2=0x7efe0c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c005090 0x7efe0c00b9d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7efe18001f60 tx=0x7efe1801c880 comp rx=0 tx=0).stop 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/2743736962 shutdown_connections 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 --2- 172.21.15.70:0/2743736962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c005090 0x7efe0c00b9d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/2743736962 >> 172.21.15.70:0/2743736962 conn(0x7efe2006b1f0 msgr2=0x7efe2006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/2743736962 shutdown_connections 2024-04-22T01:16:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/2743736962 wait complete. 2024-04-22T01:16:11.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 Processor -- start 2024-04-22T01:16:11.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- start start 2024-04-22T01:16:11.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c005090 0x7efe2014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe2015f160 0x7efe2015f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 0x7efe201598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe2006f5c0 con 0x7efe20159450 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe2006e540 con 0x7efe2015f160 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efe2006f740 con 0x7efe0c005090 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 0x7efe201598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 0x7efe201598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38292/0 (socket says 172.21.15.70:38292) 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 -- 172.21.15.70:0/3371459796 learned_addr learned my addr 172.21.15.70:0/3371459796 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2511b640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c005090 0x7efe2014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2491a640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe2015f160 0x7efe2015f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 -- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c005090 msgr2=0x7efe2014a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c005090 0x7efe2014a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 -- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe2015f160 msgr2=0x7efe2015f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe2015f160 0x7efe2015f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 -- 172.21.15.70:0/3371459796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe2015a1a0 con 0x7efe20159450 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2511b640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c005090 0x7efe2014a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2591c640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 0x7efe201598d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7efe1801d120 tx=0x7efe0c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18001b40 con 0x7efe20159450 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe18006d70 con 0x7efe20159450 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe2015a480 con 0x7efe20159450 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe180a6690 con 0x7efe20159450 2024-04-22T01:16:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.659+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe2015aa40 con 0x7efe20159450 2024-04-22T01:16:11.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.663+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe20068210 con 0x7efe20159450 2024-04-22T01:16:11.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.663+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efe180a6830 con 0x7efe20159450 2024-04-22T01:16:11.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.663+0000 7efe15ffb640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efe040678c0 0x7efe04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.663+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7efe1811f470 con 0x7efe20159450 2024-04-22T01:16:11.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.663+0000 7efe2511b640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efe040678c0 0x7efe04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.663+0000 7efe2511b640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efe040678c0 0x7efe04069d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe1c004880 tx=0x7efe1c004810 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.667+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7efe18016590 con 0x7efe20159450 2024-04-22T01:16:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.831+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe2006e040 con 0x7efe20159450 2024-04-22T01:16:11.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.831+0000 7efe15ffb640 1 -- 172.21.15.70:0/3371459796 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16919 (secure 0 0 0) 0x7efe180efe20 con 0x7efe20159450 2024-04-22T01:16:11.837 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:11.837 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:10.523681+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:11.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efe040678c0 msgr2=0x7efe04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efe040678c0 0x7efe04069d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe1c004880 tx=0x7efe1c004810 comp rx=0 tx=0).stop 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 msgr2=0x7efe201598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 0x7efe201598d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7efe1801d120 tx=0x7efe0c005840 comp rx=0 tx=0).stop 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 shutdown_connections 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efe040678c0 0x7efe04069d80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe20159450 0x7efe201598d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe2015f160 0x7efe2015f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 --2- 172.21.15.70:0/3371459796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe0c005090 0x7efe2014a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 >> 172.21.15.70:0/3371459796 conn(0x7efe2006b1f0 msgr2=0x7efe200ad030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 shutdown_connections 2024-04-22T01:16:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.835+0000 7efe2737f640 1 -- 172.21.15.70:0/3371459796 wait complete. 2024-04-22T01:16:11.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:16:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.975+0000 7fb47102c640 1 Processor -- start 2024-04-22T01:16:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.975+0000 7fb47102c640 1 -- start start 2024-04-22T01:16:11.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb47102c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb46c15def0 0x7fb46c0cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb47102c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb46c0d08c0 0x7fb46c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb47102c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb46c15e2d0 con 0x7fb46c0cfce0 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb47102c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb46c15e570 con 0x7fb46c0d08c0 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb47102c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb46c15e810 con 0x7fb46c15def0 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb46c0cfce0 0x7fb46c0d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33086/0 (socket says 172.21.15.70:33086) 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46a59c640 1 -- 172.21.15.70:0/3278277347 learned_addr learned my addr 172.21.15.70:0/3278277347 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46b59e640 1 --2- 172.21.15.70:0/3278277347 >> v2:172.21.15.70:3300/0 conn(0x7fb46c0d08c0 0x7fb46c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46ad9d640 1 --2- 172.21.15.70:0/3278277347 >> v2:172.21.15.98:3300/0 conn(0x7fb46c15def0 0x7fb46c0cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46b59e640 1 -- 172.21.15.70:0/3278277347 >> v2:172.21.15.98:3300/0 conn(0x7fb46c15def0 msgr2=0x7fb46c0cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46b59e640 1 --2- 172.21.15.70:0/3278277347 >> v2:172.21.15.98:3300/0 conn(0x7fb46c15def0 0x7fb46c0cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46b59e640 1 -- 172.21.15.70:0/3278277347 >> v1:172.21.15.70:6789/0 conn(0x7fb46c0cfce0 legacy=0x7fb46c0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46b59e640 1 -- 172.21.15.70:0/3278277347 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46c15eaf0 con 0x7fb46c0d08c0 2024-04-22T01:16:11.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb469d9b640 1 -- 172.21.15.70:0/3278277347 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931396007 0 0) 0x7fb46c15e2d0 con 0x7fb46c0cfce0 2024-04-22T01:16:11.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb46b59e640 1 --2- 172.21.15.70:0/3278277347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c0d08c0 0x7fb46c06d480 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0097c0 tx=0x7fb45c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c17fdd7e5d64d80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb469d9b640 1 -- 172.21.15.70:0/3278277347 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0ab430 con 0x7fb46c0d08c0 2024-04-22T01:16:11.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb469d9b640 1 -- 172.21.15.70:0/3278277347 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45c0b0070 con 0x7fb46c0d08c0 2024-04-22T01:16:11.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.979+0000 7fb469d9b640 1 -- 172.21.15.70:0/3278277347 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0ab7f0 con 0x7fb46c0d08c0 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/3278277347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c0d08c0 msgr2=0x7fb46c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 --2- 172.21.15.70:0/3278277347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c0d08c0 0x7fb46c06d480 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0097c0 tx=0x7fb45c01c430 comp rx=0 tx=0).stop 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/3278277347 shutdown_connections 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 --2- 172.21.15.70:0/3278277347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c0d08c0 0x7fb46c06d480 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 --2- 172.21.15.70:0/3278277347 >> v2:172.21.15.98:3300/0 conn(0x7fb46c15def0 0x7fb46c0cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/3278277347 >> 172.21.15.70:0/3278277347 conn(0x7fb46c0b36c0 msgr2=0x7fb46c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/3278277347 shutdown_connections 2024-04-22T01:16:11.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/3278277347 wait complete. 2024-04-22T01:16:11.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 Processor -- start 2024-04-22T01:16:11.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- start start 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb46c0cfce0 0x7fb46c14f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb46c0d08c0 0x7fb46c0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 0x7fb46c0b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c0ccc10 con 0x7fb46c15def0 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c0cca90 con 0x7fb46c0d08c0 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb46c0ccd90 con 0x7fb46c0cfce0 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 0x7fb46c0b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 0x7fb46c0b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38302/0 (socket says 172.21.15.70:38302) 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 -- 172.21.15.70:0/4155227512 learned_addr learned my addr 172.21.15.70:0/4155227512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46ad9d640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb46c0cfce0 0x7fb46c14f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46a59c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb46c0d08c0 0x7fb46c0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 -- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb46c0cfce0 msgr2=0x7fb46c14f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb46c0cfce0 0x7fb46c14f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 -- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb46c0d08c0 msgr2=0x7fb46c0b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb46c0d08c0 0x7fb46c0b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 -- 172.21.15.70:0/4155227512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46c070ce0 con 0x7fb46c15def0 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb46b59e640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 0x7fb46c0b2d90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0ad040 tx=0x7fb45c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c006e20 con 0x7fb46c15def0 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45c004230 con 0x7fb46c15def0 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46c14fa50 con 0x7fb46c15def0 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c003620 con 0x7fb46c15def0 2024-04-22T01:16:11.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.983+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb46c15a700 con 0x7fb46c15def0 2024-04-22T01:16:11.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.987+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb46c068920 con 0x7fb46c15def0 2024-04-22T01:16:11.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.987+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb45c0037c0 con 0x7fb46c15def0 2024-04-22T01:16:11.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.987+0000 7fb4637fe640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb44c0678c0 0x7fb44c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:11.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.987+0000 7fb46ad9d640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb44c0678c0 0x7fb44c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:11.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.987+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7fb45c0a8030 con 0x7fb46c15def0 2024-04-22T01:16:11.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.987+0000 7fb46ad9d640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb44c0678c0 0x7fb44c069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb464003fd0 tx=0x7fb4640039a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:11.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:11.991+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb45c0c4060 con 0x7fb46c15def0 2024-04-22T01:16:12.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:12.183+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb46c15a0d0 con 0x7fb46c15def0 2024-04-22T01:16:12.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:16:12.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:16:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.187+0000 7fb4637fe640 1 -- 172.21.15.70:0/4155227512 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7fb45c016050 con 0x7fb46c15def0 2024-04-22T01:16:13.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb44c0678c0 msgr2=0x7fb44c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb44c0678c0 0x7fb44c069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb464003fd0 tx=0x7fb4640039a0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 msgr2=0x7fb46c0b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 0x7fb46c0b2d90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0ad040 tx=0x7fb45c01ac70 comp rx=0 tx=0).stop 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 shutdown_connections 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb44c0678c0 0x7fb44c069d80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb46c15def0 0x7fb46c0b2d90 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb46c0d08c0 0x7fb46c0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 --2- 172.21.15.70:0/4155227512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb46c0cfce0 0x7fb46c14f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 >> 172.21.15.70:0/4155227512 conn(0x7fb46c0b36c0 msgr2=0x7fb46c0713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 shutdown_connections 2024-04-22T01:16:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.191+0000 7fb47102c640 1 -- 172.21.15.70:0/4155227512 wait complete. 2024-04-22T01:16:13.214 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-22T01:16:13.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 Processor -- start 2024-04-22T01:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 -- start start 2024-04-22T01:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fce180b4450 0x7fce18071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fce180720b0 0x7fce180ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce1815c290 con 0x7fce180cd050 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fce180b3a50 con 0x7fce180b4450 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce2062e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fce1815c570 con 0x7fce180720b0 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1e3ca640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fce180b4450 0x7fce18071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1ebcb640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fce180cd050 0x7fce1815f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33096/0 (socket says 172.21.15.70:33096) 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1e3ca640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fce180b4450 0x7fce18071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38306/0 (socket says 172.21.15.70:38306) 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1ebcb640 1 -- 172.21.15.70:0/3942671508 learned_addr learned my addr 172.21.15.70:0/3942671508 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1e3ca640 1 -- 172.21.15.70:0/3942671508 >> v2:172.21.15.98:3300/0 conn(0x7fce180720b0 msgr2=0x7fce180ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1dbc9640 1 --2- 172.21.15.70:0/3942671508 >> v2:172.21.15.98:3300/0 conn(0x7fce180720b0 0x7fce180ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1d3c8640 1 -- 172.21.15.70:0/3942671508 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007229052 0 0) 0x7fce1815c290 con 0x7fce180cd050 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1e3ca640 1 --2- 172.21.15.70:0/3942671508 >> v2:172.21.15.98:3300/0 conn(0x7fce180720b0 0x7fce180ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1e3ca640 1 -- 172.21.15.70:0/3942671508 >> v1:172.21.15.70:6789/0 conn(0x7fce180cd050 legacy=0x7fce1815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1e3ca640 1 -- 172.21.15.70:0/3942671508 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1815c850 con 0x7fce180b4450 2024-04-22T01:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.339+0000 7fce1dbc9640 1 --2- 172.21.15.70:0/3942671508 >> v2:172.21.15.98:3300/0 conn(0x7fce180720b0 0x7fce180ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1e3ca640 1 --2- 172.21.15.70:0/3942671508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce18071b70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fce180cf9a0 tx=0x7fce1401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83edb05dc3f72665 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1d3c8640 1 -- 172.21.15.70:0/3942671508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce140b0070 con 0x7fce180b4450 2024-04-22T01:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1d3c8640 1 -- 172.21.15.70:0/3942671508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce140ab600 con 0x7fce180b4450 2024-04-22T01:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1d3c8640 1 -- 172.21.15.70:0/3942671508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce140ab9d0 con 0x7fce180b4450 2024-04-22T01:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- 172.21.15.70:0/3942671508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 msgr2=0x7fce18071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 --2- 172.21.15.70:0/3942671508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce18071b70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fce180cf9a0 tx=0x7fce1401c480 comp rx=0 tx=0).stop 2024-04-22T01:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- 172.21.15.70:0/3942671508 shutdown_connections 2024-04-22T01:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 --2- 172.21.15.70:0/3942671508 >> v2:172.21.15.98:3300/0 conn(0x7fce180720b0 0x7fce180ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 --2- 172.21.15.70:0/3942671508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce18071b70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- 172.21.15.70:0/3942671508 >> 172.21.15.70:0/3942671508 conn(0x7fce180b1d40 msgr2=0x7fce180cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- 172.21.15.70:0/3942671508 shutdown_connections 2024-04-22T01:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- 172.21.15.70:0/3942671508 wait complete. 2024-04-22T01:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 Processor -- start 2024-04-22T01:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- start start 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce180720b0 0x7fce18149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce181474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce180cd050 0x7fce1814a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fce180b49b0 con 0x7fce180b4450 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce180b4830 con 0x7fce180cd050 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce2062e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fce180cd800 con 0x7fce180720b0 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1dbc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce181474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1dbc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce181474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38314/0 (socket says 172.21.15.70:38314) 2024-04-22T01:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.343+0000 7fce1dbc9640 1 -- 172.21.15.70:0/2597944735 learned_addr learned my addr 172.21.15.70:0/2597944735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1dbc9640 1 -- 172.21.15.70:0/2597944735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce180720b0 msgr2=0x7fce18149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1dbc9640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce180720b0 0x7fce18149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1dbc9640 1 -- 172.21.15.70:0/2597944735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce180cd050 msgr2=0x7fce1814a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1dbc9640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce180cd050 0x7fce1814a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1dbc9640 1 -- 172.21.15.70:0/2597944735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1814b3e0 con 0x7fce180b4450 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1dbc9640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce181474d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fce0800c7f0 tx=0x7fce0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08015070 con 0x7fce180b4450 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce08007820 con 0x7fce180b4450 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1814deb0 con 0x7fce180b4450 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08007c10 con 0x7fce180b4450 2024-04-22T01:16:13.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1814e470 con 0x7fce180b4450 2024-04-22T01:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce18068210 con 0x7fce180b4450 2024-04-22T01:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce08018070 con 0x7fce180b4450 2024-04-22T01:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce0effd640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcdfc0678c0 0x7fcdfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fce08085f30 con 0x7fce180b4450 2024-04-22T01:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1e3ca640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcdfc0678c0 0x7fcdfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.347+0000 7fce1e3ca640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcdfc0678c0 0x7fcdfc069d80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fce140ad000 tx=0x7fce14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:13.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.351+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fce08056970 con 0x7fce180b4450 2024-04-22T01:16:13.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:13.531+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fce180d01e0 con 0x7fce180b4450 2024-04-22T01:16:14.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.235+0000 7fce0effd640 1 -- 172.21.15.70:0/2597944735 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7fce08056270 con 0x7fce180b4450 2024-04-22T01:16:14.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcdfc0678c0 msgr2=0x7fcdfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcdfc0678c0 0x7fcdfc069d80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fce140ad000 tx=0x7fce14008040 comp rx=0 tx=0).stop 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 msgr2=0x7fce181474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce181474d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fce0800c7f0 tx=0x7fce0800ccb0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 shutdown_connections 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcdfc0678c0 0x7fcdfc069d80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce180cd050 0x7fce1814a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce180b4450 0x7fce181474d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 --2- 172.21.15.70:0/2597944735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce180720b0 0x7fce18149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 >> 172.21.15.70:0/2597944735 conn(0x7fce180b1d40 msgr2=0x7fce1815e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 shutdown_connections 2024-04-22T01:16:14.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.239+0000 7fce2062e640 1 -- 172.21.15.70:0/2597944735 wait complete. 2024-04-22T01:16:14.263 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:16:14.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9c1dbd640 1 Processor -- start 2024-04-22T01:16:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9c1dbd640 1 -- start start 2024-04-22T01:16:14.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9c1dbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 0x7ff9bc099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9c1dbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9bc15c220 con 0x7ff9bc066f50 2024-04-22T01:16:14.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9c1dbd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9bc0674a0 con 0x7ff9bc0ad880 2024-04-22T01:16:14.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9c1dbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc15c570 con 0x7ff9bc15be40 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9b3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff9bc066f50 0x7ff9bc0ad170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33112/0 (socket says 172.21.15.70:33112) 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9b3fff640 1 -- 172.21.15.70:0/2718649283 learned_addr learned my addr 172.21.15.70:0/2718649283 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 0x7ff9bc099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993411178 0 0) 0x7ff9bc15c220 con 0x7ff9bc066f50 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9a8003530 con 0x7ff9bc066f50 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2603033009 0 0) 0x7ff9a8003530 con 0x7ff9bc066f50 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 msgr2=0x7ff9bc099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 0x7ff9bc099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 >> v1:172.21.15.98:6789/0 conn(0x7ff9bc0ad880 legacy=0x7ff9bc15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc06c4b0 con 0x7ff9bc066f50 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7ff9a4001680 con 0x7ff9bc066f50 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 >> v1:172.21.15.70:6789/0 conn(0x7ff9bc066f50 legacy=0x7ff9bc0ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a80039f0 0x7ff9a8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 0x7ff9a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9a8004d50 0x7ff9a80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc15c570 con 0x7ff9a80039f0 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a800ba80 con 0x7ff9a8004d50 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a800bc80 con 0x7ff9bc15be40 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a80039f0 0x7ff9a8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9b3fff640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 0x7ff9a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bbfff640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9a8004d50 0x7ff9a80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 -- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 msgr2=0x7ff9a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9bc15be40 0x7ff9a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 -- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9a8004d50 msgr2=0x7ff9a80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9a8004d50 0x7ff9a80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.383+0000 7ff9bb7fe640 1 -- 172.21.15.70:0/2718649283 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a8003530 con 0x7ff9a80039f0 2024-04-22T01:16:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a80039f0 0x7ff9a8003e30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac009ab0 tx=0x7ff9ac01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5deafd03e5679617 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0096b0 con 0x7ff9a80039f0 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ac0a6510 con 0x7ff9a80039f0 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9baffd640 1 -- 172.21.15.70:0/2718649283 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0a6900 con 0x7ff9a80039f0 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/2718649283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a80039f0 msgr2=0x7ff9a8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/2718649283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a80039f0 0x7ff9a8003e30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac009ab0 tx=0x7ff9ac01ca20 comp rx=0 tx=0).stop 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9bbfff640 1 -- 172.21.15.70:0/2718649283 reap_dead start 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/2718649283 shutdown_connections 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/2718649283 >> 172.21.15.70:0/2718649283 conn(0x7ff9bc0b3d90 msgr2=0x7ff9bc0b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/2718649283 shutdown_connections 2024-04-22T01:16:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/2718649283 wait complete. 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 Processor -- start 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- start start 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9a80039f0 0x7ff9bc14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc14fd90 0x7ff9bc154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 0x7ff9bc154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.387+0000 7ff9c1dbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc15daa0 con 0x7ff9bc145e50 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9c1dbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc15d920 con 0x7ff9bc14fd90 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9c1dbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc15dc20 con 0x7ff9a80039f0 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 0x7ff9bc154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 0x7ff9bc154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38328/0 (socket says 172.21.15.70:38328) 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 -- 172.21.15.70:0/1999200238 learned_addr learned my addr 172.21.15.70:0/1999200238 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9a80039f0 0x7ff9bc14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b3fff640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc14fd90 0x7ff9bc154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 -- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9a80039f0 msgr2=0x7ff9bc14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9a80039f0 0x7ff9bc14f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 -- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc14fd90 msgr2=0x7ff9bc154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc14fd90 0x7ff9bc154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 -- 172.21.15.70:0/1999200238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc1513b0 con 0x7ff9bc145e50 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b3fff640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc14fd90 0x7ff9bc154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bbfff640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 0x7ff9bc154ba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e780 tx=0x7ff9b400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4003980 con 0x7ff9bc145e50 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b4018070 con 0x7ff9bc145e50 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4003da0 con 0x7ff9bc145e50 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc150310 con 0x7ff9bc145e50 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc1508d0 con 0x7ff9bc145e50 2024-04-22T01:16:14.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc06ccb0 con 0x7ff9bc145e50 2024-04-22T01:16:14.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff9b40050d0 con 0x7ff9bc145e50 2024-04-22T01:16:14.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b8ff9640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff99c0678c0 0x7ff99c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7ff9b4085a70 con 0x7ff9bc145e50 2024-04-22T01:16:14.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff99c0678c0 0x7ff99c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.391+0000 7ff9bb7fe640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff99c0678c0 0x7ff99c069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0af000 tx=0x7ff9ac0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:14.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.395+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff9b4056420 con 0x7ff9bc145e50 2024-04-22T01:16:14.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9bc0b4810 con 0x7ff9bc145e50 2024-04-22T01:16:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9b8ff9640 1 -- 172.21.15.70:0/1999200238 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5178 (secure 0 0 0) 0x7ff9b401be40 con 0x7ff9bc145e50 2024-04-22T01:16:14.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:14.595 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:10.524071+0000","modified":"2024-04-22T01:16:13.536979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5478":{"gid":5478,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3248824947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3248824947},{"type":"v1","addr":"172.21.15.70:6837","nonce":3248824947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5478,"qdb_cluster":[5478]},"id":2}]} 2024-04-22T01:16:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-22T01:16:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff99c0678c0 msgr2=0x7ff99c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff99c0678c0 0x7ff99c069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0af000 tx=0x7ff9ac0046d0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 msgr2=0x7ff9bc154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 0x7ff9bc154ba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e780 tx=0x7ff9b400ec40 comp rx=0 tx=0).stop 2024-04-22T01:16:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 shutdown_connections 2024-04-22T01:16:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff99c0678c0 0x7ff99c069d80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc145e50 0x7ff9bc154ba0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9bc14fd90 0x7ff9bc154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 --2- 172.21.15.70:0/1999200238 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9a80039f0 0x7ff9bc14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 >> 172.21.15.70:0/1999200238 conn(0x7ff9bc0b3d90 msgr2=0x7ff9bc15f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:14.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 shutdown_connections 2024-04-22T01:16:14.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.591+0000 7ff9c1dbd640 1 -- 172.21.15.70:0/1999200238 wait complete. 2024-04-22T01:16:14.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 Processor -- start 2024-04-22T01:16:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 -- start start 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e541455b0 0x7f1e540cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 0x7f1e5406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e540d0250 con 0x7f1e540cc150 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1e540cd7b0 con 0x7f1e541455b0 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5c0ae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1e540cc710 con 0x7f1e54072230 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e59649640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1e540cc150 0x7f1e54071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33128/0 (socket says 172.21.15.70:33128) 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e59649640 1 -- 172.21.15.70:0/2708796473 learned_addr learned my addr 172.21.15.70:0/2708796473 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 0x7f1e5406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e59e4a640 1 --2- 172.21.15.70:0/2708796473 >> v2:172.21.15.98:3300/0 conn(0x7f1e541455b0 0x7f1e540cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375422360 0 0) 0x7f1e540d0250 con 0x7f1e540cc150 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e40003530 con 0x7f1e540cc150 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2068969577 0 0) 0x7f1e40003530 con 0x7f1e540cc150 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 msgr2=0x7f1e5406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 0x7f1e5406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 >> v2:172.21.15.98:3300/0 conn(0x7f1e541455b0 msgr2=0x7f1e540cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 --2- 172.21.15.70:0/2708796473 >> v2:172.21.15.98:3300/0 conn(0x7f1e541455b0 0x7f1e540cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e540cc9f0 con 0x7f1e540cc150 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1e440016a0 con 0x7f1e540cc150 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e59e4a640 1 --2- 172.21.15.70:0/2708796473 >> v2:172.21.15.98:3300/0 conn(0x7f1e541455b0 0x7f1e540cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 >> v1:172.21.15.70:6789/0 conn(0x7f1e540cc150 legacy=0x7f1e54071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 0x7f1e400039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e40003f40 0x7f1e40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e40004d50 0x7f1e400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e540cd7b0 con 0x7f1e40004d50 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e540cc710 con 0x7f1e40003f40 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.731+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4000bb60 con 0x7f1e54072230 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e40004d50 0x7f1e400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e59e4a640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 0x7f1e400039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e59649640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e40003f40 0x7f1e40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 -- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 msgr2=0x7f1e400039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e54072230 0x7f1e400039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 -- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e40003f40 msgr2=0x7f1e40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e40003f40 0x7f1e40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 -- 172.21.15.70:0/2708796473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e40003530 con 0x7f1e40004d50 2024-04-22T01:16:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e59649640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e40003f40 0x7f1e40004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e40004d50 0x7f1e400051d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009f90 tx=0x7f1e4c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684a523b2d91a946 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0b2070 con 0x7f1e40004d50 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e4c0a5a30 con 0x7f1e40004d50 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e58e48640 1 -- 172.21.15.70:0/2708796473 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c009c10 con 0x7f1e40004d50 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/2708796473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e40004d50 msgr2=0x7f1e400051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/2708796473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e40004d50 0x7f1e400051d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009f90 tx=0x7f1e4c01cc40 comp rx=0 tx=0).stop 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5a64b640 1 -- 172.21.15.70:0/2708796473 reap_dead start 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/2708796473 shutdown_connections 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/2708796473 >> 172.21.15.70:0/2708796473 conn(0x7f1e5406b1f0 msgr2=0x7f1e5406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/2708796473 shutdown_connections 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.735+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/2708796473 wait complete. 2024-04-22T01:16:14.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 Processor -- start 2024-04-22T01:16:14.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- start start 2024-04-22T01:16:14.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e5413da40 0x7f1e540b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e540afe70 0x7f1e540b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 0x7f1e54143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5406eda0 con 0x7f1e54143aa0 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5406ec20 con 0x7f1e5413da40 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5406ef20 con 0x7f1e540afe70 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 0x7f1e54143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 0x7f1e54143f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38344/0 (socket says 172.21.15.70:38344) 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 -- 172.21.15.70:0/3167793209 learned_addr learned my addr 172.21.15.70:0/3167793209 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e59e4a640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e5413da40 0x7f1e540b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 -- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e540afe70 msgr2=0x7f1e540b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e540afe70 0x7f1e540b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 -- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e5413da40 msgr2=0x7f1e540b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e5413da40 0x7f1e540b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 -- 172.21.15.70:0/3167793209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e540b0930 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5a64b640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 0x7f1e54143f50 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009f90 tx=0x7f1e4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0bb070 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e4c0a5830 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e540b0bd0 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0b2040 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e540b1190 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e54068210 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1e4c0be070 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e4a7fc640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1e380678f0 0x7f1e38069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f1e4c11f210 con 0x7f1e54143aa0 2024-04-22T01:16:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.739+0000 7f1e59e4a640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1e380678f0 0x7f1e38069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:14.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.743+0000 7f1e59e4a640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1e380678f0 0x7f1e38069db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e54144780 tx=0x7f1e50008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:14.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.743+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1e4c016950 con 0x7f1e54143aa0 2024-04-22T01:16:14.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.895+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e5406bac0 con 0x7f1e54143aa0 2024-04-22T01:16:14.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e4a7fc640 1 -- 172.21.15.70:0/3167793209 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16906 (secure 0 0 0) 0x7f1e4c0ef600 con 0x7f1e54143aa0 2024-04-22T01:16:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:14.899 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:12.187805+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1e380678f0 msgr2=0x7f1e38069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1e380678f0 0x7f1e38069db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e54144780 tx=0x7f1e50008040 comp rx=0 tx=0).stop 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 msgr2=0x7f1e54143f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 0x7f1e54143f50 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009f90 tx=0x7f1e4c009750 comp rx=0 tx=0).stop 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 shutdown_connections 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1e380678f0 0x7f1e38069db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e54143aa0 0x7f1e54143f50 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e540afe70 0x7f1e540b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 --2- 172.21.15.70:0/3167793209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e5413da40 0x7f1e540b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 >> 172.21.15.70:0/3167793209 conn(0x7f1e5406b1f0 msgr2=0x7f1e540cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 shutdown_connections 2024-04-22T01:16:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:14.899+0000 7f1e5c0ae640 1 -- 172.21.15.70:0/3167793209 wait complete. 2024-04-22T01:16:14.917 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:16:14.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:16:15.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.031+0000 7f6010a3c640 1 Processor -- start 2024-04-22T01:16:15.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.031+0000 7f6010a3c640 1 -- start start 2024-04-22T01:16:15.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6010a3c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f600c15aa30 0x7f600c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6010a3c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 0x7f600c0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6010a3c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c06dbd0 con 0x7f600c15bb50 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6010a3c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f600c06de70 con 0x7f600c15aa30 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6010a3c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f600c06e150 con 0x7f600c0aee40 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6009d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f600c15bb50 0x7f600c0ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33130/0 (socket says 172.21.15.70:33130) 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6009d9b640 1 -- 172.21.15.70:0/235576889 learned_addr learned my addr 172.21.15.70:0/235576889 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 0x7f600c0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600a59c640 1 --2- 172.21.15.70:0/235576889 >> v2:172.21.15.98:3300/0 conn(0x7f600c15aa30 0x7f600c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838540227 0 0) 0x7f600c06dbd0 con 0x7f600c15bb50 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff4003530 con 0x7f600c15bb50 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1411724768 0 0) 0x7f5ff4003530 con 0x7f600c15bb50 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 msgr2=0x7f600c0b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 0x7f600c0b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 >> v2:172.21.15.98:3300/0 conn(0x7f600c15aa30 msgr2=0x7f600c0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 --2- 172.21.15.70:0/235576889 >> v2:172.21.15.98:3300/0 conn(0x7f600c15aa30 0x7f600c0b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c06e430 con 0x7f600c15bb50 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6000001680 con 0x7f600c15bb50 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 >> v1:172.21.15.70:6789/0 conn(0x7f600c15bb50 legacy=0x7f600c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 0x7f5ff40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ff4003f40 0x7f5ff4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ff4004d50 0x7f5ff40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f600c06de70 con 0x7f5ff4004d50 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f600c06e150 con 0x7f5ff4003f40 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5ff400b9d0 con 0x7f600c0aee40 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ff4004d50 0x7f5ff40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600a59c640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 0x7f5ff40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6009d9b640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ff4003f40 0x7f5ff4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 -- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 msgr2=0x7f5ff40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0aee40 0x7f5ff40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 -- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ff4003f40 msgr2=0x7f5ff4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ff4003f40 0x7f5ff4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 -- 172.21.15.70:0/235576889 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff4003530 con 0x7f5ff4004d50 2024-04-22T01:16:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f6009d9b640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ff4003f40 0x7f5ff4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.035+0000 7f600ad9d640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ff4004d50 0x7f5ff40051d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6004009fc0 tx=0x7f600401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd08e57ed2182d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60040b2070 con 0x7f5ff4004d50 2024-04-22T01:16:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60040a5c90 con 0x7f5ff4004d50 2024-04-22T01:16:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600959a640 1 -- 172.21.15.70:0/235576889 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004009c10 con 0x7f5ff4004d50 2024-04-22T01:16:15.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/235576889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ff4004d50 msgr2=0x7f5ff40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 --2- 172.21.15.70:0/235576889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ff4004d50 0x7f5ff40051d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6004009fc0 tx=0x7f600401cc20 comp rx=0 tx=0).stop 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600ad9d640 1 -- 172.21.15.70:0/235576889 reap_dead start 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/235576889 shutdown_connections 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/235576889 >> 172.21.15.70:0/235576889 conn(0x7f600c06b1f0 msgr2=0x7f600c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/235576889 shutdown_connections 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/235576889 wait complete. 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 Processor -- start 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- start start 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c0a8720 0x7f600c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 0x7f600c144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0a8b00 0x7f600c1454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f600c15ae10 con 0x7f600c152d30 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f600c0b4d00 con 0x7f600c0a8720 2024-04-22T01:16:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f600c15af90 con 0x7f600c0a8b00 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 0x7f600c144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 0x7f600c144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38360/0 (socket says 172.21.15.70:38360) 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 -- 172.21.15.70:0/1615727027 learned_addr learned my addr 172.21.15.70:0/1615727027 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600ad9d640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0a8b00 0x7f600c1454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600a59c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c0a8720 0x7f600c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 -- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0a8b00 msgr2=0x7f600c1454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0a8b00 0x7f600c1454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 -- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c0a8720 msgr2=0x7f600c1527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c0a8720 0x7f600c1527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 -- 172.21.15.70:0/1615727027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c157df0 con 0x7f600c152d30 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600ad9d640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0a8b00 0x7f600c1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f600a59c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c0a8720 0x7f600c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6009d9b640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 0x7f600c144f70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f600000daf0 tx=0x7f6000007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000019070 con 0x7f600c152d30 2024-04-22T01:16:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6000004cc0 con 0x7f600c152d30 2024-04-22T01:16:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000007310 con 0x7f600c152d30 2024-04-22T01:16:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600c1580d0 con 0x7f600c152d30 2024-04-22T01:16:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.039+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600c158690 con 0x7f600c152d30 2024-04-22T01:16:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.043+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f600c068210 con 0x7f600c152d30 2024-04-22T01:16:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.043+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f600001c070 con 0x7f600c152d30 2024-04-22T01:16:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.043+0000 7f5ffaffd640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5fec0678f0 0x7f5fec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.043+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f60000859a0 con 0x7f600c152d30 2024-04-22T01:16:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.043+0000 7f600a59c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5fec0678f0 0x7f5fec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.043+0000 7f600a59c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5fec0678f0 0x7f5fec069db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009aa0 tx=0x7f5ffc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.047+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f60000564a0 con 0x7f600c152d30 2024-04-22T01:16:15.148 INFO:tasks.ceph.mds.c.smithi070.stderr:2024-04-22T01:16:15.147+0000 7f2efe046640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:16:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.203+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f600c06f190 con 0x7f5fec0678f0 2024-04-22T01:16:15.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.219+0000 7f5ffaffd640 1 -- 172.21.15.70:0/1615727027 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251965 (secure 0 0 0) 0x7f600c06f190 con 0x7f5fec0678f0 2024-04-22T01:16:15.223 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:16:15.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5fec0678f0 msgr2=0x7f5fec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5fec0678f0 0x7f5fec069db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009aa0 tx=0x7f5ffc008040 comp rx=0 tx=0).stop 2024-04-22T01:16:15.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 msgr2=0x7f600c144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 0x7f600c144f70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f600000daf0 tx=0x7f6000007e40 comp rx=0 tx=0).stop 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 shutdown_connections 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5fec0678f0 0x7f5fec069db0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f600c0a8b00 0x7f600c1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f600c152d30 0x7f600c144f70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 --2- 172.21.15.70:0/1615727027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600c0a8720 0x7f600c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 >> 172.21.15.70:0/1615727027 conn(0x7f600c06b1f0 msgr2=0x7f600c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 shutdown_connections 2024-04-22T01:16:15.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.223+0000 7f6010a3c640 1 -- 172.21.15.70:0/1615727027 wait complete. 2024-04-22T01:16:15.242 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-22T01:16:14.614745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241636,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733084,"statfs":{"total":773094113280,"available":772846678016,"internally_reserved":0,"allocated":4759552,"data_stored":2705780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.455847"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423881+0000","last_change":"2024-04-22T01:16:11.220294+0000","last_active":"2024-04-22T01:16:12.423881+0000","last_peered":"2024-04-22T01:16:12.423881+0000","last_clean":"2024-04-22T01:16:12.423881+0000","last_became_active":"2024-04-22T01:16:11.215809+0000","last_became_peered":"2024-04-22T01:16:11.215809+0000","last_unstale":"2024-04-22T01:16:12.423881+0000","last_undegraded":"2024-04-22T01:16:12.423881+0000","last_fullsized":"2024-04-22T01:16:12.423881+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:14.623707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423030+0000","last_change":"2024-04-22T01:16:11.212938+0000","last_active":"2024-04-22T01:16:12.423030+0000","last_peered":"2024-04-22T01:16:12.423030+0000","last_clean":"2024-04-22T01:16:12.423030+0000","last_became_active":"2024-04-22T01:16:11.212636+0000","last_became_peered":"2024-04-22T01:16:11.212636+0000","last_unstale":"2024-04-22T01:16:12.423030+0000","last_undegraded":"2024-04-22T01:16:12.423030+0000","last_fullsized":"2024-04-22T01:16:12.423030+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:21.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420878+0000","last_change":"2024-04-22T01:16:11.253136+0000","last_active":"2024-04-22T01:16:12.420878+0000","last_peered":"2024-04-22T01:16:12.420878+0000","last_clean":"2024-04-22T01:16:12.420878+0000","last_became_active":"2024-04-22T01:16:11.252935+0000","last_became_peered":"2024-04-22T01:16:11.252935+0000","last_unstale":"2024-04-22T01:16:12.420878+0000","last_undegraded":"2024-04-22T01:16:12.420878+0000","last_fullsized":"2024-04-22T01:16:12.420878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193295+0000","last_change":"2024-04-22T01:16:11.209314+0000","last_active":"2024-04-22T01:16:12.193295+0000","last_peered":"2024-04-22T01:16:12.193295+0000","last_clean":"2024-04-22T01:16:12.193295+0000","last_became_active":"2024-04-22T01:16:11.208990+0000","last_became_peered":"2024-04-22T01:16:11.208990+0000","last_unstale":"2024-04-22T01:16:12.193295+0000","last_undegraded":"2024-04-22T01:16:12.193295+0000","last_fullsized":"2024-04-22T01:16:12.193295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:26.370977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193014+0000","last_change":"2024-04-22T01:16:11.208551+0000","last_active":"2024-04-22T01:16:12.193014+0000","last_peered":"2024-04-22T01:16:12.193014+0000","last_clean":"2024-04-22T01:16:12.193014+0000","last_became_active":"2024-04-22T01:16:11.208269+0000","last_became_peered":"2024-04-22T01:16:11.208269+0000","last_unstale":"2024-04-22T01:16:12.193014+0000","last_undegraded":"2024-04-22T01:16:12.193014+0000","last_fullsized":"2024-04-22T01:16:12.193014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:44:02.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421400+0000","last_change":"2024-04-22T01:16:11.210799+0000","last_active":"2024-04-22T01:16:12.421400+0000","last_peered":"2024-04-22T01:16:12.421400+0000","last_clean":"2024-04-22T01:16:12.421400+0000","last_became_active":"2024-04-22T01:16:11.210504+0000","last_became_peered":"2024-04-22T01:16:11.210504+0000","last_unstale":"2024-04-22T01:16:12.421400+0000","last_undegraded":"2024-04-22T01:16:12.421400+0000","last_fullsized":"2024-04-22T01:16:12.421400+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198152+0000","last_change":"2024-04-22T01:16:11.220918+0000","last_active":"2024-04-22T01:16:12.198152+0000","last_peered":"2024-04-22T01:16:12.198152+0000","last_clean":"2024-04-22T01:16:12.198152+0000","last_became_active":"2024-04-22T01:16:11.213351+0000","last_became_peered":"2024-04-22T01:16:11.213351+0000","last_unstale":"2024-04-22T01:16:12.198152+0000","last_undegraded":"2024-04-22T01:16:12.198152+0000","last_fullsized":"2024-04-22T01:16:12.198152+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:21.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422567+0000","last_change":"2024-04-22T01:16:11.249656+0000","last_active":"2024-04-22T01:16:12.422567+0000","last_peered":"2024-04-22T01:16:12.422567+0000","last_clean":"2024-04-22T01:16:12.422567+0000","last_became_active":"2024-04-22T01:16:11.249298+0000","last_became_peered":"2024-04-22T01:16:11.249298+0000","last_unstale":"2024-04-22T01:16:12.422567+0000","last_undegraded":"2024-04-22T01:16:12.422567+0000","last_fullsized":"2024-04-22T01:16:12.422567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:48.784466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421131+0000","last_change":"2024-04-22T01:16:11.249480+0000","last_active":"2024-04-22T01:16:12.421131+0000","last_peered":"2024-04-22T01:16:12.421131+0000","last_clean":"2024-04-22T01:16:12.421131+0000","last_became_active":"2024-04-22T01:16:11.249282+0000","last_became_peered":"2024-04-22T01:16:11.249282+0000","last_unstale":"2024-04-22T01:16:12.421131+0000","last_undegraded":"2024-04-22T01:16:12.421131+0000","last_fullsized":"2024-04-22T01:16:12.421131+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:01.750620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200689+0000","last_change":"2024-04-22T01:16:11.247990+0000","last_active":"2024-04-22T01:16:12.200689+0000","last_peered":"2024-04-22T01:16:12.200689+0000","last_clean":"2024-04-22T01:16:12.200689+0000","last_became_active":"2024-04-22T01:16:11.247308+0000","last_became_peered":"2024-04-22T01:16:11.247308+0000","last_unstale":"2024-04-22T01:16:12.200689+0000","last_undegraded":"2024-04-22T01:16:12.200689+0000","last_fullsized":"2024-04-22T01:16:12.200689+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:43.138333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200404+0000","last_change":"2024-04-22T01:16:11.233076+0000","last_active":"2024-04-22T01:16:12.200404+0000","last_peered":"2024-04-22T01:16:12.200404+0000","last_clean":"2024-04-22T01:16:12.200404+0000","last_became_active":"2024-04-22T01:16:11.212081+0000","last_became_peered":"2024-04-22T01:16:11.212081+0000","last_unstale":"2024-04-22T01:16:12.200404+0000","last_undegraded":"2024-04-22T01:16:12.200404+0000","last_fullsized":"2024-04-22T01:16:12.200404+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:25.849341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421539+0000","last_change":"2024-04-22T01:16:11.211992+0000","last_active":"2024-04-22T01:16:12.421539+0000","last_peered":"2024-04-22T01:16:12.421539+0000","last_clean":"2024-04-22T01:16:12.421539+0000","last_became_active":"2024-04-22T01:16:11.211717+0000","last_became_peered":"2024-04-22T01:16:11.211717+0000","last_unstale":"2024-04-22T01:16:12.421539+0000","last_undegraded":"2024-04-22T01:16:12.421539+0000","last_fullsized":"2024-04-22T01:16:12.421539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:40.324674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198174+0000","last_change":"2024-04-22T01:16:11.233217+0000","last_active":"2024-04-22T01:16:12.198174+0000","last_peered":"2024-04-22T01:16:12.198174+0000","last_clean":"2024-04-22T01:16:12.198174+0000","last_became_active":"2024-04-22T01:16:11.232469+0000","last_became_peered":"2024-04-22T01:16:11.232469+0000","last_unstale":"2024-04-22T01:16:12.198174+0000","last_undegraded":"2024-04-22T01:16:12.198174+0000","last_fullsized":"2024-04-22T01:16:12.198174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:14.415647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419725+0000","last_change":"2024-04-22T01:16:11.257328+0000","last_active":"2024-04-22T01:16:12.419725+0000","last_peered":"2024-04-22T01:16:12.419725+0000","last_clean":"2024-04-22T01:16:12.419725+0000","last_became_active":"2024-04-22T01:16:11.256896+0000","last_became_peered":"2024-04-22T01:16:11.256896+0000","last_unstale":"2024-04-22T01:16:12.419725+0000","last_undegraded":"2024-04-22T01:16:12.419725+0000","last_fullsized":"2024-04-22T01:16:12.419725+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198679+0000","last_change":"2024-04-22T01:16:11.219791+0000","last_active":"2024-04-22T01:16:12.198679+0000","last_peered":"2024-04-22T01:16:12.198679+0000","last_clean":"2024-04-22T01:16:12.198679+0000","last_became_active":"2024-04-22T01:16:11.212232+0000","last_became_peered":"2024-04-22T01:16:11.212232+0000","last_unstale":"2024-04-22T01:16:12.198679+0000","last_undegraded":"2024-04-22T01:16:12.198679+0000","last_fullsized":"2024-04-22T01:16:12.198679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:14.623707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421006+0000","last_change":"2024-04-22T01:16:11.260182+0000","last_active":"2024-04-22T01:16:12.421006+0000","last_peered":"2024-04-22T01:16:12.421006+0000","last_clean":"2024-04-22T01:16:12.421006+0000","last_became_active":"2024-04-22T01:16:11.259537+0000","last_became_peered":"2024-04-22T01:16:11.259537+0000","last_unstale":"2024-04-22T01:16:12.421006+0000","last_undegraded":"2024-04-22T01:16:12.421006+0000","last_fullsized":"2024-04-22T01:16:12.421006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422162+0000","last_change":"2024-04-22T01:16:11.256851+0000","last_active":"2024-04-22T01:16:12.422162+0000","last_peered":"2024-04-22T01:16:12.422162+0000","last_clean":"2024-04-22T01:16:12.422162+0000","last_became_active":"2024-04-22T01:16:11.256278+0000","last_became_peered":"2024-04-22T01:16:11.256278+0000","last_unstale":"2024-04-22T01:16:12.422162+0000","last_undegraded":"2024-04-22T01:16:12.422162+0000","last_fullsized":"2024-04-22T01:16:12.422162+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:38:12.697995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199708+0000","last_change":"2024-04-22T01:16:11.244336+0000","last_active":"2024-04-22T01:16:12.199708+0000","last_peered":"2024-04-22T01:16:12.199708+0000","last_clean":"2024-04-22T01:16:12.199708+0000","last_became_active":"2024-04-22T01:16:11.243637+0000","last_became_peered":"2024-04-22T01:16:11.243637+0000","last_unstale":"2024-04-22T01:16:12.199708+0000","last_undegraded":"2024-04-22T01:16:12.199708+0000","last_fullsized":"2024-04-22T01:16:12.199708+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193556+0000","last_change":"2024-04-22T01:16:11.206006+0000","last_active":"2024-04-22T01:16:12.193556+0000","last_peered":"2024-04-22T01:16:12.193556+0000","last_clean":"2024-04-22T01:16:12.193556+0000","last_became_active":"2024-04-22T01:16:11.205731+0000","last_became_peered":"2024-04-22T01:16:11.205731+0000","last_unstale":"2024-04-22T01:16:12.193556+0000","last_undegraded":"2024-04-22T01:16:12.193556+0000","last_fullsized":"2024-04-22T01:16:12.193556+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:22.720037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199110+0000","last_change":"2024-04-22T01:16:11.255648+0000","last_active":"2024-04-22T01:16:12.199110+0000","last_peered":"2024-04-22T01:16:12.199110+0000","last_clean":"2024-04-22T01:16:12.199110+0000","last_became_active":"2024-04-22T01:16:11.255089+0000","last_became_peered":"2024-04-22T01:16:11.255089+0000","last_unstale":"2024-04-22T01:16:12.199110+0000","last_undegraded":"2024-04-22T01:16:12.199110+0000","last_fullsized":"2024-04-22T01:16:12.199110+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:00.140586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198426+0000","last_change":"2024-04-22T01:16:11.213176+0000","last_active":"2024-04-22T01:16:12.198426+0000","last_peered":"2024-04-22T01:16:12.198426+0000","last_clean":"2024-04-22T01:16:12.198426+0000","last_became_active":"2024-04-22T01:16:11.212457+0000","last_became_peered":"2024-04-22T01:16:11.212457+0000","last_unstale":"2024-04-22T01:16:12.198426+0000","last_undegraded":"2024-04-22T01:16:12.198426+0000","last_fullsized":"2024-04-22T01:16:12.198426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:38:12.697995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420583+0000","last_change":"2024-04-22T01:16:11.205942+0000","last_active":"2024-04-22T01:16:12.420583+0000","last_peered":"2024-04-22T01:16:12.420583+0000","last_clean":"2024-04-22T01:16:12.420583+0000","last_became_active":"2024-04-22T01:16:11.204590+0000","last_became_peered":"2024-04-22T01:16:11.204590+0000","last_unstale":"2024-04-22T01:16:12.420583+0000","last_undegraded":"2024-04-22T01:16:12.420583+0000","last_fullsized":"2024-04-22T01:16:12.420583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:28.631710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421029+0000","last_change":"2024-04-22T01:16:11.217798+0000","last_active":"2024-04-22T01:16:12.421029+0000","last_peered":"2024-04-22T01:16:12.421029+0000","last_clean":"2024-04-22T01:16:12.421029+0000","last_became_active":"2024-04-22T01:16:11.216732+0000","last_became_peered":"2024-04-22T01:16:11.216732+0000","last_unstale":"2024-04-22T01:16:12.421029+0000","last_undegraded":"2024-04-22T01:16:12.421029+0000","last_fullsized":"2024-04-22T01:16:12.421029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:01.750620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.192887+0000","last_change":"2024-04-22T01:16:11.210915+0000","last_active":"2024-04-22T01:16:12.192887+0000","last_peered":"2024-04-22T01:16:12.192887+0000","last_clean":"2024-04-22T01:16:12.192887+0000","last_became_active":"2024-04-22T01:16:11.210630+0000","last_became_peered":"2024-04-22T01:16:11.210630+0000","last_unstale":"2024-04-22T01:16:12.192887+0000","last_undegraded":"2024-04-22T01:16:12.192887+0000","last_fullsized":"2024-04-22T01:16:12.192887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:20.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199985+0000","last_change":"2024-04-22T01:16:11.244186+0000","last_active":"2024-04-22T01:16:12.199985+0000","last_peered":"2024-04-22T01:16:12.199985+0000","last_clean":"2024-04-22T01:16:12.199985+0000","last_became_active":"2024-04-22T01:16:11.243443+0000","last_became_peered":"2024-04-22T01:16:11.243443+0000","last_unstale":"2024-04-22T01:16:12.199985+0000","last_undegraded":"2024-04-22T01:16:12.199985+0000","last_fullsized":"2024-04-22T01:16:12.199985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422955+0000","last_change":"2024-04-22T01:16:11.254761+0000","last_active":"2024-04-22T01:16:12.422955+0000","last_peered":"2024-04-22T01:16:12.422955+0000","last_clean":"2024-04-22T01:16:12.422955+0000","last_became_active":"2024-04-22T01:16:11.254492+0000","last_became_peered":"2024-04-22T01:16:11.254492+0000","last_unstale":"2024-04-22T01:16:12.422955+0000","last_undegraded":"2024-04-22T01:16:12.422955+0000","last_fullsized":"2024-04-22T01:16:12.422955+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:16.781837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199230+0000","last_change":"2024-04-22T01:16:11.255259+0000","last_active":"2024-04-22T01:16:12.199230+0000","last_peered":"2024-04-22T01:16:12.199230+0000","last_clean":"2024-04-22T01:16:12.199230+0000","last_became_active":"2024-04-22T01:16:11.254822+0000","last_became_peered":"2024-04-22T01:16:11.254822+0000","last_unstale":"2024-04-22T01:16:12.199230+0000","last_undegraded":"2024-04-22T01:16:12.199230+0000","last_fullsized":"2024-04-22T01:16:12.199230+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:22.658275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199719+0000","last_change":"2024-04-22T01:16:11.221218+0000","last_active":"2024-04-22T01:16:12.199719+0000","last_peered":"2024-04-22T01:16:12.199719+0000","last_clean":"2024-04-22T01:16:12.199719+0000","last_became_active":"2024-04-22T01:16:11.220474+0000","last_became_peered":"2024-04-22T01:16:11.220474+0000","last_unstale":"2024-04-22T01:16:12.199719+0000","last_undegraded":"2024-04-22T01:16:12.199719+0000","last_fullsized":"2024-04-22T01:16:12.199719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:50.580485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199139+0000","last_change":"2024-04-22T01:16:11.244373+0000","last_active":"2024-04-22T01:16:12.199139+0000","last_peered":"2024-04-22T01:16:12.199139+0000","last_clean":"2024-04-22T01:16:12.199139+0000","last_became_active":"2024-04-22T01:16:11.243875+0000","last_became_peered":"2024-04-22T01:16:11.243875+0000","last_unstale":"2024-04-22T01:16:12.199139+0000","last_undegraded":"2024-04-22T01:16:12.199139+0000","last_fullsized":"2024-04-22T01:16:12.199139+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:58.850159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197651+0000","last_change":"2024-04-22T01:16:11.255896+0000","last_active":"2024-04-22T01:16:12.197651+0000","last_peered":"2024-04-22T01:16:12.197651+0000","last_clean":"2024-04-22T01:16:12.197651+0000","last_became_active":"2024-04-22T01:16:11.255048+0000","last_became_peered":"2024-04-22T01:16:11.255048+0000","last_unstale":"2024-04-22T01:16:12.197651+0000","last_undegraded":"2024-04-22T01:16:12.197651+0000","last_fullsized":"2024-04-22T01:16:12.197651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:06.277212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421508+0000","last_change":"2024-04-22T01:16:11.223414+0000","last_active":"2024-04-22T01:16:12.421508+0000","last_peered":"2024-04-22T01:16:12.421508+0000","last_clean":"2024-04-22T01:16:12.421508+0000","last_became_active":"2024-04-22T01:16:11.217640+0000","last_became_peered":"2024-04-22T01:16:11.217640+0000","last_unstale":"2024-04-22T01:16:12.421508+0000","last_undegraded":"2024-04-22T01:16:12.421508+0000","last_fullsized":"2024-04-22T01:16:12.421508+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:24.822551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422187+0000","last_change":"2024-04-22T01:16:11.213024+0000","last_active":"2024-04-22T01:16:12.422187+0000","last_peered":"2024-04-22T01:16:12.422187+0000","last_clean":"2024-04-22T01:16:12.422187+0000","last_became_active":"2024-04-22T01:16:11.212176+0000","last_became_peered":"2024-04-22T01:16:11.212176+0000","last_unstale":"2024-04-22T01:16:12.422187+0000","last_undegraded":"2024-04-22T01:16:12.422187+0000","last_fullsized":"2024-04-22T01:16:12.422187+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:28.631710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421902+0000","last_change":"2024-04-22T01:16:11.254915+0000","last_active":"2024-04-22T01:16:12.421902+0000","last_peered":"2024-04-22T01:16:12.421902+0000","last_clean":"2024-04-22T01:16:12.421902+0000","last_became_active":"2024-04-22T01:16:11.254468+0000","last_became_peered":"2024-04-22T01:16:11.254468+0000","last_unstale":"2024-04-22T01:16:12.421902+0000","last_undegraded":"2024-04-22T01:16:12.421902+0000","last_fullsized":"2024-04-22T01:16:12.421902+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:21.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199860+0000","last_change":"2024-04-22T01:16:10.208983+0000","last_active":"2024-04-22T01:16:12.199860+0000","last_peered":"2024-04-22T01:16:12.199860+0000","last_clean":"2024-04-22T01:16:12.199860+0000","last_became_active":"2024-04-22T01:16:10.208262+0000","last_became_peered":"2024-04-22T01:16:10.208262+0000","last_unstale":"2024-04-22T01:16:12.199860+0000","last_undegraded":"2024-04-22T01:16:12.199860+0000","last_fullsized":"2024-04-22T01:16:12.199860+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.422243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423718+0000","last_change":"2024-04-22T01:16:11.251187+0000","last_active":"2024-04-22T01:16:12.423718+0000","last_peered":"2024-04-22T01:16:12.423718+0000","last_clean":"2024-04-22T01:16:12.423718+0000","last_became_active":"2024-04-22T01:16:11.250593+0000","last_became_peered":"2024-04-22T01:16:11.250593+0000","last_unstale":"2024-04-22T01:16:12.423718+0000","last_undegraded":"2024-04-22T01:16:12.423718+0000","last_fullsized":"2024-04-22T01:16:12.423718+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420852+0000","last_change":"2024-04-22T01:16:10.204605+0000","last_active":"2024-04-22T01:16:12.420852+0000","last_peered":"2024-04-22T01:16:12.420852+0000","last_clean":"2024-04-22T01:16:12.420852+0000","last_became_active":"2024-04-22T01:16:10.204140+0000","last_became_peered":"2024-04-22T01:16:10.204140+0000","last_unstale":"2024-04-22T01:16:12.420852+0000","last_undegraded":"2024-04-22T01:16:12.420852+0000","last_fullsized":"2024-04-22T01:16:12.420852+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:21.708617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420954+0000","last_change":"2024-04-22T01:16:11.246154+0000","last_active":"2024-04-22T01:16:12.420954+0000","last_peered":"2024-04-22T01:16:12.420954+0000","last_clean":"2024-04-22T01:16:12.420954+0000","last_became_active":"2024-04-22T01:16:11.245275+0000","last_became_peered":"2024-04-22T01:16:11.245275+0000","last_unstale":"2024-04-22T01:16:12.420954+0000","last_undegraded":"2024-04-22T01:16:12.420954+0000","last_fullsized":"2024-04-22T01:16:12.420954+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:01.750620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424243+0000","last_change":"2024-04-22T01:16:10.206430+0000","last_active":"2024-04-22T01:16:12.424243+0000","last_peered":"2024-04-22T01:16:12.424243+0000","last_clean":"2024-04-22T01:16:12.424243+0000","last_became_active":"2024-04-22T01:16:10.206160+0000","last_became_peered":"2024-04-22T01:16:10.206160+0000","last_unstale":"2024-04-22T01:16:12.424243+0000","last_undegraded":"2024-04-22T01:16:12.424243+0000","last_fullsized":"2024-04-22T01:16:12.424243+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.422243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423298+0000","last_change":"2024-04-22T01:16:11.257715+0000","last_active":"2024-04-22T01:16:12.423298+0000","last_peered":"2024-04-22T01:16:12.423298+0000","last_clean":"2024-04-22T01:16:12.423298+0000","last_became_active":"2024-04-22T01:16:11.257533+0000","last_became_peered":"2024-04-22T01:16:11.257533+0000","last_unstale":"2024-04-22T01:16:12.423298+0000","last_undegraded":"2024-04-22T01:16:12.423298+0000","last_fullsized":"2024-04-22T01:16:12.423298+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:51.252811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420242+0000","last_change":"2024-04-22T01:16:10.203902+0000","last_active":"2024-04-22T01:16:12.420242+0000","last_peered":"2024-04-22T01:16:12.420242+0000","last_clean":"2024-04-22T01:16:12.420242+0000","last_became_active":"2024-04-22T01:16:10.201785+0000","last_became_peered":"2024-04-22T01:16:10.201785+0000","last_unstale":"2024-04-22T01:16:12.420242+0000","last_undegraded":"2024-04-22T01:16:12.420242+0000","last_fullsized":"2024-04-22T01:16:12.420242+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420234+0000","last_change":"2024-04-22T01:16:11.203592+0000","last_active":"2024-04-22T01:16:12.420234+0000","last_peered":"2024-04-22T01:16:12.420234+0000","last_clean":"2024-04-22T01:16:12.420234+0000","last_became_active":"2024-04-22T01:16:11.203323+0000","last_became_peered":"2024-04-22T01:16:11.203323+0000","last_unstale":"2024-04-22T01:16:12.420234+0000","last_undegraded":"2024-04-22T01:16:12.420234+0000","last_fullsized":"2024-04-22T01:16:12.420234+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198376+0000","last_change":"2024-04-22T01:16:10.191468+0000","last_active":"2024-04-22T01:16:12.198376+0000","last_peered":"2024-04-22T01:16:12.198376+0000","last_clean":"2024-04-22T01:16:12.198376+0000","last_became_active":"2024-04-22T01:16:10.190212+0000","last_became_peered":"2024-04-22T01:16:10.190212+0000","last_unstale":"2024-04-22T01:16:12.198376+0000","last_undegraded":"2024-04-22T01:16:12.198376+0000","last_fullsized":"2024-04-22T01:16:12.198376+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:18.786008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198569+0000","last_change":"2024-04-22T01:16:11.233440+0000","last_active":"2024-04-22T01:16:12.198569+0000","last_peered":"2024-04-22T01:16:12.198569+0000","last_clean":"2024-04-22T01:16:12.198569+0000","last_became_active":"2024-04-22T01:16:11.221648+0000","last_became_peered":"2024-04-22T01:16:11.221648+0000","last_unstale":"2024-04-22T01:16:12.198569+0000","last_undegraded":"2024-04-22T01:16:12.198569+0000","last_fullsized":"2024-04-22T01:16:12.198569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:22:04.840889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421864+0000","last_change":"2024-04-22T01:16:10.213968+0000","last_active":"2024-04-22T01:16:12.421864+0000","last_peered":"2024-04-22T01:16:12.421864+0000","last_clean":"2024-04-22T01:16:12.421864+0000","last_became_active":"2024-04-22T01:16:10.212756+0000","last_became_peered":"2024-04-22T01:16:10.212756+0000","last_unstale":"2024-04-22T01:16:12.421864+0000","last_undegraded":"2024-04-22T01:16:12.421864+0000","last_fullsized":"2024-04-22T01:16:12.421864+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421177+0000","last_change":"2024-04-22T01:16:11.238257+0000","last_active":"2024-04-22T01:16:12.421177+0000","last_peered":"2024-04-22T01:16:12.421177+0000","last_clean":"2024-04-22T01:16:12.421177+0000","last_became_active":"2024-04-22T01:16:11.236300+0000","last_became_peered":"2024-04-22T01:16:11.236300+0000","last_unstale":"2024-04-22T01:16:12.421177+0000","last_undegraded":"2024-04-22T01:16:12.421177+0000","last_fullsized":"2024-04-22T01:16:12.421177+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:40.324674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421629+0000","last_change":"2024-04-22T01:16:10.190343+0000","last_active":"2024-04-22T01:16:12.421629+0000","last_peered":"2024-04-22T01:16:12.421629+0000","last_clean":"2024-04-22T01:16:12.421629+0000","last_became_active":"2024-04-22T01:16:10.189942+0000","last_became_peered":"2024-04-22T01:16:10.189942+0000","last_unstale":"2024-04-22T01:16:12.421629+0000","last_undegraded":"2024-04-22T01:16:12.421629+0000","last_fullsized":"2024-04-22T01:16:12.421629+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424438+0000","last_change":"2024-04-22T01:16:11.219075+0000","last_active":"2024-04-22T01:16:12.424438+0000","last_peered":"2024-04-22T01:16:12.424438+0000","last_clean":"2024-04-22T01:16:12.424438+0000","last_became_active":"2024-04-22T01:16:11.218799+0000","last_became_peered":"2024-04-22T01:16:11.218799+0000","last_unstale":"2024-04-22T01:16:12.424438+0000","last_undegraded":"2024-04-22T01:16:12.424438+0000","last_fullsized":"2024-04-22T01:16:12.424438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:50.580485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419432+0000","last_change":"2024-04-22T01:16:10.203255+0000","last_active":"2024-04-22T01:16:12.419432+0000","last_peered":"2024-04-22T01:16:12.419432+0000","last_clean":"2024-04-22T01:16:12.419432+0000","last_became_active":"2024-04-22T01:16:10.201710+0000","last_became_peered":"2024-04-22T01:16:10.201710+0000","last_unstale":"2024-04-22T01:16:12.419432+0000","last_undegraded":"2024-04-22T01:16:12.419432+0000","last_fullsized":"2024-04-22T01:16:12.419432+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:44:01.704153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193299+0000","last_change":"2024-04-22T01:16:11.201649+0000","last_active":"2024-04-22T01:16:12.193299+0000","last_peered":"2024-04-22T01:16:12.193299+0000","last_clean":"2024-04-22T01:16:12.193299+0000","last_became_active":"2024-04-22T01:16:11.201118+0000","last_became_peered":"2024-04-22T01:16:11.201118+0000","last_unstale":"2024-04-22T01:16:12.193299+0000","last_undegraded":"2024-04-22T01:16:12.193299+0000","last_fullsized":"2024-04-22T01:16:12.193299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:40.146273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424119+0000","last_change":"2024-04-22T01:16:10.184747+0000","last_active":"2024-04-22T01:16:12.424119+0000","last_peered":"2024-04-22T01:16:12.424119+0000","last_clean":"2024-04-22T01:16:12.424119+0000","last_became_active":"2024-04-22T01:16:10.184440+0000","last_became_peered":"2024-04-22T01:16:10.184440+0000","last_unstale":"2024-04-22T01:16:12.424119+0000","last_undegraded":"2024-04-22T01:16:12.424119+0000","last_fullsized":"2024-04-22T01:16:12.424119+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:59.129166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422137+0000","last_change":"2024-04-22T01:16:11.257778+0000","last_active":"2024-04-22T01:16:12.422137+0000","last_peered":"2024-04-22T01:16:12.422137+0000","last_clean":"2024-04-22T01:16:12.422137+0000","last_became_active":"2024-04-22T01:16:11.257288+0000","last_became_peered":"2024-04-22T01:16:11.257288+0000","last_unstale":"2024-04-22T01:16:12.422137+0000","last_undegraded":"2024-04-22T01:16:12.422137+0000","last_fullsized":"2024-04-22T01:16:12.422137+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:51.252811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200874+0000","last_change":"2024-04-22T01:16:10.190858+0000","last_active":"2024-04-22T01:16:12.200874+0000","last_peered":"2024-04-22T01:16:12.200874+0000","last_clean":"2024-04-22T01:16:12.200874+0000","last_became_active":"2024-04-22T01:16:10.189367+0000","last_became_peered":"2024-04-22T01:16:10.189367+0000","last_unstale":"2024-04-22T01:16:12.200874+0000","last_undegraded":"2024-04-22T01:16:12.200874+0000","last_fullsized":"2024-04-22T01:16:12.200874+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:59.129166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420675+0000","last_change":"2024-04-22T01:16:11.214201+0000","last_active":"2024-04-22T01:16:12.420675+0000","last_peered":"2024-04-22T01:16:12.420675+0000","last_clean":"2024-04-22T01:16:12.420675+0000","last_became_active":"2024-04-22T01:16:11.213885+0000","last_became_peered":"2024-04-22T01:16:11.213885+0000","last_unstale":"2024-04-22T01:16:12.420675+0000","last_undegraded":"2024-04-22T01:16:12.420675+0000","last_fullsized":"2024-04-22T01:16:12.420675+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:16.781837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200719+0000","last_change":"2024-04-22T01:16:10.191221+0000","last_active":"2024-04-22T01:16:12.200719+0000","last_peered":"2024-04-22T01:16:12.200719+0000","last_clean":"2024-04-22T01:16:12.200719+0000","last_became_active":"2024-04-22T01:16:10.190302+0000","last_became_peered":"2024-04-22T01:16:10.190302+0000","last_unstale":"2024-04-22T01:16:12.200719+0000","last_undegraded":"2024-04-22T01:16:12.200719+0000","last_fullsized":"2024-04-22T01:16:12.200719+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:05.265793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420696+0000","last_change":"2024-04-22T01:16:11.207820+0000","last_active":"2024-04-22T01:16:12.420696+0000","last_peered":"2024-04-22T01:16:12.420696+0000","last_clean":"2024-04-22T01:16:12.420696+0000","last_became_active":"2024-04-22T01:16:11.207083+0000","last_became_peered":"2024-04-22T01:16:11.207083+0000","last_unstale":"2024-04-22T01:16:12.420696+0000","last_undegraded":"2024-04-22T01:16:12.420696+0000","last_fullsized":"2024-04-22T01:16:12.420696+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:16.781837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424088+0000","last_change":"2024-04-22T01:16:10.207758+0000","last_active":"2024-04-22T01:16:12.424088+0000","last_peered":"2024-04-22T01:16:12.424088+0000","last_clean":"2024-04-22T01:16:12.424088+0000","last_became_active":"2024-04-22T01:16:10.207492+0000","last_became_peered":"2024-04-22T01:16:10.207492+0000","last_unstale":"2024-04-22T01:16:12.424088+0000","last_undegraded":"2024-04-22T01:16:12.424088+0000","last_fullsized":"2024-04-22T01:16:12.424088+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:34.113075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420230+0000","last_change":"2024-04-22T01:16:11.223856+0000","last_active":"2024-04-22T01:16:12.420230+0000","last_peered":"2024-04-22T01:16:12.420230+0000","last_clean":"2024-04-22T01:16:12.420230+0000","last_became_active":"2024-04-22T01:16:11.223412+0000","last_became_peered":"2024-04-22T01:16:11.223412+0000","last_unstale":"2024-04-22T01:16:12.420230+0000","last_undegraded":"2024-04-22T01:16:12.420230+0000","last_fullsized":"2024-04-22T01:16:12.420230+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424889+0000","last_change":"2024-04-22T01:16:10.184749+0000","last_active":"2024-04-22T01:16:12.424889+0000","last_peered":"2024-04-22T01:16:12.424889+0000","last_clean":"2024-04-22T01:16:12.424889+0000","last_became_active":"2024-04-22T01:16:10.184477+0000","last_became_peered":"2024-04-22T01:16:10.184477+0000","last_unstale":"2024-04-22T01:16:12.424889+0000","last_undegraded":"2024-04-22T01:16:12.424889+0000","last_fullsized":"2024-04-22T01:16:12.424889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:05.265793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423052+0000","last_change":"2024-04-22T01:16:11.212536+0000","last_active":"2024-04-22T01:16:12.423052+0000","last_peered":"2024-04-22T01:16:12.423052+0000","last_clean":"2024-04-22T01:16:12.423052+0000","last_became_active":"2024-04-22T01:16:11.212248+0000","last_became_peered":"2024-04-22T01:16:11.212248+0000","last_unstale":"2024-04-22T01:16:12.423052+0000","last_undegraded":"2024-04-22T01:16:12.423052+0000","last_fullsized":"2024-04-22T01:16:12.423052+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:14.415647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421651+0000","last_change":"2024-04-22T01:16:10.190411+0000","last_active":"2024-04-22T01:16:12.421651+0000","last_peered":"2024-04-22T01:16:12.421651+0000","last_clean":"2024-04-22T01:16:12.421651+0000","last_became_active":"2024-04-22T01:16:10.190036+0000","last_became_peered":"2024-04-22T01:16:10.190036+0000","last_unstale":"2024-04-22T01:16:12.421651+0000","last_undegraded":"2024-04-22T01:16:12.421651+0000","last_fullsized":"2024-04-22T01:16:12.421651+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:21.708617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421566+0000","last_change":"2024-04-22T01:16:11.216600+0000","last_active":"2024-04-22T01:16:12.421566+0000","last_peered":"2024-04-22T01:16:12.421566+0000","last_clean":"2024-04-22T01:16:12.421566+0000","last_became_active":"2024-04-22T01:16:11.215745+0000","last_became_peered":"2024-04-22T01:16:11.215745+0000","last_unstale":"2024-04-22T01:16:12.421566+0000","last_undegraded":"2024-04-22T01:16:12.421566+0000","last_fullsized":"2024-04-22T01:16:12.421566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:48.784466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197364+0000","last_change":"2024-04-22T01:16:10.210763+0000","last_active":"2024-04-22T01:16:12.197364+0000","last_peered":"2024-04-22T01:16:12.197364+0000","last_clean":"2024-04-22T01:16:12.197364+0000","last_became_active":"2024-04-22T01:16:10.210172+0000","last_became_peered":"2024-04-22T01:16:10.210172+0000","last_unstale":"2024-04-22T01:16:12.197364+0000","last_undegraded":"2024-04-22T01:16:12.197364+0000","last_fullsized":"2024-04-22T01:16:12.197364+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:44:01.704153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424660+0000","last_change":"2024-04-22T01:16:11.211991+0000","last_active":"2024-04-22T01:16:12.424660+0000","last_peered":"2024-04-22T01:16:12.424660+0000","last_clean":"2024-04-22T01:16:12.424660+0000","last_became_active":"2024-04-22T01:16:11.211714+0000","last_became_peered":"2024-04-22T01:16:11.211714+0000","last_unstale":"2024-04-22T01:16:12.424660+0000","last_undegraded":"2024-04-22T01:16:12.424660+0000","last_fullsized":"2024-04-22T01:16:12.424660+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:25.849341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420966+0000","last_change":"2024-04-22T01:16:10.209133+0000","last_active":"2024-04-22T01:16:12.420966+0000","last_peered":"2024-04-22T01:16:12.420966+0000","last_clean":"2024-04-22T01:16:12.420966+0000","last_became_active":"2024-04-22T01:16:10.208380+0000","last_became_peered":"2024-04-22T01:16:10.208380+0000","last_unstale":"2024-04-22T01:16:12.420966+0000","last_undegraded":"2024-04-22T01:16:12.420966+0000","last_fullsized":"2024-04-22T01:16:12.420966+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:21.708617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423818+0000","last_change":"2024-04-22T01:16:11.245265+0000","last_active":"2024-04-22T01:16:12.423818+0000","last_peered":"2024-04-22T01:16:12.423818+0000","last_clean":"2024-04-22T01:16:12.423818+0000","last_became_active":"2024-04-22T01:16:11.244958+0000","last_became_peered":"2024-04-22T01:16:11.244958+0000","last_unstale":"2024-04-22T01:16:12.423818+0000","last_undegraded":"2024-04-22T01:16:12.423818+0000","last_fullsized":"2024-04-22T01:16:12.423818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:22:04.840889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200923+0000","last_change":"2024-04-22T01:16:10.206641+0000","last_active":"2024-04-22T01:16:12.200923+0000","last_peered":"2024-04-22T01:16:12.200923+0000","last_clean":"2024-04-22T01:16:12.200923+0000","last_became_active":"2024-04-22T01:16:10.206018+0000","last_became_peered":"2024-04-22T01:16:10.206018+0000","last_unstale":"2024-04-22T01:16:12.200923+0000","last_undegraded":"2024-04-22T01:16:12.200923+0000","last_fullsized":"2024-04-22T01:16:12.200923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:30.477033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200637+0000","last_change":"2024-04-22T01:16:11.234503+0000","last_active":"2024-04-22T01:16:12.200637+0000","last_peered":"2024-04-22T01:16:12.200637+0000","last_clean":"2024-04-22T01:16:12.200637+0000","last_became_active":"2024-04-22T01:16:11.233736+0000","last_became_peered":"2024-04-22T01:16:11.233736+0000","last_unstale":"2024-04-22T01:16:12.200637+0000","last_undegraded":"2024-04-22T01:16:12.200637+0000","last_fullsized":"2024-04-22T01:16:12.200637+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.364265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421064+0000","last_change":"2024-04-22T01:16:10.204693+0000","last_active":"2024-04-22T01:16:12.421064+0000","last_peered":"2024-04-22T01:16:12.421064+0000","last_clean":"2024-04-22T01:16:12.421064+0000","last_became_active":"2024-04-22T01:16:10.203946+0000","last_became_peered":"2024-04-22T01:16:10.203946+0000","last_unstale":"2024-04-22T01:16:12.421064+0000","last_undegraded":"2024-04-22T01:16:12.421064+0000","last_fullsized":"2024-04-22T01:16:12.421064+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420781+0000","last_change":"2024-04-22T01:16:11.208260+0000","last_active":"2024-04-22T01:16:12.420781+0000","last_peered":"2024-04-22T01:16:12.420781+0000","last_clean":"2024-04-22T01:16:12.420781+0000","last_became_active":"2024-04-22T01:16:11.207403+0000","last_became_peered":"2024-04-22T01:16:11.207403+0000","last_unstale":"2024-04-22T01:16:12.420781+0000","last_undegraded":"2024-04-22T01:16:12.420781+0000","last_fullsized":"2024-04-22T01:16:12.420781+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193367+0000","last_change":"2024-04-22T01:16:10.203958+0000","last_active":"2024-04-22T01:16:12.193367+0000","last_peered":"2024-04-22T01:16:12.193367+0000","last_clean":"2024-04-22T01:16:12.193367+0000","last_became_active":"2024-04-22T01:16:10.203412+0000","last_became_peered":"2024-04-22T01:16:10.203412+0000","last_unstale":"2024-04-22T01:16:12.193367+0000","last_undegraded":"2024-04-22T01:16:12.193367+0000","last_fullsized":"2024-04-22T01:16:12.193367+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:49.884021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193551+0000","last_change":"2024-04-22T01:16:11.212277+0000","last_active":"2024-04-22T01:16:12.193551+0000","last_peered":"2024-04-22T01:16:12.193551+0000","last_clean":"2024-04-22T01:16:12.193551+0000","last_became_active":"2024-04-22T01:16:11.211784+0000","last_became_peered":"2024-04-22T01:16:11.211784+0000","last_unstale":"2024-04-22T01:16:12.193551+0000","last_undegraded":"2024-04-22T01:16:12.193551+0000","last_fullsized":"2024-04-22T01:16:12.193551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:29.629103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421745+0000","last_change":"2024-04-22T01:16:10.208644+0000","last_active":"2024-04-22T01:16:12.421745+0000","last_peered":"2024-04-22T01:16:12.421745+0000","last_clean":"2024-04-22T01:16:12.421745+0000","last_became_active":"2024-04-22T01:16:10.208359+0000","last_became_peered":"2024-04-22T01:16:10.208359+0000","last_unstale":"2024-04-22T01:16:12.421745+0000","last_undegraded":"2024-04-22T01:16:12.421745+0000","last_fullsized":"2024-04-22T01:16:12.421745+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421420+0000","last_change":"2024-04-22T01:16:11.218221+0000","last_active":"2024-04-22T01:16:12.421420+0000","last_peered":"2024-04-22T01:16:12.421420+0000","last_clean":"2024-04-22T01:16:12.421420+0000","last_became_active":"2024-04-22T01:16:11.217469+0000","last_became_peered":"2024-04-22T01:16:11.217469+0000","last_unstale":"2024-04-22T01:16:12.421420+0000","last_undegraded":"2024-04-22T01:16:12.421420+0000","last_fullsized":"2024-04-22T01:16:12.421420+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:40.324674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193091+0000","last_change":"2024-04-22T01:16:10.185392+0000","last_active":"2024-04-22T01:16:12.193091+0000","last_peered":"2024-04-22T01:16:12.193091+0000","last_clean":"2024-04-22T01:16:12.193091+0000","last_became_active":"2024-04-22T01:16:10.185016+0000","last_became_peered":"2024-04-22T01:16:10.185016+0000","last_unstale":"2024-04-22T01:16:12.193091+0000","last_undegraded":"2024-04-22T01:16:12.193091+0000","last_fullsized":"2024-04-22T01:16:12.193091+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:56.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421765+0000","last_change":"2024-04-22T01:16:11.214137+0000","last_active":"2024-04-22T01:16:12.421765+0000","last_peered":"2024-04-22T01:16:12.421765+0000","last_clean":"2024-04-22T01:16:12.421765+0000","last_became_active":"2024-04-22T01:16:11.213802+0000","last_became_peered":"2024-04-22T01:16:11.213802+0000","last_unstale":"2024-04-22T01:16:12.421765+0000","last_undegraded":"2024-04-22T01:16:12.421765+0000","last_fullsized":"2024-04-22T01:16:12.421765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:24.822551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.192918+0000","last_change":"2024-04-22T01:16:10.209151+0000","last_active":"2024-04-22T01:16:12.192918+0000","last_peered":"2024-04-22T01:16:12.192918+0000","last_clean":"2024-04-22T01:16:12.192918+0000","last_became_active":"2024-04-22T01:16:10.208837+0000","last_became_peered":"2024-04-22T01:16:10.208837+0000","last_unstale":"2024-04-22T01:16:12.192918+0000","last_undegraded":"2024-04-22T01:16:12.192918+0000","last_fullsized":"2024-04-22T01:16:12.192918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:42.265214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198295+0000","last_change":"2024-04-22T01:16:11.257062+0000","last_active":"2024-04-22T01:16:12.198295+0000","last_peered":"2024-04-22T01:16:12.198295+0000","last_clean":"2024-04-22T01:16:12.198295+0000","last_became_active":"2024-04-22T01:16:11.256578+0000","last_became_peered":"2024-04-22T01:16:11.256578+0000","last_unstale":"2024-04-22T01:16:12.198295+0000","last_undegraded":"2024-04-22T01:16:12.198295+0000","last_fullsized":"2024-04-22T01:16:12.198295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:51.252811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421714+0000","last_change":"2024-04-22T01:16:10.205911+0000","last_active":"2024-04-22T01:16:12.421714+0000","last_peered":"2024-04-22T01:16:12.421714+0000","last_clean":"2024-04-22T01:16:12.421714+0000","last_became_active":"2024-04-22T01:16:10.203774+0000","last_became_peered":"2024-04-22T01:16:10.203774+0000","last_unstale":"2024-04-22T01:16:12.421714+0000","last_undegraded":"2024-04-22T01:16:12.421714+0000","last_fullsized":"2024-04-22T01:16:12.421714+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.422243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424795+0000","last_change":"2024-04-22T01:16:11.219149+0000","last_active":"2024-04-22T01:16:12.424795+0000","last_peered":"2024-04-22T01:16:12.424795+0000","last_clean":"2024-04-22T01:16:12.424795+0000","last_became_active":"2024-04-22T01:16:11.218871+0000","last_became_peered":"2024-04-22T01:16:11.218871+0000","last_unstale":"2024-04-22T01:16:12.424795+0000","last_undegraded":"2024-04-22T01:16:12.424795+0000","last_fullsized":"2024-04-22T01:16:12.424795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.364265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193076+0000","last_change":"2024-04-22T01:16:10.186203+0000","last_active":"2024-04-22T01:16:12.193076+0000","last_peered":"2024-04-22T01:16:12.193076+0000","last_clean":"2024-04-22T01:16:12.193076+0000","last_became_active":"2024-04-22T01:16:10.185852+0000","last_became_peered":"2024-04-22T01:16:10.185852+0000","last_unstale":"2024-04-22T01:16:12.193076+0000","last_undegraded":"2024-04-22T01:16:12.193076+0000","last_fullsized":"2024-04-22T01:16:12.193076+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:32.689122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421520+0000","last_change":"2024-04-22T01:16:11.254557+0000","last_active":"2024-04-22T01:16:12.421520+0000","last_peered":"2024-04-22T01:16:12.421520+0000","last_clean":"2024-04-22T01:16:12.421520+0000","last_became_active":"2024-04-22T01:16:11.254331+0000","last_became_peered":"2024-04-22T01:16:11.254331+0000","last_unstale":"2024-04-22T01:16:12.421520+0000","last_undegraded":"2024-04-22T01:16:12.421520+0000","last_fullsized":"2024-04-22T01:16:12.421520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.201139+0000","last_change":"2024-04-22T01:16:10.191928+0000","last_active":"2024-04-22T01:16:12.201139+0000","last_peered":"2024-04-22T01:16:12.201139+0000","last_clean":"2024-04-22T01:16:12.201139+0000","last_became_active":"2024-04-22T01:16:10.191151+0000","last_became_peered":"2024-04-22T01:16:10.191151+0000","last_unstale":"2024-04-22T01:16:12.201139+0000","last_undegraded":"2024-04-22T01:16:12.201139+0000","last_fullsized":"2024-04-22T01:16:12.201139+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:34.113075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422358+0000","last_change":"2024-04-22T01:16:11.256651+0000","last_active":"2024-04-22T01:16:12.422358+0000","last_peered":"2024-04-22T01:16:12.422358+0000","last_clean":"2024-04-22T01:16:12.422358+0000","last_became_active":"2024-04-22T01:16:11.256139+0000","last_became_peered":"2024-04-22T01:16:11.256139+0000","last_unstale":"2024-04-22T01:16:12.422358+0000","last_undegraded":"2024-04-22T01:16:12.422358+0000","last_fullsized":"2024-04-22T01:16:12.422358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:22:04.840889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197039+0000","last_change":"2024-04-22T01:16:10.180657+0000","last_active":"2024-04-22T01:16:12.197039+0000","last_peered":"2024-04-22T01:16:12.197039+0000","last_clean":"2024-04-22T01:16:12.197039+0000","last_became_active":"2024-04-22T01:16:10.179917+0000","last_became_peered":"2024-04-22T01:16:10.179917+0000","last_unstale":"2024-04-22T01:16:12.197039+0000","last_undegraded":"2024-04-22T01:16:12.197039+0000","last_fullsized":"2024-04-22T01:16:12.197039+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:16:13.999843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423478+0000","last_change":"2024-04-22T01:16:11.238333+0000","last_active":"2024-04-22T01:16:12.423478+0000","last_peered":"2024-04-22T01:16:12.423478+0000","last_clean":"2024-04-22T01:16:12.423478+0000","last_became_active":"2024-04-22T01:16:11.238056+0000","last_became_peered":"2024-04-22T01:16:11.238056+0000","last_unstale":"2024-04-22T01:16:12.423478+0000","last_undegraded":"2024-04-22T01:16:12.423478+0000","last_fullsized":"2024-04-22T01:16:12.423478+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:38:12.697995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420823+0000","last_change":"2024-04-22T01:16:10.184592+0000","last_active":"2024-04-22T01:16:12.420823+0000","last_peered":"2024-04-22T01:16:12.420823+0000","last_clean":"2024-04-22T01:16:12.420823+0000","last_became_active":"2024-04-22T01:16:10.184194+0000","last_became_peered":"2024-04-22T01:16:10.184194+0000","last_unstale":"2024-04-22T01:16:12.420823+0000","last_undegraded":"2024-04-22T01:16:12.420823+0000","last_fullsized":"2024-04-22T01:16:12.420823+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420668+0000","last_change":"2024-04-22T01:16:11.208708+0000","last_active":"2024-04-22T01:16:12.420668+0000","last_peered":"2024-04-22T01:16:12.420668+0000","last_clean":"2024-04-22T01:16:12.420668+0000","last_became_active":"2024-04-22T01:16:11.208476+0000","last_became_peered":"2024-04-22T01:16:11.208476+0000","last_unstale":"2024-04-22T01:16:12.420668+0000","last_undegraded":"2024-04-22T01:16:12.420668+0000","last_fullsized":"2024-04-22T01:16:12.420668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:28.631710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421117+0000","last_change":"2024-04-22T01:16:10.196542+0000","last_active":"2024-04-22T01:16:12.421117+0000","last_peered":"2024-04-22T01:16:12.421117+0000","last_clean":"2024-04-22T01:16:12.421117+0000","last_became_active":"2024-04-22T01:16:10.195844+0000","last_became_peered":"2024-04-22T01:16:10.195844+0000","last_unstale":"2024-04-22T01:16:12.421117+0000","last_undegraded":"2024-04-22T01:16:12.421117+0000","last_fullsized":"2024-04-22T01:16:12.421117+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421340+0000","last_change":"2024-04-22T01:16:11.256609+0000","last_active":"2024-04-22T01:16:12.421340+0000","last_peered":"2024-04-22T01:16:12.421340+0000","last_clean":"2024-04-22T01:16:12.421340+0000","last_became_active":"2024-04-22T01:16:11.256326+0000","last_became_peered":"2024-04-22T01:16:11.256326+0000","last_unstale":"2024-04-22T01:16:12.421340+0000","last_undegraded":"2024-04-22T01:16:12.421340+0000","last_fullsized":"2024-04-22T01:16:12.421340+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:22.658275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197860+0000","last_change":"2024-04-22T01:16:10.184129+0000","last_active":"2024-04-22T01:16:12.197860+0000","last_peered":"2024-04-22T01:16:12.197860+0000","last_clean":"2024-04-22T01:16:12.197860+0000","last_became_active":"2024-04-22T01:16:10.183376+0000","last_became_peered":"2024-04-22T01:16:10.183376+0000","last_unstale":"2024-04-22T01:16:12.197860+0000","last_undegraded":"2024-04-22T01:16:12.197860+0000","last_fullsized":"2024-04-22T01:16:12.197860+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:25.359558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419819+0000","last_change":"2024-04-22T01:16:11.218192+0000","last_active":"2024-04-22T01:16:12.419819+0000","last_peered":"2024-04-22T01:16:12.419819+0000","last_clean":"2024-04-22T01:16:12.419819+0000","last_became_active":"2024-04-22T01:16:11.217475+0000","last_became_peered":"2024-04-22T01:16:11.217475+0000","last_unstale":"2024-04-22T01:16:12.419819+0000","last_undegraded":"2024-04-22T01:16:12.419819+0000","last_fullsized":"2024-04-22T01:16:12.419819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419596+0000","last_change":"2024-04-22T01:16:10.205390+0000","last_active":"2024-04-22T01:16:12.419596+0000","last_peered":"2024-04-22T01:16:12.419596+0000","last_clean":"2024-04-22T01:16:12.419596+0000","last_became_active":"2024-04-22T01:16:10.204619+0000","last_became_peered":"2024-04-22T01:16:10.204619+0000","last_unstale":"2024-04-22T01:16:12.419596+0000","last_undegraded":"2024-04-22T01:16:12.419596+0000","last_fullsized":"2024-04-22T01:16:12.419596+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:25.359558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421976+0000","last_change":"2024-04-22T01:16:11.246065+0000","last_active":"2024-04-22T01:16:12.421976+0000","last_peered":"2024-04-22T01:16:12.421976+0000","last_clean":"2024-04-22T01:16:12.421976+0000","last_became_active":"2024-04-22T01:16:11.245480+0000","last_became_peered":"2024-04-22T01:16:11.245480+0000","last_unstale":"2024-04-22T01:16:12.421976+0000","last_undegraded":"2024-04-22T01:16:12.421976+0000","last_fullsized":"2024-04-22T01:16:12.421976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:14.415647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422631+0000","last_change":"2024-04-22T01:16:10.185451+0000","last_active":"2024-04-22T01:16:12.422631+0000","last_peered":"2024-04-22T01:16:12.422631+0000","last_clean":"2024-04-22T01:16:12.422631+0000","last_became_active":"2024-04-22T01:16:10.184984+0000","last_became_peered":"2024-04-22T01:16:10.184984+0000","last_unstale":"2024-04-22T01:16:12.422631+0000","last_undegraded":"2024-04-22T01:16:12.422631+0000","last_fullsized":"2024-04-22T01:16:12.422631+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:30.477033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200754+0000","last_change":"2024-04-22T01:16:11.221372+0000","last_active":"2024-04-22T01:16:12.200754+0000","last_peered":"2024-04-22T01:16:12.200754+0000","last_clean":"2024-04-22T01:16:12.200754+0000","last_became_active":"2024-04-22T01:16:11.213113+0000","last_became_peered":"2024-04-22T01:16:11.213113+0000","last_unstale":"2024-04-22T01:16:12.200754+0000","last_undegraded":"2024-04-22T01:16:12.200754+0000","last_fullsized":"2024-04-22T01:16:12.200754+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:42.062614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421859+0000","last_change":"2024-04-22T01:16:10.206528+0000","last_active":"2024-04-22T01:16:12.421859+0000","last_peered":"2024-04-22T01:16:12.421859+0000","last_clean":"2024-04-22T01:16:12.421859+0000","last_became_active":"2024-04-22T01:16:10.205811+0000","last_became_peered":"2024-04-22T01:16:10.205811+0000","last_unstale":"2024-04-22T01:16:12.421859+0000","last_undegraded":"2024-04-22T01:16:12.421859+0000","last_fullsized":"2024-04-22T01:16:12.421859+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":201,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421172+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:16:12.421172+0000","last_peered":"2024-04-22T01:16:12.421172+0000","last_clean":"2024-04-22T01:16:12.421172+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:16:12.421172+0000","last_undegraded":"2024-04-22T01:16:12.421172+0000","last_fullsized":"2024-04-22T01:16:12.421172+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":782078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31632,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":454656,"data_stored":191674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31616,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340224,"statfs":{"total":96636764160,"available":96604389376,"internally_reserved":0,"allocated":442368,"data_stored":190098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":442368,"data_stored":190118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1036288,"data_stored":780430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:16:15.247 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:16:15.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.391+0000 7f300d807640 1 Processor -- start 2024-04-22T01:16:15.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.391+0000 7f300d807640 1 -- start start 2024-04-22T01:16:15.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f300d807640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f300815aa30 0x7f30080b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f300d807640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300806dbd0 con 0x7f30080aee40 2024-04-22T01:16:15.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f300d807640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300806de70 con 0x7f300815bb50 2024-04-22T01:16:15.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f300d807640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f300806e150 con 0x7f300815aa30 2024-04-22T01:16:15.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f30080aee40 0x7f30080b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33136/0 (socket says 172.21.15.70:33136) 2024-04-22T01:16:15.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 -- 172.21.15.70:0/3299130200 learned_addr learned my addr 172.21.15.70:0/3299130200 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796511779 0 0) 0x7f300806dbd0 con 0x7f30080aee40 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ff0003530 con 0x7f30080aee40 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3006ffd640 1 --2- 172.21.15.70:0/3299130200 >> v2:172.21.15.98:3300/0 conn(0x7f300815aa30 0x7f30080b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2157138625 0 0) 0x7f2ff0003530 con 0x7f30080aee40 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 >> v2:172.21.15.98:3300/0 conn(0x7f300815aa30 msgr2=0x7f30080b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 --2- 172.21.15.70:0/3299130200 >> v2:172.21.15.98:3300/0 conn(0x7f300815aa30 0x7f30080b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 >> v1:172.21.15.98:6789/0 conn(0x7f300815bb50 legacy=0x7f30080ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300806e430 con 0x7f30080aee40 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3000001a20 con 0x7f30080aee40 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 >> v1:172.21.15.70:6789/0 conn(0x7f30080aee40 legacy=0x7f30080b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff00039c0 0x7f2ff0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff0004810 0x7f2ff0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff00051d0 0x7f2ff000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f300806e150 con 0x7f2ff00051d0 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff000c070 con 0x7f2ff00039c0 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff000c270 con 0x7f2ff0004810 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff00051d0 0x7f2ff000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30067fc640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff0004810 0x7f2ff0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f3006ffd640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff00039c0 0x7f2ff0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 -- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff0004810 msgr2=0x7f2ff0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff0004810 0x7f2ff0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 -- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff00039c0 msgr2=0x7f2ff0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff00039c0 0x7f2ff0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.395+0000 7f30077fe640 1 -- 172.21.15.70:0/3299130200 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff0003530 con 0x7f2ff00051d0 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30077fe640 1 -- 172.21.15.70:0/3299130200 reap_dead start 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30077fe640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff00051d0 0x7f2ff000b9d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f300806dda0 tx=0x7f300001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c90da9021e378f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300000a4a0 con 0x7f2ff00051d0 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30000ad650 con 0x7f2ff00051d0 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f3005ffb640 1 -- 172.21.15.70:0/3299130200 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30000ada40 con 0x7f2ff00051d0 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- 172.21.15.70:0/3299130200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff00051d0 msgr2=0x7f2ff000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff00051d0 0x7f2ff000b9d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f300806dda0 tx=0x7f300001d320 comp rx=0 tx=0).stop 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- 172.21.15.70:0/3299130200 shutdown_connections 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 --2- 172.21.15.70:0/3299130200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff00051d0 0x7f2ff000b9d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- 172.21.15.70:0/3299130200 >> 172.21.15.70:0/3299130200 conn(0x7f300806b1f0 msgr2=0x7f300806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:15.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- 172.21.15.70:0/3299130200 shutdown_connections 2024-04-22T01:16:15.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- 172.21.15.70:0/3299130200 wait complete. 2024-04-22T01:16:15.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 Processor -- start 2024-04-22T01:16:15.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- start start 2024-04-22T01:16:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff00051d0 0x7f3008152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 0x7f30080acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30080aff50 0x7f30080ad110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f300815af90 con 0x7f30080afad0 2024-04-22T01:16:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f300815ae10 con 0x7f30080aff50 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f300806e950 con 0x7f2ff00051d0 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 0x7f30080acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 0x7f30080acbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38384/0 (socket says 172.21.15.70:38384) 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 -- 172.21.15.70:0/1056451700 learned_addr learned my addr 172.21.15.70:0/1056451700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f3006ffd640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff00051d0 0x7f3008152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 -- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff00051d0 msgr2=0x7f3008152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff00051d0 0x7f3008152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30077fe640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30080aff50 0x7f30080ad110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 -- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30080aff50 msgr2=0x7f30080ad110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30080aff50 0x7f30080ad110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 -- 172.21.15.70:0/1056451700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30080ac130 con 0x7f30080afad0 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f30067fc640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 0x7f30080acbd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00e4b0 tx=0x7f2ffc00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc0052e0 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ffc018070 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.399+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30080a9140 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc005480 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30080ac3d0 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3008068210 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2ffc003840 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f2feffff640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ff40678f0 0x7f2ff4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f3006ffd640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ff40678f0 0x7f2ff4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f2ffc085790 con 0x7f30080afad0 2024-04-22T01:16:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.403+0000 7f3006ffd640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ff40678f0 0x7f2ff4069db0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80098f0 tx=0x7f2ff8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.407+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2ffc056140 con 0x7f30080afad0 2024-04-22T01:16:15.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.559+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f300806f190 con 0x7f2ff40678f0 2024-04-22T01:16:15.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.575+0000 7f2feffff640 1 -- 172.21.15.70:0/1056451700 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251965 (secure 0 0 0) 0x7f300806f190 con 0x7f2ff40678f0 2024-04-22T01:16:15.577 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ff40678f0 msgr2=0x7f2ff4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ff40678f0 0x7f2ff4069db0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80098f0 tx=0x7f2ff8008040 comp rx=0 tx=0).stop 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 msgr2=0x7f30080acbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 0x7f30080acbd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00e4b0 tx=0x7f2ffc00e970 comp rx=0 tx=0).stop 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 shutdown_connections 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ff40678f0 0x7f2ff4069db0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30080aff50 0x7f30080ad110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30080afad0 0x7f30080acbd0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 --2- 172.21.15.70:0/1056451700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ff00051d0 0x7f3008152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 >> 172.21.15.70:0/1056451700 conn(0x7f300806b1f0 msgr2=0x7f300815cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 shutdown_connections 2024-04-22T01:16:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.579+0000 7f300d807640 1 -- 172.21.15.70:0/1056451700 wait complete. 2024-04-22T01:16:15.594 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-22T01:16:14.614745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241636,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733084,"statfs":{"total":773094113280,"available":772846678016,"internally_reserved":0,"allocated":4759552,"data_stored":2705780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.455847"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423881+0000","last_change":"2024-04-22T01:16:11.220294+0000","last_active":"2024-04-22T01:16:12.423881+0000","last_peered":"2024-04-22T01:16:12.423881+0000","last_clean":"2024-04-22T01:16:12.423881+0000","last_became_active":"2024-04-22T01:16:11.215809+0000","last_became_peered":"2024-04-22T01:16:11.215809+0000","last_unstale":"2024-04-22T01:16:12.423881+0000","last_undegraded":"2024-04-22T01:16:12.423881+0000","last_fullsized":"2024-04-22T01:16:12.423881+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:14.623707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423030+0000","last_change":"2024-04-22T01:16:11.212938+0000","last_active":"2024-04-22T01:16:12.423030+0000","last_peered":"2024-04-22T01:16:12.423030+0000","last_clean":"2024-04-22T01:16:12.423030+0000","last_became_active":"2024-04-22T01:16:11.212636+0000","last_became_peered":"2024-04-22T01:16:11.212636+0000","last_unstale":"2024-04-22T01:16:12.423030+0000","last_undegraded":"2024-04-22T01:16:12.423030+0000","last_fullsized":"2024-04-22T01:16:12.423030+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:21.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420878+0000","last_change":"2024-04-22T01:16:11.253136+0000","last_active":"2024-04-22T01:16:12.420878+0000","last_peered":"2024-04-22T01:16:12.420878+0000","last_clean":"2024-04-22T01:16:12.420878+0000","last_became_active":"2024-04-22T01:16:11.252935+0000","last_became_peered":"2024-04-22T01:16:11.252935+0000","last_unstale":"2024-04-22T01:16:12.420878+0000","last_undegraded":"2024-04-22T01:16:12.420878+0000","last_fullsized":"2024-04-22T01:16:12.420878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193295+0000","last_change":"2024-04-22T01:16:11.209314+0000","last_active":"2024-04-22T01:16:12.193295+0000","last_peered":"2024-04-22T01:16:12.193295+0000","last_clean":"2024-04-22T01:16:12.193295+0000","last_became_active":"2024-04-22T01:16:11.208990+0000","last_became_peered":"2024-04-22T01:16:11.208990+0000","last_unstale":"2024-04-22T01:16:12.193295+0000","last_undegraded":"2024-04-22T01:16:12.193295+0000","last_fullsized":"2024-04-22T01:16:12.193295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:26.370977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193014+0000","last_change":"2024-04-22T01:16:11.208551+0000","last_active":"2024-04-22T01:16:12.193014+0000","last_peered":"2024-04-22T01:16:12.193014+0000","last_clean":"2024-04-22T01:16:12.193014+0000","last_became_active":"2024-04-22T01:16:11.208269+0000","last_became_peered":"2024-04-22T01:16:11.208269+0000","last_unstale":"2024-04-22T01:16:12.193014+0000","last_undegraded":"2024-04-22T01:16:12.193014+0000","last_fullsized":"2024-04-22T01:16:12.193014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:44:02.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421400+0000","last_change":"2024-04-22T01:16:11.210799+0000","last_active":"2024-04-22T01:16:12.421400+0000","last_peered":"2024-04-22T01:16:12.421400+0000","last_clean":"2024-04-22T01:16:12.421400+0000","last_became_active":"2024-04-22T01:16:11.210504+0000","last_became_peered":"2024-04-22T01:16:11.210504+0000","last_unstale":"2024-04-22T01:16:12.421400+0000","last_undegraded":"2024-04-22T01:16:12.421400+0000","last_fullsized":"2024-04-22T01:16:12.421400+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198152+0000","last_change":"2024-04-22T01:16:11.220918+0000","last_active":"2024-04-22T01:16:12.198152+0000","last_peered":"2024-04-22T01:16:12.198152+0000","last_clean":"2024-04-22T01:16:12.198152+0000","last_became_active":"2024-04-22T01:16:11.213351+0000","last_became_peered":"2024-04-22T01:16:11.213351+0000","last_unstale":"2024-04-22T01:16:12.198152+0000","last_undegraded":"2024-04-22T01:16:12.198152+0000","last_fullsized":"2024-04-22T01:16:12.198152+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:21.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422567+0000","last_change":"2024-04-22T01:16:11.249656+0000","last_active":"2024-04-22T01:16:12.422567+0000","last_peered":"2024-04-22T01:16:12.422567+0000","last_clean":"2024-04-22T01:16:12.422567+0000","last_became_active":"2024-04-22T01:16:11.249298+0000","last_became_peered":"2024-04-22T01:16:11.249298+0000","last_unstale":"2024-04-22T01:16:12.422567+0000","last_undegraded":"2024-04-22T01:16:12.422567+0000","last_fullsized":"2024-04-22T01:16:12.422567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:48.784466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421131+0000","last_change":"2024-04-22T01:16:11.249480+0000","last_active":"2024-04-22T01:16:12.421131+0000","last_peered":"2024-04-22T01:16:12.421131+0000","last_clean":"2024-04-22T01:16:12.421131+0000","last_became_active":"2024-04-22T01:16:11.249282+0000","last_became_peered":"2024-04-22T01:16:11.249282+0000","last_unstale":"2024-04-22T01:16:12.421131+0000","last_undegraded":"2024-04-22T01:16:12.421131+0000","last_fullsized":"2024-04-22T01:16:12.421131+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:01.750620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200689+0000","last_change":"2024-04-22T01:16:11.247990+0000","last_active":"2024-04-22T01:16:12.200689+0000","last_peered":"2024-04-22T01:16:12.200689+0000","last_clean":"2024-04-22T01:16:12.200689+0000","last_became_active":"2024-04-22T01:16:11.247308+0000","last_became_peered":"2024-04-22T01:16:11.247308+0000","last_unstale":"2024-04-22T01:16:12.200689+0000","last_undegraded":"2024-04-22T01:16:12.200689+0000","last_fullsized":"2024-04-22T01:16:12.200689+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:43.138333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200404+0000","last_change":"2024-04-22T01:16:11.233076+0000","last_active":"2024-04-22T01:16:12.200404+0000","last_peered":"2024-04-22T01:16:12.200404+0000","last_clean":"2024-04-22T01:16:12.200404+0000","last_became_active":"2024-04-22T01:16:11.212081+0000","last_became_peered":"2024-04-22T01:16:11.212081+0000","last_unstale":"2024-04-22T01:16:12.200404+0000","last_undegraded":"2024-04-22T01:16:12.200404+0000","last_fullsized":"2024-04-22T01:16:12.200404+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:25.849341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421539+0000","last_change":"2024-04-22T01:16:11.211992+0000","last_active":"2024-04-22T01:16:12.421539+0000","last_peered":"2024-04-22T01:16:12.421539+0000","last_clean":"2024-04-22T01:16:12.421539+0000","last_became_active":"2024-04-22T01:16:11.211717+0000","last_became_peered":"2024-04-22T01:16:11.211717+0000","last_unstale":"2024-04-22T01:16:12.421539+0000","last_undegraded":"2024-04-22T01:16:12.421539+0000","last_fullsized":"2024-04-22T01:16:12.421539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:40.324674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198174+0000","last_change":"2024-04-22T01:16:11.233217+0000","last_active":"2024-04-22T01:16:12.198174+0000","last_peered":"2024-04-22T01:16:12.198174+0000","last_clean":"2024-04-22T01:16:12.198174+0000","last_became_active":"2024-04-22T01:16:11.232469+0000","last_became_peered":"2024-04-22T01:16:11.232469+0000","last_unstale":"2024-04-22T01:16:12.198174+0000","last_undegraded":"2024-04-22T01:16:12.198174+0000","last_fullsized":"2024-04-22T01:16:12.198174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:14.415647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419725+0000","last_change":"2024-04-22T01:16:11.257328+0000","last_active":"2024-04-22T01:16:12.419725+0000","last_peered":"2024-04-22T01:16:12.419725+0000","last_clean":"2024-04-22T01:16:12.419725+0000","last_became_active":"2024-04-22T01:16:11.256896+0000","last_became_peered":"2024-04-22T01:16:11.256896+0000","last_unstale":"2024-04-22T01:16:12.419725+0000","last_undegraded":"2024-04-22T01:16:12.419725+0000","last_fullsized":"2024-04-22T01:16:12.419725+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198679+0000","last_change":"2024-04-22T01:16:11.219791+0000","last_active":"2024-04-22T01:16:12.198679+0000","last_peered":"2024-04-22T01:16:12.198679+0000","last_clean":"2024-04-22T01:16:12.198679+0000","last_became_active":"2024-04-22T01:16:11.212232+0000","last_became_peered":"2024-04-22T01:16:11.212232+0000","last_unstale":"2024-04-22T01:16:12.198679+0000","last_undegraded":"2024-04-22T01:16:12.198679+0000","last_fullsized":"2024-04-22T01:16:12.198679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:14.623707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421006+0000","last_change":"2024-04-22T01:16:11.260182+0000","last_active":"2024-04-22T01:16:12.421006+0000","last_peered":"2024-04-22T01:16:12.421006+0000","last_clean":"2024-04-22T01:16:12.421006+0000","last_became_active":"2024-04-22T01:16:11.259537+0000","last_became_peered":"2024-04-22T01:16:11.259537+0000","last_unstale":"2024-04-22T01:16:12.421006+0000","last_undegraded":"2024-04-22T01:16:12.421006+0000","last_fullsized":"2024-04-22T01:16:12.421006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422162+0000","last_change":"2024-04-22T01:16:11.256851+0000","last_active":"2024-04-22T01:16:12.422162+0000","last_peered":"2024-04-22T01:16:12.422162+0000","last_clean":"2024-04-22T01:16:12.422162+0000","last_became_active":"2024-04-22T01:16:11.256278+0000","last_became_peered":"2024-04-22T01:16:11.256278+0000","last_unstale":"2024-04-22T01:16:12.422162+0000","last_undegraded":"2024-04-22T01:16:12.422162+0000","last_fullsized":"2024-04-22T01:16:12.422162+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:38:12.697995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199708+0000","last_change":"2024-04-22T01:16:11.244336+0000","last_active":"2024-04-22T01:16:12.199708+0000","last_peered":"2024-04-22T01:16:12.199708+0000","last_clean":"2024-04-22T01:16:12.199708+0000","last_became_active":"2024-04-22T01:16:11.243637+0000","last_became_peered":"2024-04-22T01:16:11.243637+0000","last_unstale":"2024-04-22T01:16:12.199708+0000","last_undegraded":"2024-04-22T01:16:12.199708+0000","last_fullsized":"2024-04-22T01:16:12.199708+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193556+0000","last_change":"2024-04-22T01:16:11.206006+0000","last_active":"2024-04-22T01:16:12.193556+0000","last_peered":"2024-04-22T01:16:12.193556+0000","last_clean":"2024-04-22T01:16:12.193556+0000","last_became_active":"2024-04-22T01:16:11.205731+0000","last_became_peered":"2024-04-22T01:16:11.205731+0000","last_unstale":"2024-04-22T01:16:12.193556+0000","last_undegraded":"2024-04-22T01:16:12.193556+0000","last_fullsized":"2024-04-22T01:16:12.193556+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:22.720037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199110+0000","last_change":"2024-04-22T01:16:11.255648+0000","last_active":"2024-04-22T01:16:12.199110+0000","last_peered":"2024-04-22T01:16:12.199110+0000","last_clean":"2024-04-22T01:16:12.199110+0000","last_became_active":"2024-04-22T01:16:11.255089+0000","last_became_peered":"2024-04-22T01:16:11.255089+0000","last_unstale":"2024-04-22T01:16:12.199110+0000","last_undegraded":"2024-04-22T01:16:12.199110+0000","last_fullsized":"2024-04-22T01:16:12.199110+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:00.140586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198426+0000","last_change":"2024-04-22T01:16:11.213176+0000","last_active":"2024-04-22T01:16:12.198426+0000","last_peered":"2024-04-22T01:16:12.198426+0000","last_clean":"2024-04-22T01:16:12.198426+0000","last_became_active":"2024-04-22T01:16:11.212457+0000","last_became_peered":"2024-04-22T01:16:11.212457+0000","last_unstale":"2024-04-22T01:16:12.198426+0000","last_undegraded":"2024-04-22T01:16:12.198426+0000","last_fullsized":"2024-04-22T01:16:12.198426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:38:12.697995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420583+0000","last_change":"2024-04-22T01:16:11.205942+0000","last_active":"2024-04-22T01:16:12.420583+0000","last_peered":"2024-04-22T01:16:12.420583+0000","last_clean":"2024-04-22T01:16:12.420583+0000","last_became_active":"2024-04-22T01:16:11.204590+0000","last_became_peered":"2024-04-22T01:16:11.204590+0000","last_unstale":"2024-04-22T01:16:12.420583+0000","last_undegraded":"2024-04-22T01:16:12.420583+0000","last_fullsized":"2024-04-22T01:16:12.420583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:28.631710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421029+0000","last_change":"2024-04-22T01:16:11.217798+0000","last_active":"2024-04-22T01:16:12.421029+0000","last_peered":"2024-04-22T01:16:12.421029+0000","last_clean":"2024-04-22T01:16:12.421029+0000","last_became_active":"2024-04-22T01:16:11.216732+0000","last_became_peered":"2024-04-22T01:16:11.216732+0000","last_unstale":"2024-04-22T01:16:12.421029+0000","last_undegraded":"2024-04-22T01:16:12.421029+0000","last_fullsized":"2024-04-22T01:16:12.421029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:01.750620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.192887+0000","last_change":"2024-04-22T01:16:11.210915+0000","last_active":"2024-04-22T01:16:12.192887+0000","last_peered":"2024-04-22T01:16:12.192887+0000","last_clean":"2024-04-22T01:16:12.192887+0000","last_became_active":"2024-04-22T01:16:11.210630+0000","last_became_peered":"2024-04-22T01:16:11.210630+0000","last_unstale":"2024-04-22T01:16:12.192887+0000","last_undegraded":"2024-04-22T01:16:12.192887+0000","last_fullsized":"2024-04-22T01:16:12.192887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:20.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199985+0000","last_change":"2024-04-22T01:16:11.244186+0000","last_active":"2024-04-22T01:16:12.199985+0000","last_peered":"2024-04-22T01:16:12.199985+0000","last_clean":"2024-04-22T01:16:12.199985+0000","last_became_active":"2024-04-22T01:16:11.243443+0000","last_became_peered":"2024-04-22T01:16:11.243443+0000","last_unstale":"2024-04-22T01:16:12.199985+0000","last_undegraded":"2024-04-22T01:16:12.199985+0000","last_fullsized":"2024-04-22T01:16:12.199985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422955+0000","last_change":"2024-04-22T01:16:11.254761+0000","last_active":"2024-04-22T01:16:12.422955+0000","last_peered":"2024-04-22T01:16:12.422955+0000","last_clean":"2024-04-22T01:16:12.422955+0000","last_became_active":"2024-04-22T01:16:11.254492+0000","last_became_peered":"2024-04-22T01:16:11.254492+0000","last_unstale":"2024-04-22T01:16:12.422955+0000","last_undegraded":"2024-04-22T01:16:12.422955+0000","last_fullsized":"2024-04-22T01:16:12.422955+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:16.781837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199230+0000","last_change":"2024-04-22T01:16:11.255259+0000","last_active":"2024-04-22T01:16:12.199230+0000","last_peered":"2024-04-22T01:16:12.199230+0000","last_clean":"2024-04-22T01:16:12.199230+0000","last_became_active":"2024-04-22T01:16:11.254822+0000","last_became_peered":"2024-04-22T01:16:11.254822+0000","last_unstale":"2024-04-22T01:16:12.199230+0000","last_undegraded":"2024-04-22T01:16:12.199230+0000","last_fullsized":"2024-04-22T01:16:12.199230+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:22.658275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199719+0000","last_change":"2024-04-22T01:16:11.221218+0000","last_active":"2024-04-22T01:16:12.199719+0000","last_peered":"2024-04-22T01:16:12.199719+0000","last_clean":"2024-04-22T01:16:12.199719+0000","last_became_active":"2024-04-22T01:16:11.220474+0000","last_became_peered":"2024-04-22T01:16:11.220474+0000","last_unstale":"2024-04-22T01:16:12.199719+0000","last_undegraded":"2024-04-22T01:16:12.199719+0000","last_fullsized":"2024-04-22T01:16:12.199719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:50.580485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199139+0000","last_change":"2024-04-22T01:16:11.244373+0000","last_active":"2024-04-22T01:16:12.199139+0000","last_peered":"2024-04-22T01:16:12.199139+0000","last_clean":"2024-04-22T01:16:12.199139+0000","last_became_active":"2024-04-22T01:16:11.243875+0000","last_became_peered":"2024-04-22T01:16:11.243875+0000","last_unstale":"2024-04-22T01:16:12.199139+0000","last_undegraded":"2024-04-22T01:16:12.199139+0000","last_fullsized":"2024-04-22T01:16:12.199139+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:58.850159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197651+0000","last_change":"2024-04-22T01:16:11.255896+0000","last_active":"2024-04-22T01:16:12.197651+0000","last_peered":"2024-04-22T01:16:12.197651+0000","last_clean":"2024-04-22T01:16:12.197651+0000","last_became_active":"2024-04-22T01:16:11.255048+0000","last_became_peered":"2024-04-22T01:16:11.255048+0000","last_unstale":"2024-04-22T01:16:12.197651+0000","last_undegraded":"2024-04-22T01:16:12.197651+0000","last_fullsized":"2024-04-22T01:16:12.197651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:06.277212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421508+0000","last_change":"2024-04-22T01:16:11.223414+0000","last_active":"2024-04-22T01:16:12.421508+0000","last_peered":"2024-04-22T01:16:12.421508+0000","last_clean":"2024-04-22T01:16:12.421508+0000","last_became_active":"2024-04-22T01:16:11.217640+0000","last_became_peered":"2024-04-22T01:16:11.217640+0000","last_unstale":"2024-04-22T01:16:12.421508+0000","last_undegraded":"2024-04-22T01:16:12.421508+0000","last_fullsized":"2024-04-22T01:16:12.421508+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:24.822551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422187+0000","last_change":"2024-04-22T01:16:11.213024+0000","last_active":"2024-04-22T01:16:12.422187+0000","last_peered":"2024-04-22T01:16:12.422187+0000","last_clean":"2024-04-22T01:16:12.422187+0000","last_became_active":"2024-04-22T01:16:11.212176+0000","last_became_peered":"2024-04-22T01:16:11.212176+0000","last_unstale":"2024-04-22T01:16:12.422187+0000","last_undegraded":"2024-04-22T01:16:12.422187+0000","last_fullsized":"2024-04-22T01:16:12.422187+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:28.631710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421902+0000","last_change":"2024-04-22T01:16:11.254915+0000","last_active":"2024-04-22T01:16:12.421902+0000","last_peered":"2024-04-22T01:16:12.421902+0000","last_clean":"2024-04-22T01:16:12.421902+0000","last_became_active":"2024-04-22T01:16:11.254468+0000","last_became_peered":"2024-04-22T01:16:11.254468+0000","last_unstale":"2024-04-22T01:16:12.421902+0000","last_undegraded":"2024-04-22T01:16:12.421902+0000","last_fullsized":"2024-04-22T01:16:12.421902+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:21.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.199860+0000","last_change":"2024-04-22T01:16:10.208983+0000","last_active":"2024-04-22T01:16:12.199860+0000","last_peered":"2024-04-22T01:16:12.199860+0000","last_clean":"2024-04-22T01:16:12.199860+0000","last_became_active":"2024-04-22T01:16:10.208262+0000","last_became_peered":"2024-04-22T01:16:10.208262+0000","last_unstale":"2024-04-22T01:16:12.199860+0000","last_undegraded":"2024-04-22T01:16:12.199860+0000","last_fullsized":"2024-04-22T01:16:12.199860+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.422243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423718+0000","last_change":"2024-04-22T01:16:11.251187+0000","last_active":"2024-04-22T01:16:12.423718+0000","last_peered":"2024-04-22T01:16:12.423718+0000","last_clean":"2024-04-22T01:16:12.423718+0000","last_became_active":"2024-04-22T01:16:11.250593+0000","last_became_peered":"2024-04-22T01:16:11.250593+0000","last_unstale":"2024-04-22T01:16:12.423718+0000","last_undegraded":"2024-04-22T01:16:12.423718+0000","last_fullsized":"2024-04-22T01:16:12.423718+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420852+0000","last_change":"2024-04-22T01:16:10.204605+0000","last_active":"2024-04-22T01:16:12.420852+0000","last_peered":"2024-04-22T01:16:12.420852+0000","last_clean":"2024-04-22T01:16:12.420852+0000","last_became_active":"2024-04-22T01:16:10.204140+0000","last_became_peered":"2024-04-22T01:16:10.204140+0000","last_unstale":"2024-04-22T01:16:12.420852+0000","last_undegraded":"2024-04-22T01:16:12.420852+0000","last_fullsized":"2024-04-22T01:16:12.420852+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:21.708617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420954+0000","last_change":"2024-04-22T01:16:11.246154+0000","last_active":"2024-04-22T01:16:12.420954+0000","last_peered":"2024-04-22T01:16:12.420954+0000","last_clean":"2024-04-22T01:16:12.420954+0000","last_became_active":"2024-04-22T01:16:11.245275+0000","last_became_peered":"2024-04-22T01:16:11.245275+0000","last_unstale":"2024-04-22T01:16:12.420954+0000","last_undegraded":"2024-04-22T01:16:12.420954+0000","last_fullsized":"2024-04-22T01:16:12.420954+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:01.750620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424243+0000","last_change":"2024-04-22T01:16:10.206430+0000","last_active":"2024-04-22T01:16:12.424243+0000","last_peered":"2024-04-22T01:16:12.424243+0000","last_clean":"2024-04-22T01:16:12.424243+0000","last_became_active":"2024-04-22T01:16:10.206160+0000","last_became_peered":"2024-04-22T01:16:10.206160+0000","last_unstale":"2024-04-22T01:16:12.424243+0000","last_undegraded":"2024-04-22T01:16:12.424243+0000","last_fullsized":"2024-04-22T01:16:12.424243+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.422243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423298+0000","last_change":"2024-04-22T01:16:11.257715+0000","last_active":"2024-04-22T01:16:12.423298+0000","last_peered":"2024-04-22T01:16:12.423298+0000","last_clean":"2024-04-22T01:16:12.423298+0000","last_became_active":"2024-04-22T01:16:11.257533+0000","last_became_peered":"2024-04-22T01:16:11.257533+0000","last_unstale":"2024-04-22T01:16:12.423298+0000","last_undegraded":"2024-04-22T01:16:12.423298+0000","last_fullsized":"2024-04-22T01:16:12.423298+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:51.252811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420242+0000","last_change":"2024-04-22T01:16:10.203902+0000","last_active":"2024-04-22T01:16:12.420242+0000","last_peered":"2024-04-22T01:16:12.420242+0000","last_clean":"2024-04-22T01:16:12.420242+0000","last_became_active":"2024-04-22T01:16:10.201785+0000","last_became_peered":"2024-04-22T01:16:10.201785+0000","last_unstale":"2024-04-22T01:16:12.420242+0000","last_undegraded":"2024-04-22T01:16:12.420242+0000","last_fullsized":"2024-04-22T01:16:12.420242+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420234+0000","last_change":"2024-04-22T01:16:11.203592+0000","last_active":"2024-04-22T01:16:12.420234+0000","last_peered":"2024-04-22T01:16:12.420234+0000","last_clean":"2024-04-22T01:16:12.420234+0000","last_became_active":"2024-04-22T01:16:11.203323+0000","last_became_peered":"2024-04-22T01:16:11.203323+0000","last_unstale":"2024-04-22T01:16:12.420234+0000","last_undegraded":"2024-04-22T01:16:12.420234+0000","last_fullsized":"2024-04-22T01:16:12.420234+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198376+0000","last_change":"2024-04-22T01:16:10.191468+0000","last_active":"2024-04-22T01:16:12.198376+0000","last_peered":"2024-04-22T01:16:12.198376+0000","last_clean":"2024-04-22T01:16:12.198376+0000","last_became_active":"2024-04-22T01:16:10.190212+0000","last_became_peered":"2024-04-22T01:16:10.190212+0000","last_unstale":"2024-04-22T01:16:12.198376+0000","last_undegraded":"2024-04-22T01:16:12.198376+0000","last_fullsized":"2024-04-22T01:16:12.198376+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:18.786008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198569+0000","last_change":"2024-04-22T01:16:11.233440+0000","last_active":"2024-04-22T01:16:12.198569+0000","last_peered":"2024-04-22T01:16:12.198569+0000","last_clean":"2024-04-22T01:16:12.198569+0000","last_became_active":"2024-04-22T01:16:11.221648+0000","last_became_peered":"2024-04-22T01:16:11.221648+0000","last_unstale":"2024-04-22T01:16:12.198569+0000","last_undegraded":"2024-04-22T01:16:12.198569+0000","last_fullsized":"2024-04-22T01:16:12.198569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:22:04.840889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421864+0000","last_change":"2024-04-22T01:16:10.213968+0000","last_active":"2024-04-22T01:16:12.421864+0000","last_peered":"2024-04-22T01:16:12.421864+0000","last_clean":"2024-04-22T01:16:12.421864+0000","last_became_active":"2024-04-22T01:16:10.212756+0000","last_became_peered":"2024-04-22T01:16:10.212756+0000","last_unstale":"2024-04-22T01:16:12.421864+0000","last_undegraded":"2024-04-22T01:16:12.421864+0000","last_fullsized":"2024-04-22T01:16:12.421864+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421177+0000","last_change":"2024-04-22T01:16:11.238257+0000","last_active":"2024-04-22T01:16:12.421177+0000","last_peered":"2024-04-22T01:16:12.421177+0000","last_clean":"2024-04-22T01:16:12.421177+0000","last_became_active":"2024-04-22T01:16:11.236300+0000","last_became_peered":"2024-04-22T01:16:11.236300+0000","last_unstale":"2024-04-22T01:16:12.421177+0000","last_undegraded":"2024-04-22T01:16:12.421177+0000","last_fullsized":"2024-04-22T01:16:12.421177+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:40.324674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421629+0000","last_change":"2024-04-22T01:16:10.190343+0000","last_active":"2024-04-22T01:16:12.421629+0000","last_peered":"2024-04-22T01:16:12.421629+0000","last_clean":"2024-04-22T01:16:12.421629+0000","last_became_active":"2024-04-22T01:16:10.189942+0000","last_became_peered":"2024-04-22T01:16:10.189942+0000","last_unstale":"2024-04-22T01:16:12.421629+0000","last_undegraded":"2024-04-22T01:16:12.421629+0000","last_fullsized":"2024-04-22T01:16:12.421629+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424438+0000","last_change":"2024-04-22T01:16:11.219075+0000","last_active":"2024-04-22T01:16:12.424438+0000","last_peered":"2024-04-22T01:16:12.424438+0000","last_clean":"2024-04-22T01:16:12.424438+0000","last_became_active":"2024-04-22T01:16:11.218799+0000","last_became_peered":"2024-04-22T01:16:11.218799+0000","last_unstale":"2024-04-22T01:16:12.424438+0000","last_undegraded":"2024-04-22T01:16:12.424438+0000","last_fullsized":"2024-04-22T01:16:12.424438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:50.580485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419432+0000","last_change":"2024-04-22T01:16:10.203255+0000","last_active":"2024-04-22T01:16:12.419432+0000","last_peered":"2024-04-22T01:16:12.419432+0000","last_clean":"2024-04-22T01:16:12.419432+0000","last_became_active":"2024-04-22T01:16:10.201710+0000","last_became_peered":"2024-04-22T01:16:10.201710+0000","last_unstale":"2024-04-22T01:16:12.419432+0000","last_undegraded":"2024-04-22T01:16:12.419432+0000","last_fullsized":"2024-04-22T01:16:12.419432+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:44:01.704153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193299+0000","last_change":"2024-04-22T01:16:11.201649+0000","last_active":"2024-04-22T01:16:12.193299+0000","last_peered":"2024-04-22T01:16:12.193299+0000","last_clean":"2024-04-22T01:16:12.193299+0000","last_became_active":"2024-04-22T01:16:11.201118+0000","last_became_peered":"2024-04-22T01:16:11.201118+0000","last_unstale":"2024-04-22T01:16:12.193299+0000","last_undegraded":"2024-04-22T01:16:12.193299+0000","last_fullsized":"2024-04-22T01:16:12.193299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:40.146273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424119+0000","last_change":"2024-04-22T01:16:10.184747+0000","last_active":"2024-04-22T01:16:12.424119+0000","last_peered":"2024-04-22T01:16:12.424119+0000","last_clean":"2024-04-22T01:16:12.424119+0000","last_became_active":"2024-04-22T01:16:10.184440+0000","last_became_peered":"2024-04-22T01:16:10.184440+0000","last_unstale":"2024-04-22T01:16:12.424119+0000","last_undegraded":"2024-04-22T01:16:12.424119+0000","last_fullsized":"2024-04-22T01:16:12.424119+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:59.129166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422137+0000","last_change":"2024-04-22T01:16:11.257778+0000","last_active":"2024-04-22T01:16:12.422137+0000","last_peered":"2024-04-22T01:16:12.422137+0000","last_clean":"2024-04-22T01:16:12.422137+0000","last_became_active":"2024-04-22T01:16:11.257288+0000","last_became_peered":"2024-04-22T01:16:11.257288+0000","last_unstale":"2024-04-22T01:16:12.422137+0000","last_undegraded":"2024-04-22T01:16:12.422137+0000","last_fullsized":"2024-04-22T01:16:12.422137+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:51.252811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200874+0000","last_change":"2024-04-22T01:16:10.190858+0000","last_active":"2024-04-22T01:16:12.200874+0000","last_peered":"2024-04-22T01:16:12.200874+0000","last_clean":"2024-04-22T01:16:12.200874+0000","last_became_active":"2024-04-22T01:16:10.189367+0000","last_became_peered":"2024-04-22T01:16:10.189367+0000","last_unstale":"2024-04-22T01:16:12.200874+0000","last_undegraded":"2024-04-22T01:16:12.200874+0000","last_fullsized":"2024-04-22T01:16:12.200874+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:59.129166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420675+0000","last_change":"2024-04-22T01:16:11.214201+0000","last_active":"2024-04-22T01:16:12.420675+0000","last_peered":"2024-04-22T01:16:12.420675+0000","last_clean":"2024-04-22T01:16:12.420675+0000","last_became_active":"2024-04-22T01:16:11.213885+0000","last_became_peered":"2024-04-22T01:16:11.213885+0000","last_unstale":"2024-04-22T01:16:12.420675+0000","last_undegraded":"2024-04-22T01:16:12.420675+0000","last_fullsized":"2024-04-22T01:16:12.420675+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:16.781837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200719+0000","last_change":"2024-04-22T01:16:10.191221+0000","last_active":"2024-04-22T01:16:12.200719+0000","last_peered":"2024-04-22T01:16:12.200719+0000","last_clean":"2024-04-22T01:16:12.200719+0000","last_became_active":"2024-04-22T01:16:10.190302+0000","last_became_peered":"2024-04-22T01:16:10.190302+0000","last_unstale":"2024-04-22T01:16:12.200719+0000","last_undegraded":"2024-04-22T01:16:12.200719+0000","last_fullsized":"2024-04-22T01:16:12.200719+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:05.265793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420696+0000","last_change":"2024-04-22T01:16:11.207820+0000","last_active":"2024-04-22T01:16:12.420696+0000","last_peered":"2024-04-22T01:16:12.420696+0000","last_clean":"2024-04-22T01:16:12.420696+0000","last_became_active":"2024-04-22T01:16:11.207083+0000","last_became_peered":"2024-04-22T01:16:11.207083+0000","last_unstale":"2024-04-22T01:16:12.420696+0000","last_undegraded":"2024-04-22T01:16:12.420696+0000","last_fullsized":"2024-04-22T01:16:12.420696+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:16.781837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424088+0000","last_change":"2024-04-22T01:16:10.207758+0000","last_active":"2024-04-22T01:16:12.424088+0000","last_peered":"2024-04-22T01:16:12.424088+0000","last_clean":"2024-04-22T01:16:12.424088+0000","last_became_active":"2024-04-22T01:16:10.207492+0000","last_became_peered":"2024-04-22T01:16:10.207492+0000","last_unstale":"2024-04-22T01:16:12.424088+0000","last_undegraded":"2024-04-22T01:16:12.424088+0000","last_fullsized":"2024-04-22T01:16:12.424088+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:34.113075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420230+0000","last_change":"2024-04-22T01:16:11.223856+0000","last_active":"2024-04-22T01:16:12.420230+0000","last_peered":"2024-04-22T01:16:12.420230+0000","last_clean":"2024-04-22T01:16:12.420230+0000","last_became_active":"2024-04-22T01:16:11.223412+0000","last_became_peered":"2024-04-22T01:16:11.223412+0000","last_unstale":"2024-04-22T01:16:12.420230+0000","last_undegraded":"2024-04-22T01:16:12.420230+0000","last_fullsized":"2024-04-22T01:16:12.420230+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:56:59.594623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424889+0000","last_change":"2024-04-22T01:16:10.184749+0000","last_active":"2024-04-22T01:16:12.424889+0000","last_peered":"2024-04-22T01:16:12.424889+0000","last_clean":"2024-04-22T01:16:12.424889+0000","last_became_active":"2024-04-22T01:16:10.184477+0000","last_became_peered":"2024-04-22T01:16:10.184477+0000","last_unstale":"2024-04-22T01:16:12.424889+0000","last_undegraded":"2024-04-22T01:16:12.424889+0000","last_fullsized":"2024-04-22T01:16:12.424889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:05.265793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423052+0000","last_change":"2024-04-22T01:16:11.212536+0000","last_active":"2024-04-22T01:16:12.423052+0000","last_peered":"2024-04-22T01:16:12.423052+0000","last_clean":"2024-04-22T01:16:12.423052+0000","last_became_active":"2024-04-22T01:16:11.212248+0000","last_became_peered":"2024-04-22T01:16:11.212248+0000","last_unstale":"2024-04-22T01:16:12.423052+0000","last_undegraded":"2024-04-22T01:16:12.423052+0000","last_fullsized":"2024-04-22T01:16:12.423052+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:14.415647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421651+0000","last_change":"2024-04-22T01:16:10.190411+0000","last_active":"2024-04-22T01:16:12.421651+0000","last_peered":"2024-04-22T01:16:12.421651+0000","last_clean":"2024-04-22T01:16:12.421651+0000","last_became_active":"2024-04-22T01:16:10.190036+0000","last_became_peered":"2024-04-22T01:16:10.190036+0000","last_unstale":"2024-04-22T01:16:12.421651+0000","last_undegraded":"2024-04-22T01:16:12.421651+0000","last_fullsized":"2024-04-22T01:16:12.421651+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:21.708617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421566+0000","last_change":"2024-04-22T01:16:11.216600+0000","last_active":"2024-04-22T01:16:12.421566+0000","last_peered":"2024-04-22T01:16:12.421566+0000","last_clean":"2024-04-22T01:16:12.421566+0000","last_became_active":"2024-04-22T01:16:11.215745+0000","last_became_peered":"2024-04-22T01:16:11.215745+0000","last_unstale":"2024-04-22T01:16:12.421566+0000","last_undegraded":"2024-04-22T01:16:12.421566+0000","last_fullsized":"2024-04-22T01:16:12.421566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:48.784466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197364+0000","last_change":"2024-04-22T01:16:10.210763+0000","last_active":"2024-04-22T01:16:12.197364+0000","last_peered":"2024-04-22T01:16:12.197364+0000","last_clean":"2024-04-22T01:16:12.197364+0000","last_became_active":"2024-04-22T01:16:10.210172+0000","last_became_peered":"2024-04-22T01:16:10.210172+0000","last_unstale":"2024-04-22T01:16:12.197364+0000","last_undegraded":"2024-04-22T01:16:12.197364+0000","last_fullsized":"2024-04-22T01:16:12.197364+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:44:01.704153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424660+0000","last_change":"2024-04-22T01:16:11.211991+0000","last_active":"2024-04-22T01:16:12.424660+0000","last_peered":"2024-04-22T01:16:12.424660+0000","last_clean":"2024-04-22T01:16:12.424660+0000","last_became_active":"2024-04-22T01:16:11.211714+0000","last_became_peered":"2024-04-22T01:16:11.211714+0000","last_unstale":"2024-04-22T01:16:12.424660+0000","last_undegraded":"2024-04-22T01:16:12.424660+0000","last_fullsized":"2024-04-22T01:16:12.424660+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:25.849341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420966+0000","last_change":"2024-04-22T01:16:10.209133+0000","last_active":"2024-04-22T01:16:12.420966+0000","last_peered":"2024-04-22T01:16:12.420966+0000","last_clean":"2024-04-22T01:16:12.420966+0000","last_became_active":"2024-04-22T01:16:10.208380+0000","last_became_peered":"2024-04-22T01:16:10.208380+0000","last_unstale":"2024-04-22T01:16:12.420966+0000","last_undegraded":"2024-04-22T01:16:12.420966+0000","last_fullsized":"2024-04-22T01:16:12.420966+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:21.708617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423818+0000","last_change":"2024-04-22T01:16:11.245265+0000","last_active":"2024-04-22T01:16:12.423818+0000","last_peered":"2024-04-22T01:16:12.423818+0000","last_clean":"2024-04-22T01:16:12.423818+0000","last_became_active":"2024-04-22T01:16:11.244958+0000","last_became_peered":"2024-04-22T01:16:11.244958+0000","last_unstale":"2024-04-22T01:16:12.423818+0000","last_undegraded":"2024-04-22T01:16:12.423818+0000","last_fullsized":"2024-04-22T01:16:12.423818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:22:04.840889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200923+0000","last_change":"2024-04-22T01:16:10.206641+0000","last_active":"2024-04-22T01:16:12.200923+0000","last_peered":"2024-04-22T01:16:12.200923+0000","last_clean":"2024-04-22T01:16:12.200923+0000","last_became_active":"2024-04-22T01:16:10.206018+0000","last_became_peered":"2024-04-22T01:16:10.206018+0000","last_unstale":"2024-04-22T01:16:12.200923+0000","last_undegraded":"2024-04-22T01:16:12.200923+0000","last_fullsized":"2024-04-22T01:16:12.200923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:30.477033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200637+0000","last_change":"2024-04-22T01:16:11.234503+0000","last_active":"2024-04-22T01:16:12.200637+0000","last_peered":"2024-04-22T01:16:12.200637+0000","last_clean":"2024-04-22T01:16:12.200637+0000","last_became_active":"2024-04-22T01:16:11.233736+0000","last_became_peered":"2024-04-22T01:16:11.233736+0000","last_unstale":"2024-04-22T01:16:12.200637+0000","last_undegraded":"2024-04-22T01:16:12.200637+0000","last_fullsized":"2024-04-22T01:16:12.200637+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.364265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421064+0000","last_change":"2024-04-22T01:16:10.204693+0000","last_active":"2024-04-22T01:16:12.421064+0000","last_peered":"2024-04-22T01:16:12.421064+0000","last_clean":"2024-04-22T01:16:12.421064+0000","last_became_active":"2024-04-22T01:16:10.203946+0000","last_became_peered":"2024-04-22T01:16:10.203946+0000","last_unstale":"2024-04-22T01:16:12.421064+0000","last_undegraded":"2024-04-22T01:16:12.421064+0000","last_fullsized":"2024-04-22T01:16:12.421064+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420781+0000","last_change":"2024-04-22T01:16:11.208260+0000","last_active":"2024-04-22T01:16:12.420781+0000","last_peered":"2024-04-22T01:16:12.420781+0000","last_clean":"2024-04-22T01:16:12.420781+0000","last_became_active":"2024-04-22T01:16:11.207403+0000","last_became_peered":"2024-04-22T01:16:11.207403+0000","last_unstale":"2024-04-22T01:16:12.420781+0000","last_undegraded":"2024-04-22T01:16:12.420781+0000","last_fullsized":"2024-04-22T01:16:12.420781+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:19.969013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193367+0000","last_change":"2024-04-22T01:16:10.203958+0000","last_active":"2024-04-22T01:16:12.193367+0000","last_peered":"2024-04-22T01:16:12.193367+0000","last_clean":"2024-04-22T01:16:12.193367+0000","last_became_active":"2024-04-22T01:16:10.203412+0000","last_became_peered":"2024-04-22T01:16:10.203412+0000","last_unstale":"2024-04-22T01:16:12.193367+0000","last_undegraded":"2024-04-22T01:16:12.193367+0000","last_fullsized":"2024-04-22T01:16:12.193367+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:49.884021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193551+0000","last_change":"2024-04-22T01:16:11.212277+0000","last_active":"2024-04-22T01:16:12.193551+0000","last_peered":"2024-04-22T01:16:12.193551+0000","last_clean":"2024-04-22T01:16:12.193551+0000","last_became_active":"2024-04-22T01:16:11.211784+0000","last_became_peered":"2024-04-22T01:16:11.211784+0000","last_unstale":"2024-04-22T01:16:12.193551+0000","last_undegraded":"2024-04-22T01:16:12.193551+0000","last_fullsized":"2024-04-22T01:16:12.193551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:29.629103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421745+0000","last_change":"2024-04-22T01:16:10.208644+0000","last_active":"2024-04-22T01:16:12.421745+0000","last_peered":"2024-04-22T01:16:12.421745+0000","last_clean":"2024-04-22T01:16:12.421745+0000","last_became_active":"2024-04-22T01:16:10.208359+0000","last_became_peered":"2024-04-22T01:16:10.208359+0000","last_unstale":"2024-04-22T01:16:12.421745+0000","last_undegraded":"2024-04-22T01:16:12.421745+0000","last_fullsized":"2024-04-22T01:16:12.421745+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421420+0000","last_change":"2024-04-22T01:16:11.218221+0000","last_active":"2024-04-22T01:16:12.421420+0000","last_peered":"2024-04-22T01:16:12.421420+0000","last_clean":"2024-04-22T01:16:12.421420+0000","last_became_active":"2024-04-22T01:16:11.217469+0000","last_became_peered":"2024-04-22T01:16:11.217469+0000","last_unstale":"2024-04-22T01:16:12.421420+0000","last_undegraded":"2024-04-22T01:16:12.421420+0000","last_fullsized":"2024-04-22T01:16:12.421420+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:40.324674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193091+0000","last_change":"2024-04-22T01:16:10.185392+0000","last_active":"2024-04-22T01:16:12.193091+0000","last_peered":"2024-04-22T01:16:12.193091+0000","last_clean":"2024-04-22T01:16:12.193091+0000","last_became_active":"2024-04-22T01:16:10.185016+0000","last_became_peered":"2024-04-22T01:16:10.185016+0000","last_unstale":"2024-04-22T01:16:12.193091+0000","last_undegraded":"2024-04-22T01:16:12.193091+0000","last_fullsized":"2024-04-22T01:16:12.193091+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:56.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421765+0000","last_change":"2024-04-22T01:16:11.214137+0000","last_active":"2024-04-22T01:16:12.421765+0000","last_peered":"2024-04-22T01:16:12.421765+0000","last_clean":"2024-04-22T01:16:12.421765+0000","last_became_active":"2024-04-22T01:16:11.213802+0000","last_became_peered":"2024-04-22T01:16:11.213802+0000","last_unstale":"2024-04-22T01:16:12.421765+0000","last_undegraded":"2024-04-22T01:16:12.421765+0000","last_fullsized":"2024-04-22T01:16:12.421765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:24.822551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.192918+0000","last_change":"2024-04-22T01:16:10.209151+0000","last_active":"2024-04-22T01:16:12.192918+0000","last_peered":"2024-04-22T01:16:12.192918+0000","last_clean":"2024-04-22T01:16:12.192918+0000","last_became_active":"2024-04-22T01:16:10.208837+0000","last_became_peered":"2024-04-22T01:16:10.208837+0000","last_unstale":"2024-04-22T01:16:12.192918+0000","last_undegraded":"2024-04-22T01:16:12.192918+0000","last_fullsized":"2024-04-22T01:16:12.192918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:42.265214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.198295+0000","last_change":"2024-04-22T01:16:11.257062+0000","last_active":"2024-04-22T01:16:12.198295+0000","last_peered":"2024-04-22T01:16:12.198295+0000","last_clean":"2024-04-22T01:16:12.198295+0000","last_became_active":"2024-04-22T01:16:11.256578+0000","last_became_peered":"2024-04-22T01:16:11.256578+0000","last_unstale":"2024-04-22T01:16:12.198295+0000","last_undegraded":"2024-04-22T01:16:12.198295+0000","last_fullsized":"2024-04-22T01:16:12.198295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:51.252811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421714+0000","last_change":"2024-04-22T01:16:10.205911+0000","last_active":"2024-04-22T01:16:12.421714+0000","last_peered":"2024-04-22T01:16:12.421714+0000","last_clean":"2024-04-22T01:16:12.421714+0000","last_became_active":"2024-04-22T01:16:10.203774+0000","last_became_peered":"2024-04-22T01:16:10.203774+0000","last_unstale":"2024-04-22T01:16:12.421714+0000","last_undegraded":"2024-04-22T01:16:12.421714+0000","last_fullsized":"2024-04-22T01:16:12.421714+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.422243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.424795+0000","last_change":"2024-04-22T01:16:11.219149+0000","last_active":"2024-04-22T01:16:12.424795+0000","last_peered":"2024-04-22T01:16:12.424795+0000","last_clean":"2024-04-22T01:16:12.424795+0000","last_became_active":"2024-04-22T01:16:11.218871+0000","last_became_peered":"2024-04-22T01:16:11.218871+0000","last_unstale":"2024-04-22T01:16:12.424795+0000","last_undegraded":"2024-04-22T01:16:12.424795+0000","last_fullsized":"2024-04-22T01:16:12.424795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.364265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.193076+0000","last_change":"2024-04-22T01:16:10.186203+0000","last_active":"2024-04-22T01:16:12.193076+0000","last_peered":"2024-04-22T01:16:12.193076+0000","last_clean":"2024-04-22T01:16:12.193076+0000","last_became_active":"2024-04-22T01:16:10.185852+0000","last_became_peered":"2024-04-22T01:16:10.185852+0000","last_unstale":"2024-04-22T01:16:12.193076+0000","last_undegraded":"2024-04-22T01:16:12.193076+0000","last_fullsized":"2024-04-22T01:16:12.193076+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:32.689122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421520+0000","last_change":"2024-04-22T01:16:11.254557+0000","last_active":"2024-04-22T01:16:12.421520+0000","last_peered":"2024-04-22T01:16:12.421520+0000","last_clean":"2024-04-22T01:16:12.421520+0000","last_became_active":"2024-04-22T01:16:11.254331+0000","last_became_peered":"2024-04-22T01:16:11.254331+0000","last_unstale":"2024-04-22T01:16:12.421520+0000","last_undegraded":"2024-04-22T01:16:12.421520+0000","last_fullsized":"2024-04-22T01:16:12.421520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.201139+0000","last_change":"2024-04-22T01:16:10.191928+0000","last_active":"2024-04-22T01:16:12.201139+0000","last_peered":"2024-04-22T01:16:12.201139+0000","last_clean":"2024-04-22T01:16:12.201139+0000","last_became_active":"2024-04-22T01:16:10.191151+0000","last_became_peered":"2024-04-22T01:16:10.191151+0000","last_unstale":"2024-04-22T01:16:12.201139+0000","last_undegraded":"2024-04-22T01:16:12.201139+0000","last_fullsized":"2024-04-22T01:16:12.201139+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:34.113075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422358+0000","last_change":"2024-04-22T01:16:11.256651+0000","last_active":"2024-04-22T01:16:12.422358+0000","last_peered":"2024-04-22T01:16:12.422358+0000","last_clean":"2024-04-22T01:16:12.422358+0000","last_became_active":"2024-04-22T01:16:11.256139+0000","last_became_peered":"2024-04-22T01:16:11.256139+0000","last_unstale":"2024-04-22T01:16:12.422358+0000","last_undegraded":"2024-04-22T01:16:12.422358+0000","last_fullsized":"2024-04-22T01:16:12.422358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:22:04.840889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197039+0000","last_change":"2024-04-22T01:16:10.180657+0000","last_active":"2024-04-22T01:16:12.197039+0000","last_peered":"2024-04-22T01:16:12.197039+0000","last_clean":"2024-04-22T01:16:12.197039+0000","last_became_active":"2024-04-22T01:16:10.179917+0000","last_became_peered":"2024-04-22T01:16:10.179917+0000","last_unstale":"2024-04-22T01:16:12.197039+0000","last_undegraded":"2024-04-22T01:16:12.197039+0000","last_fullsized":"2024-04-22T01:16:12.197039+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:16:13.999843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.423478+0000","last_change":"2024-04-22T01:16:11.238333+0000","last_active":"2024-04-22T01:16:12.423478+0000","last_peered":"2024-04-22T01:16:12.423478+0000","last_clean":"2024-04-22T01:16:12.423478+0000","last_became_active":"2024-04-22T01:16:11.238056+0000","last_became_peered":"2024-04-22T01:16:11.238056+0000","last_unstale":"2024-04-22T01:16:12.423478+0000","last_undegraded":"2024-04-22T01:16:12.423478+0000","last_fullsized":"2024-04-22T01:16:12.423478+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:38:12.697995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420823+0000","last_change":"2024-04-22T01:16:10.184592+0000","last_active":"2024-04-22T01:16:12.420823+0000","last_peered":"2024-04-22T01:16:12.420823+0000","last_clean":"2024-04-22T01:16:12.420823+0000","last_became_active":"2024-04-22T01:16:10.184194+0000","last_became_peered":"2024-04-22T01:16:10.184194+0000","last_unstale":"2024-04-22T01:16:12.420823+0000","last_undegraded":"2024-04-22T01:16:12.420823+0000","last_fullsized":"2024-04-22T01:16:12.420823+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:28.617684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.420668+0000","last_change":"2024-04-22T01:16:11.208708+0000","last_active":"2024-04-22T01:16:12.420668+0000","last_peered":"2024-04-22T01:16:12.420668+0000","last_clean":"2024-04-22T01:16:12.420668+0000","last_became_active":"2024-04-22T01:16:11.208476+0000","last_became_peered":"2024-04-22T01:16:11.208476+0000","last_unstale":"2024-04-22T01:16:12.420668+0000","last_undegraded":"2024-04-22T01:16:12.420668+0000","last_fullsized":"2024-04-22T01:16:12.420668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:28.631710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421117+0000","last_change":"2024-04-22T01:16:10.196542+0000","last_active":"2024-04-22T01:16:12.421117+0000","last_peered":"2024-04-22T01:16:12.421117+0000","last_clean":"2024-04-22T01:16:12.421117+0000","last_became_active":"2024-04-22T01:16:10.195844+0000","last_became_peered":"2024-04-22T01:16:10.195844+0000","last_unstale":"2024-04-22T01:16:12.421117+0000","last_undegraded":"2024-04-22T01:16:12.421117+0000","last_fullsized":"2024-04-22T01:16:12.421117+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421340+0000","last_change":"2024-04-22T01:16:11.256609+0000","last_active":"2024-04-22T01:16:12.421340+0000","last_peered":"2024-04-22T01:16:12.421340+0000","last_clean":"2024-04-22T01:16:12.421340+0000","last_became_active":"2024-04-22T01:16:11.256326+0000","last_became_peered":"2024-04-22T01:16:11.256326+0000","last_unstale":"2024-04-22T01:16:12.421340+0000","last_undegraded":"2024-04-22T01:16:12.421340+0000","last_fullsized":"2024-04-22T01:16:12.421340+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:22.658275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.197860+0000","last_change":"2024-04-22T01:16:10.184129+0000","last_active":"2024-04-22T01:16:12.197860+0000","last_peered":"2024-04-22T01:16:12.197860+0000","last_clean":"2024-04-22T01:16:12.197860+0000","last_became_active":"2024-04-22T01:16:10.183376+0000","last_became_peered":"2024-04-22T01:16:10.183376+0000","last_unstale":"2024-04-22T01:16:12.197860+0000","last_undegraded":"2024-04-22T01:16:12.197860+0000","last_fullsized":"2024-04-22T01:16:12.197860+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:25.359558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419819+0000","last_change":"2024-04-22T01:16:11.218192+0000","last_active":"2024-04-22T01:16:12.419819+0000","last_peered":"2024-04-22T01:16:12.419819+0000","last_clean":"2024-04-22T01:16:12.419819+0000","last_became_active":"2024-04-22T01:16:11.217475+0000","last_became_peered":"2024-04-22T01:16:11.217475+0000","last_unstale":"2024-04-22T01:16:12.419819+0000","last_undegraded":"2024-04-22T01:16:12.419819+0000","last_fullsized":"2024-04-22T01:16:12.419819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:06:19.560316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.419596+0000","last_change":"2024-04-22T01:16:10.205390+0000","last_active":"2024-04-22T01:16:12.419596+0000","last_peered":"2024-04-22T01:16:12.419596+0000","last_clean":"2024-04-22T01:16:12.419596+0000","last_became_active":"2024-04-22T01:16:10.204619+0000","last_became_peered":"2024-04-22T01:16:10.204619+0000","last_unstale":"2024-04-22T01:16:12.419596+0000","last_undegraded":"2024-04-22T01:16:12.419596+0000","last_fullsized":"2024-04-22T01:16:12.419596+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:25.359558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421976+0000","last_change":"2024-04-22T01:16:11.246065+0000","last_active":"2024-04-22T01:16:12.421976+0000","last_peered":"2024-04-22T01:16:12.421976+0000","last_clean":"2024-04-22T01:16:12.421976+0000","last_became_active":"2024-04-22T01:16:11.245480+0000","last_became_peered":"2024-04-22T01:16:11.245480+0000","last_unstale":"2024-04-22T01:16:12.421976+0000","last_undegraded":"2024-04-22T01:16:12.421976+0000","last_fullsized":"2024-04-22T01:16:12.421976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:14.415647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.422631+0000","last_change":"2024-04-22T01:16:10.185451+0000","last_active":"2024-04-22T01:16:12.422631+0000","last_peered":"2024-04-22T01:16:12.422631+0000","last_clean":"2024-04-22T01:16:12.422631+0000","last_became_active":"2024-04-22T01:16:10.184984+0000","last_became_peered":"2024-04-22T01:16:10.184984+0000","last_unstale":"2024-04-22T01:16:12.422631+0000","last_undegraded":"2024-04-22T01:16:12.422631+0000","last_fullsized":"2024-04-22T01:16:12.422631+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:30.477033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.200754+0000","last_change":"2024-04-22T01:16:11.221372+0000","last_active":"2024-04-22T01:16:12.200754+0000","last_peered":"2024-04-22T01:16:12.200754+0000","last_clean":"2024-04-22T01:16:12.200754+0000","last_became_active":"2024-04-22T01:16:11.213113+0000","last_became_peered":"2024-04-22T01:16:11.213113+0000","last_unstale":"2024-04-22T01:16:12.200754+0000","last_undegraded":"2024-04-22T01:16:12.200754+0000","last_fullsized":"2024-04-22T01:16:12.200754+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:10.157704+0000","last_clean_scrub_stamp":"2024-04-22T01:16:10.157704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:42.062614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421859+0000","last_change":"2024-04-22T01:16:10.206528+0000","last_active":"2024-04-22T01:16:12.421859+0000","last_peered":"2024-04-22T01:16:12.421859+0000","last_clean":"2024-04-22T01:16:12.421859+0000","last_became_active":"2024-04-22T01:16:10.205811+0000","last_became_peered":"2024-04-22T01:16:10.205811+0000","last_unstale":"2024-04-22T01:16:12.421859+0000","last_undegraded":"2024-04-22T01:16:12.421859+0000","last_fullsized":"2024-04-22T01:16:12.421859+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:09.146285+0000","last_clean_scrub_stamp":"2024-04-22T01:16:09.146285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:39.134853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":201,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T01:16:12.421172+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:16:12.421172+0000","last_peered":"2024-04-22T01:16:12.421172+0000","last_clean":"2024-04-22T01:16:12.421172+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:16:12.421172+0000","last_undegraded":"2024-04-22T01:16:12.421172+0000","last_fullsized":"2024-04-22T01:16:12.421172+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":782078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31632,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":454656,"data_stored":191674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31616,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340224,"statfs":{"total":96636764160,"available":96604389376,"internally_reserved":0,"allocated":442368,"data_stored":190098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":442368,"data_stored":190118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1036288,"data_stored":780430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:16:15.598 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:16:15.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-22T01:16:15.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 Processor -- start 2024-04-22T01:16:15.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 -- start start 2024-04-22T01:16:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f890015a320 0x7f890006f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89000ce420 0x7f8900072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f890006da70 con 0x7f8900071b70 2024-04-22T01:16:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f890006dd50 con 0x7f890015a320 2024-04-22T01:16:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8906a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f890006e030 con 0x7f89000ce420 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88fffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f890015a320 0x7f890006f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88ff7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8900071b70 0x7f89000b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33150/0 (socket says 172.21.15.70:33150) 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88ff7fe640 1 -- 172.21.15.70:0/2778102750 learned_addr learned my addr 172.21.15.70:0/2778102750 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8904f9d640 1 --2- 172.21.15.70:0/2778102750 >> v2:172.21.15.98:3300/0 conn(0x7f89000ce420 0x7f8900072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88feffd640 1 -- 172.21.15.70:0/2778102750 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3722970930 0 0) 0x7f890006da70 con 0x7f8900071b70 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88feffd640 1 -- 172.21.15.70:0/2778102750 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88e8003530 con 0x7f8900071b70 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88fffff640 1 -- 172.21.15.70:0/2778102750 >> v2:172.21.15.98:3300/0 conn(0x7f89000ce420 msgr2=0x7f8900072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88fffff640 1 --2- 172.21.15.70:0/2778102750 >> v2:172.21.15.98:3300/0 conn(0x7f89000ce420 0x7f8900072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88fffff640 1 -- 172.21.15.70:0/2778102750 >> v1:172.21.15.70:6789/0 conn(0x7f8900071b70 legacy=0x7f89000b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f88fffff640 1 -- 172.21.15.70:0/2778102750 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890006e310 con 0x7f890015a320 2024-04-22T01:16:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.767+0000 7f8904f9d640 1 --2- 172.21.15.70:0/2778102750 >> v2:172.21.15.98:3300/0 conn(0x7f89000ce420 0x7f8900072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.771+0000 7f88fffff640 1 --2- 172.21.15.70:0/2778102750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f890015a320 0x7f890006f150 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8900071b00 tx=0x7f88f001d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9a3896af8d136c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.771+0000 7f88feffd640 1 -- 172.21.15.70:0/2778102750 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f00b3070 con 0x7f890015a320 2024-04-22T01:16:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.771+0000 7f88feffd640 1 -- 172.21.15.70:0/2778102750 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f00a67b0 con 0x7f890015a320 2024-04-22T01:16:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.771+0000 7f88feffd640 1 -- 172.21.15.70:0/2778102750 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f000b040 con 0x7f890015a320 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/2778102750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f890015a320 msgr2=0x7f890006f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 --2- 172.21.15.70:0/2778102750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f890015a320 0x7f890006f150 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8900071b00 tx=0x7f88f001d850 comp rx=0 tx=0).stop 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/2778102750 shutdown_connections 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 --2- 172.21.15.70:0/2778102750 >> v2:172.21.15.98:3300/0 conn(0x7f89000ce420 0x7f8900072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 --2- 172.21.15.70:0/2778102750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f890015a320 0x7f890006f150 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/2778102750 >> 172.21.15.70:0/2778102750 conn(0x7f89000b1d40 msgr2=0x7f89000cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/2778102750 shutdown_connections 2024-04-22T01:16:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/2778102750 wait complete. 2024-04-22T01:16:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 Processor -- start 2024-04-22T01:16:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- start start 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8900071b70 0x7f890015d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89000ce420 0x7f89000afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 0x7f89001f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f890006f810 con 0x7f89000b0110 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f890006f690 con 0x7f89000ce420 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f89000b4470 con 0x7f8900071b70 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 0x7f89001f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 0x7f89001f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38408/0 (socket says 172.21.15.70:38408) 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 -- 172.21.15.70:0/4184596574 learned_addr learned my addr 172.21.15.70:0/4184596574 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f88fffff640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8900071b70 0x7f890015d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 -- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8900071b70 msgr2=0x7f890015d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f88ff7fe640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89000ce420 0x7f89000afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8900071b70 0x7f890015d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 -- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89000ce420 msgr2=0x7f89000afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89000ce420 0x7f89000afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 -- 172.21.15.70:0/4184596574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8900133ca0 con 0x7f89000b0110 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8904f9d640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 0x7f89001f4b90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88f8007e70 tx=0x7f88f800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f801a070 con 0x7f89000b0110 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f8003bc0 con 0x7f89000b0110 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8900133f80 con 0x7f89000b0110 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8004030 con 0x7f89000b0110 2024-04-22T01:16:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.775+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8900134540 con 0x7f89000b0110 2024-04-22T01:16:15.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.779+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89001477a0 con 0x7f89000b0110 2024-04-22T01:16:15.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.779+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f88f801d070 con 0x7f89000b0110 2024-04-22T01:16:15.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.779+0000 7f88fcff9640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88ec0678c0 0x7f88ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:15.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.779+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f88f8086110 con 0x7f89000b0110 2024-04-22T01:16:15.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.779+0000 7f88fffff640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88ec0678c0 0x7f88ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:15.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.779+0000 7f88fffff640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88ec0678c0 0x7f88ec069d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88f00b0000 tx=0x7f88f000c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:15.783+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f89001477a0 con 0x7f89000b0110 2024-04-22T01:16:16.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.003+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8900068210 con 0x7f89000b0110 2024-04-22T01:16:16.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f88fcff9640 1 -- 172.21.15.70:0/4184596574 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f88f8010070 con 0x7f89000b0110 2024-04-22T01:16:16.006 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:16:16.006 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:16:16.006 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-22T01:16:16.006 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-22T01:16:16.006 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-22T01:16:16.007 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-22T01:16:16.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88ec0678c0 msgr2=0x7f88ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88ec0678c0 0x7f88ec069d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88f00b0000 tx=0x7f88f000c040 comp rx=0 tx=0).stop 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 msgr2=0x7f89001f4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 0x7f89001f4b90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88f8007e70 tx=0x7f88f800d4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 shutdown_connections 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88ec0678c0 0x7f88ec069d80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89000b0110 0x7f89001f4b90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89000ce420 0x7f89000afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 --2- 172.21.15.70:0/4184596574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8900071b70 0x7f890015d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 >> 172.21.15.70:0/4184596574 conn(0x7f89000b1d40 msgr2=0x7f890006b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 shutdown_connections 2024-04-22T01:16:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.007+0000 7f8906a00640 1 -- 172.21.15.70:0/4184596574 wait complete. 2024-04-22T01:16:16.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.143+0000 7f4b71c97640 1 Processor -- start 2024-04-22T01:16:16.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.143+0000 7f4b71c97640 1 -- start start 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b71c97640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4b6c09a3d0 0x7f4b6c0cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b71c97640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c15f7b0 con 0x7f4b6c15f3d0 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b71c97640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0cd3e0 con 0x7f4b6c09a860 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b71c97640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4b6c0d0ba0 con 0x7f4b6c09a3d0 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4b6c15f3d0 0x7f4b6c099cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33518/0 (socket says 172.21.15.70:33518) 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6b7fe640 1 -- 172.21.15.70:0/3723894682 learned_addr learned my addr 172.21.15.70:0/3723894682 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:16.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3723894682 >> v2:172.21.15.98:3300/0 conn(0x7f4b6c09a3d0 0x7f4b6c0cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2507084278 0 0) 0x7f4b6c15f7b0 con 0x7f4b6c15f3d0 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b54003530 con 0x7f4b6c15f3d0 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2434781711 0 0) 0x7f4b54003530 con 0x7f4b6c15f3d0 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 >> v2:172.21.15.98:3300/0 conn(0x7f4b6c09a3d0 msgr2=0x7f4b6c0cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 --2- 172.21.15.70:0/3723894682 >> v2:172.21.15.98:3300/0 conn(0x7f4b6c09a3d0 0x7f4b6c0cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 >> v1:172.21.15.98:6789/0 conn(0x7f4b6c09a860 legacy=0x7f4b6c0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c0cd6c0 con 0x7f4b6c15f3d0 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f4b5c001680 con 0x7f4b6c15f3d0 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 >> v1:172.21.15.70:6789/0 conn(0x7f4b6c15f3d0 legacy=0x7f4b6c099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b54003a10 0x7f4b54003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54004810 0x7f4b54004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b540051d0 0x7f4b5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5400c0a0 con 0x7f4b54004810 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5400c250 con 0x7f4b54003a10 2024-04-22T01:16:16.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4b5400c450 con 0x7f4b540051d0 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54004810 0x7f4b54004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6b7fe640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b54003a10 0x7f4b54003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 -- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b540051d0 msgr2=0x7f4b5400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6bfff640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b540051d0 0x7f4b5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b540051d0 0x7f4b5400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 -- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b54003a10 msgr2=0x7f4b54003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b54003a10 0x7f4b54003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6affd640 1 -- 172.21.15.70:0/3723894682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b54003530 con 0x7f4b54004810 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6b7fe640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b54003a10 0x7f4b54003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:16.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.147+0000 7f4b6bfff640 1 -- 172.21.15.70:0/3723894682 reap_dead start 2024-04-22T01:16:16.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54004810 0x7f4b54004c90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009ae0 tx=0x7f4b6001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebe4e53851a7121 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600b2070 con 0x7f4b54004810 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b600a5660 con 0x7f4b54004810 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6a7fc640 1 -- 172.21.15.70:0/3723894682 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600096b0 con 0x7f4b54004810 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- 172.21.15.70:0/3723894682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54004810 msgr2=0x7f4b54004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54004810 0x7f4b54004c90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009ae0 tx=0x7f4b6001ca70 comp rx=0 tx=0).stop 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- 172.21.15.70:0/3723894682 shutdown_connections 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3723894682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54004810 0x7f4b54004c90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- 172.21.15.70:0/3723894682 >> 172.21.15.70:0/3723894682 conn(0x7f4b6c0b35d0 msgr2=0x7f4b6c0b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- 172.21.15.70:0/3723894682 shutdown_connections 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- 172.21.15.70:0/3723894682 wait complete. 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 Processor -- start 2024-04-22T01:16:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- start start 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b54004810 0x7f4b6c0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 0x7f4b6c1f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b6c14a620 0x7f4b6c14aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c0cdc80 con 0x7f4b6c14dd50 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c0cd190 con 0x7f4b6c14a620 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b71c97640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6c0b2b50 con 0x7f4b54004810 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 0x7f4b6c1f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 0x7f4b6c1f1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50478/0 (socket says 172.21.15.70:50478) 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 -- 172.21.15.70:0/3080578813 learned_addr learned my addr 172.21.15.70:0/3080578813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 -- 172.21.15.70:0/3080578813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b54004810 msgr2=0x7f4b6c0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b54004810 0x7f4b6c0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 -- 172.21.15.70:0/3080578813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b6c14a620 msgr2=0x7f4b6c14aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b6c14a620 0x7f4b6c14aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 -- 172.21.15.70:0/3080578813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c14afe0 con 0x7f4b6c14dd50 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.151+0000 7f4b6affd640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 0x7f4b6c1f1a70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4b6001c8a0 tx=0x7f4b60019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600b2040 con 0x7f4b6c14dd50 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6000f000 con 0x7f4b6c14dd50 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c1f25f0 con 0x7f4b6c14dd50 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600096b0 con 0x7f4b6c14dd50 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c0ca310 con 0x7f4b6c14dd50 2024-04-22T01:16:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c068920 con 0x7f4b6c14dd50 2024-04-22T01:16:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b600bf030 con 0x7f4b6c14dd50 2024-04-22T01:16:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b53fff640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b580678c0 0x7f4b58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b6b7fe640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b580678c0 0x7f4b58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4b6011fce0 con 0x7f4b6c14dd50 2024-04-22T01:16:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.155+0000 7f4b6b7fe640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b580678c0 0x7f4b58069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0017f0 tx=0x7f4b5c001cf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.159+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4b600c6060 con 0x7f4b6c14dd50 2024-04-22T01:16:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.355+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b6c0d0270 con 0x7f4b6c14dd50 2024-04-22T01:16:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.355+0000 7f4b53fff640 1 -- 172.21.15.70:0/3080578813 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5178 (secure 0 0 0) 0x7f4b60016780 con 0x7f4b6c14dd50 2024-04-22T01:16:16.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:16.356 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:10.524071+0000","modified":"2024-04-22T01:16:13.536979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5478":{"gid":5478,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3248824947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3248824947},{"type":"v1","addr":"172.21.15.70:6837","nonce":3248824947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5478,"qdb_cluster":[5478]},"id":2}]} 2024-04-22T01:16:16.356 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b580678c0 msgr2=0x7f4b58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b580678c0 0x7f4b58069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0017f0 tx=0x7f4b5c001cf0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 msgr2=0x7f4b6c1f1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 0x7f4b6c1f1a70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4b6001c8a0 tx=0x7f4b60019810 comp rx=0 tx=0).stop 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 shutdown_connections 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4b580678c0 0x7f4b58069d80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b6c14a620 0x7f4b6c14aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b6c14dd50 0x7f4b6c1f1a70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 --2- 172.21.15.70:0/3080578813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b54004810 0x7f4b6c0cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 >> 172.21.15.70:0/3080578813 conn(0x7f4b6c0b35d0 msgr2=0x7f4b6c0745a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 shutdown_connections 2024-04-22T01:16:16.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.359+0000 7f4b71c97640 1 -- 172.21.15.70:0/3080578813 wait complete. 2024-04-22T01:16:16.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:16:10.524071+0000', 'modified': '2024-04-22T01:16:13.536979+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5478': {'gid': 5478, 'name': 'c', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/3248824947', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3248824947}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3248824947}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5478, 'qdb_cluster': [5478]} 2024-04-22T01:16:16.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:16:16.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T01:16:16.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 Processor -- start 2024-04-22T01:16:16.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 -- start start 2024-04-22T01:16:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20d41445b0 0x7f20d4072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20d40b4360 0x7f20d40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d4144990 con 0x7f20d40cdce0 2024-04-22T01:16:16.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f20d40b3bd0 con 0x7f20d41445b0 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.907+0000 7f20d9fbd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f20d40b2b50 con 0x7f20d40b4360 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20d41445b0 0x7f20d4072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20d41445b0 0x7f20d4072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50488/0 (socket says 172.21.15.70:50488) 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 -- 172.21.15.70:0/4087788000 learned_addr learned my addr 172.21.15.70:0/4087788000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/4087788000 >> v2:172.21.15.98:3300/0 conn(0x7f20d40b4360 0x7f20d40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 -- 172.21.15.70:0/4087788000 >> v2:172.21.15.98:3300/0 conn(0x7f20d40b4360 msgr2=0x7f20d40b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 --2- 172.21.15.70:0/4087788000 >> v2:172.21.15.98:3300/0 conn(0x7f20d40b4360 0x7f20d40b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 -- 172.21.15.70:0/4087788000 >> v1:172.21.15.98:6789/0 conn(0x7f20d40cdce0 legacy=0x7f20d40ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 -- 172.21.15.70:0/4087788000 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d40b2e10 con 0x7f20d41445b0 2024-04-22T01:16:16.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/4087788000 >> v2:172.21.15.98:3300/0 conn(0x7f20d40b4360 0x7f20d40b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:16.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d37fe640 1 --2- 172.21.15.70:0/4087788000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d41445b0 0x7f20d4072500 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f20d40718b0 tx=0x7f20c401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a8aae30a343471 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d27fc640 1 -- 172.21.15.70:0/4087788000 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40b1070 con 0x7f20d41445b0 2024-04-22T01:16:16.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d27fc640 1 -- 172.21.15.70:0/4087788000 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c40ac640 con 0x7f20d41445b0 2024-04-22T01:16:16.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d27fc640 1 -- 172.21.15.70:0/4087788000 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40aca10 con 0x7f20d41445b0 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 -- 172.21.15.70:0/4087788000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d41445b0 msgr2=0x7f20d4072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/4087788000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d41445b0 0x7f20d4072500 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f20d40718b0 tx=0x7f20c401c520 comp rx=0 tx=0).stop 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 -- 172.21.15.70:0/4087788000 shutdown_connections 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/4087788000 >> v2:172.21.15.98:3300/0 conn(0x7f20d40b4360 0x7f20d40b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/4087788000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d41445b0 0x7f20d4072500 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 -- 172.21.15.70:0/4087788000 >> 172.21.15.70:0/4087788000 conn(0x7f20d40cf9e0 msgr2=0x7f20d40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 -- 172.21.15.70:0/4087788000 shutdown_connections 2024-04-22T01:16:16.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.911+0000 7f20d9fbd640 1 -- 172.21.15.70:0/4087788000 wait complete. 2024-04-22T01:16:16.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 Processor -- start 2024-04-22T01:16:16.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 -- start start 2024-04-22T01:16:16.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 0x7f20d413aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20d40cdce0 0x7f20d413bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20d41445b0 0x7f20d4152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20d406edb0 con 0x7f20d40b4360 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4072a40 con 0x7f20d41445b0 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f20d406ef30 con 0x7f20d40cdce0 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 0x7f20d413aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 0x7f20d413aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50494/0 (socket says 172.21.15.70:50494) 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 -- 172.21.15.70:0/1155720994 learned_addr learned my addr 172.21.15.70:0/1155720994 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20d40cdce0 0x7f20d413bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d3fff640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20d41445b0 0x7f20d4152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 -- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20d40cdce0 msgr2=0x7f20d413bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20d40cdce0 0x7f20d413bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 -- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20d41445b0 msgr2=0x7f20d4152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20d41445b0 0x7f20d4152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 -- 172.21.15.70:0/1155720994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d41584b0 con 0x7f20d40b4360 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d37fe640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 0x7f20d413aad0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f20c4019130 tx=0x7f20c40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20bbfff640 1 -- 172.21.15.70:0/1155720994 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40bb070 con 0x7f20d40b4360 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20bbfff640 1 -- 172.21.15.70:0/1155720994 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c40b5620 con 0x7f20d40b4360 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d4157410 con 0x7f20d40b4360 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20bbfff640 1 -- 172.21.15.70:0/1155720994 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40b1040 con 0x7f20d40b4360 2024-04-22T01:16:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.915+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d41579d0 con 0x7f20d40b4360 2024-04-22T01:16:16.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20bbfff640 1 -- 172.21.15.70:0/1155720994 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f20c40be070 con 0x7f20d40b4360 2024-04-22T01:16:16.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20bbfff640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20c00678f0 0x7f20c0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20bbfff640 1 -- 172.21.15.70:0/1155720994 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f20c411ee80 con 0x7f20d40b4360 2024-04-22T01:16:16.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20c00678f0 0x7f20c0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20c00678f0 0x7f20c0069db0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f20d41400d0 tx=0x7f20c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 Processor -- start 2024-04-22T01:16:16.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 -- start start 2024-04-22T01:16:16.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 0x5609a84682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8468810 0x5609a841e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ebc0 0x5609a841efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609a826c5e0 con 0x5609a8467ef0 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5609a81f3c50 con 0x5609a8468810 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5609a82685a0 con 0x5609a841ebc0 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 0x5609a84682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 0x5609a84682d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50504/0 (socket says 172.21.15.70:50504) 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 -- 172.21.15.70:0/4026171476 learned_addr learned my addr 172.21.15.70:0/4026171476 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 -- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ebc0 msgr2=0x5609a841efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d3fff640 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ebc0 0x5609a841efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8468810 0x5609a841e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ebc0 0x5609a841efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 -- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8468810 msgr2=0x5609a841e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8468810 0x5609a841e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 -- 172.21.15.70:0/4026171476 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609a81f1d10 con 0x5609a8467ef0 2024-04-22T01:16:16.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20d37fe640 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 0x5609a84682d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f20c40af040 tx=0x7f20c40eeb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcddb1a00a260ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20b97fa640 1 -- 172.21.15.70:0/4026171476 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40f7d10 con 0x5609a8467ef0 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20b97fa640 1 -- 172.21.15.70:0/4026171476 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c40ed280 con 0x5609a8467ef0 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.919+0000 7f20b97fa640 1 -- 172.21.15.70:0/4026171476 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40ac500 con 0x5609a8467ef0 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/4026171476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 msgr2=0x5609a84682d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 0x5609a84682d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f20c40af040 tx=0x7f20c40eeb40 comp rx=0 tx=0).stop 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/4026171476 shutdown_connections 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ebc0 0x5609a841efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8468810 0x5609a841e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- 172.21.15.70:0/4026171476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a8467ef0 0x5609a84682d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/4026171476 >> 172.21.15.70:0/4026171476 conn(0x5609a841ab70 msgr2=0x5609a841cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/4026171476 shutdown_connections 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/4026171476 wait complete. 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- start start 2024-04-22T01:16:16.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 Processor -- start 2024-04-22T01:16:16.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8467890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8469c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a8468810 0x5609a846a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609a826c5e0 con 0x5609a841ebc0 2024-04-22T01:16:16.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5609a8226520 con 0x5609a8467ef0 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5609a8239340 con 0x5609a8468810 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 10 client.0 _ll_register_callbacks cb 0x5609a84095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8467890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8467890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50520/0 (socket says 172.21.15.70:50520) 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 -- 172.21.15.70:0/624037087 learned_addr learned my addr 172.21.15.70:0/624037087 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8469c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d3fff640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a8468810 0x5609a846a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 -- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a8468810 msgr2=0x5609a846a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a8468810 0x5609a846a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 -- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 msgr2=0x5609a8469c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8469c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 -- 172.21.15.70:0/624037087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609a82047b0 con 0x5609a841ebc0 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d3fff640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a8468810 0x5609a846a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:16.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d37fe640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8467890 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f20c411f1a0 tx=0x7f20c4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bcf6a57ad96812 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 1 -- 172.21.15.70:0/624037087 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40eb850 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 1 -- 172.21.15.70:0/624037087 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f20c40f7880 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20c40ed280 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 1 -- 172.21.15.70:0/624037087 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f20c40eb9f0 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 10 client.5658 fetch_fsmap learned FSMap version 17 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5609a81f3ea0 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 1 -- 172.21.15.70:0/624037087 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c40ed280 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 1 -- 172.21.15.70:0/624037087 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f20c4116b80 con 0x5609a841ebc0 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 10 client.5658 fetch_fsmap finished waiting for FSMap version 17 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 10 client.5658 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 20 client.5658 populate_metadata read hostname 'smithi070' 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a846af20 0x5609a847e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 4 client.5658 mds_command: new command op to 5478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 --> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609a8207ab0 con 0x5609a846af20 2024-04-22T01:16:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a846af20 0x5609a847e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a846af20 0x5609a847e8a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c100cc5cb2d1beb3 server_cookie=67e4d2551fc7fe62 in_seq=0 out_seq=0 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.923+0000 7f20b27fc640 10 client.5658 ms_handle_connect on v2:172.21.15.70:6836/3248824947 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20b27fc640 1 -- 172.21.15.70:0/624037087 <== mds.0 v2:172.21.15.70:6836/3248824947 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609a8207ab0 con 0x5609a846af20 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20b27fc640 10 client.5658 handle_command_reply: tid=0 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 client.5658 shutdown 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 msgr2=0x5609a8467890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8467890 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f20c411f1a0 tx=0x7f20c4000bd0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 shutdown_connections 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a846af20 0x5609a847e8a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a8468810 0x5609a846a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8469c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 --2- 172.21.15.70:0/624037087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8467890 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20b27fc640 0 client.5658 ms_handle_reset on v2:172.21.15.70:6836/3248824947 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 >> 172.21.15.70:0/624037087 conn(0x5609a841ab70 msgr2=0x5609a841d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 shutdown_connections 2024-04-22T01:16:16.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 1 -- 172.21.15.70:0/624037087 wait complete. 2024-04-22T01:16:16.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.927+0000 7f20dd136000 20 client.5658 trim_cache size 0 max 16384 2024-04-22T01:16:16.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 Processor -- start 2024-04-22T01:16:16.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- start start 2024-04-22T01:16:16.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a84795a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8479ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a847a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20d415ca20 con 0x5609a841ebc0 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f20d413e110 con 0x5609a8467ef0 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f20d415c3d0 con 0x5609a841ab70 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8479ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8479ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50524/0 (socket says 172.21.15.70:50524) 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 -- 172.21.15.70:0/3623071366 learned_addr learned my addr 172.21.15.70:0/3623071366 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 -- 172.21.15.70:0/3623071366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 msgr2=0x5609a84795a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a84795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 -- 172.21.15.70:0/3623071366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 msgr2=0x5609a847a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a847a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 -- 172.21.15.70:0/3623071366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609a81f4e80 con 0x5609a841ebc0 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8479ae0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f20d406c210 tx=0x7f20c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d379a0965e7d00e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20b8ff9640 1 -- 172.21.15.70:0/3623071366 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c802f070 con 0x5609a841ebc0 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20b8ff9640 1 -- 172.21.15.70:0/3623071366 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c8015a00 con 0x5609a841ebc0 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20b8ff9640 1 -- 172.21.15.70:0/3623071366 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c800f040 con 0x5609a841ebc0 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- 172.21.15.70:0/3623071366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 msgr2=0x5609a8479ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8479ae0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f20d406c210 tx=0x7f20c8016000 comp rx=0 tx=0).stop 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- 172.21.15.70:0/3623071366 shutdown_connections 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a847a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a8479ae0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 --2- 172.21.15.70:0/3623071366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a84795a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- 172.21.15.70:0/3623071366 >> 172.21.15.70:0/3623071366 conn(0x5609a844b4f0 msgr2=0x5609a8468bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- 172.21.15.70:0/3623071366 shutdown_connections 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.943+0000 7f20dd136000 1 -- 172.21.15.70:0/3623071366 wait complete. 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- start start 2024-04-22T01:16:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 Processor -- start 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a8482cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a84831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8483730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609a8239340 con 0x5609a841ebc0 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5609a8427250 con 0x5609a8467ef0 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5609a8226520 con 0x5609a841ab70 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 10 client.0 _ll_register_callbacks cb 0x7f20c40ef0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a84831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a84831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50528/0 (socket says 172.21.15.70:50528) 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 -- 172.21.15.70:0/2809761636 learned_addr learned my addr 172.21.15.70:0/2809761636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d37fe640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a8482cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d3fff640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8483730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 -- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 msgr2=0x5609a8482cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a8482cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 -- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 msgr2=0x5609a8483730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8483730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 -- 172.21.15.70:0/2809761636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609a8239760 con 0x5609a841ebc0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d3fff640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8483730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d2ffd640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a84831f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009690 tx=0x7f20c8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975a0129ce9f3aef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 1 -- 172.21.15.70:0/2809761636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8038060 con 0x5609a841ebc0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 1 -- 172.21.15.70:0/2809761636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f20c80293e0 con 0x5609a841ebc0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20c8015a00 con 0x5609a841ebc0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 1 -- 172.21.15.70:0/2809761636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f20c803e050 con 0x5609a841ebc0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 10 client.5664 fetch_fsmap learned FSMap version 17 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5609a82c59e0 con 0x5609a841ebc0 2024-04-22T01:16:16.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 1 -- 172.21.15.70:0/2809761636 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c8015a00 con 0x5609a841ebc0 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 1 -- 172.21.15.70:0/2809761636 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f20c8041020 con 0x5609a841ebc0 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 10 client.5664 fetch_fsmap finished waiting for FSMap version 17 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 10 client.5664 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 20 client.5664 populate_metadata read hostname 'smithi070' 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a8468810 0x5609a84e4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 4 client.5664 mds_command: new command op to 5478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 --> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] -- command(tid 0: {"prefix": "status"}) -- 0x5609a8279b20 con 0x5609a8468810 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d37fe640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a8468810 0x5609a84e4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 10 client.5664 ms_handle_connect on v2:172.21.15.70:6836/3248824947 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d37fe640 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a8468810 0x5609a84e4880 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50cb0be141c72d28 server_cookie=956495dda7c58c1 in_seq=0 out_seq=0 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 1 -- 172.21.15.70:0/2809761636 <== mds.0 v2:172.21.15.70:6836/3248824947 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5609a8279b20 con 0x5609a8468810 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 10 client.5664 handle_command_reply: tid=0 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 client.5664 shutdown 2024-04-22T01:16:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 msgr2=0x5609a84831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a84831f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009690 tx=0x7f20c8004740 comp rx=0 tx=0).stop 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 shutdown_connections 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:6836/3248824947,v1:172.21.15.70:6837/3248824947] conn(0x5609a8468810 0x5609a84e4880 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609a8467ef0 0x5609a8483730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609a841ebc0 0x5609a84831f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20b27fc640 0 client.5664 ms_handle_reset on v2:172.21.15.70:6836/3248824947 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 --2- 172.21.15.70:0/2809761636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5609a841ab70 0x5609a8482cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 >> 172.21.15.70:0/2809761636 conn(0x5609a844b4f0 msgr2=0x5609a8466d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 shutdown_connections 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 1 -- 172.21.15.70:0/2809761636 wait complete. 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20dd136000 20 client.5664 trim_cache size 0 max 16384 2024-04-22T01:16:16.955 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5478, 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 5.7612937310000003, 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 17, 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 38, 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 38, 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 12.679991080000001 2024-04-22T01:16:16.956 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:16:16.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20c00678f0 msgr2=0x7f20c0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20c00678f0 0x7f20c0069db0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f20d41400d0 tx=0x7f20c8008040 comp rx=0 tx=0).stop 2024-04-22T01:16:16.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 msgr2=0x7f20d413aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 0x7f20d413aad0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f20c4019130 tx=0x7f20c40192c0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 shutdown_connections 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20c00678f0 0x7f20c0069db0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20d41445b0 0x7f20d4152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20d40cdce0 0x7f20d413bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 --2- 172.21.15.70:0/1155720994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20d40b4360 0x7f20d413aad0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.947+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 >> 172.21.15.70:0/1155720994 conn(0x7f20d40cf9e0 msgr2=0x7f20d406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.951+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 shutdown_connections 2024-04-22T01:16:16.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:16.951+0000 7f20d9fbd640 1 -- 172.21.15.70:0/1155720994 wait complete. 2024-04-22T01:16:16.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:16:17.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 Processor -- start 2024-04-22T01:16:17.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 -- start start 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8c5c1455b0 0x7f8c5c0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c5c0cc150 0x7f8c5c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8c5c0724b0 con 0x7f8c5c1455b0 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8c5c0d0250 con 0x7f8c5c0cc150 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c63832640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5c0cd7b0 con 0x7f8c5c072060 2024-04-22T01:16:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8c5c1455b0 0x7f8c5c0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8c5c1455b0 0x7f8c5c0cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50540/0 (socket says 172.21.15.70:50540) 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 -- 172.21.15.70:0/4015590162 learned_addr learned my addr 172.21.15.70:0/4015590162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c61dcf640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c60dcd640 1 --2- 172.21.15.70:0/4015590162 >> v2:172.21.15.98:3300/0 conn(0x7f8c5c0cc150 0x7f8c5c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 -- 172.21.15.70:0/4015590162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 msgr2=0x7f8c5c15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 -- 172.21.15.70:0/4015590162 >> v2:172.21.15.98:3300/0 conn(0x7f8c5c0cc150 msgr2=0x7f8c5c071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 --2- 172.21.15.70:0/4015590162 >> v2:172.21.15.98:3300/0 conn(0x7f8c5c0cc150 0x7f8c5c071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c615ce640 1 -- 172.21.15.70:0/4015590162 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5c0cc710 con 0x7f8c5c1455b0 2024-04-22T01:16:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.075+0000 7f8c61dcf640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c615ce640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c0cfd10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0cbd20 tx=0x7f8c4c01c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73991184be5f1a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c53fff640 1 -- 172.21.15.70:0/4015590162 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0b1070 con 0x7f8c5c1455b0 2024-04-22T01:16:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c53fff640 1 -- 172.21.15.70:0/4015590162 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c4c0ac440 con 0x7f8c5c1455b0 2024-04-22T01:16:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c53fff640 1 -- 172.21.15.70:0/4015590162 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0ac810 con 0x7f8c5c1455b0 2024-04-22T01:16:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- 172.21.15.70:0/4015590162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 msgr2=0x7f8c5c0cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c0cfd10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0cbd20 tx=0x7f8c4c01c140 comp rx=0 tx=0).stop 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- 172.21.15.70:0/4015590162 shutdown_connections 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- 172.21.15.70:0/4015590162 >> v2:172.21.15.98:3300/0 conn(0x7f8c5c0cc150 0x7f8c5c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- 172.21.15.70:0/4015590162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c0cfd10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- 172.21.15.70:0/4015590162 >> 172.21.15.70:0/4015590162 conn(0x7f8c5c06b1f0 msgr2=0x7f8c5c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- 172.21.15.70:0/4015590162 shutdown_connections 2024-04-22T01:16:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- 172.21.15.70:0/4015590162 wait complete. 2024-04-22T01:16:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 Processor -- start 2024-04-22T01:16:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- start start 2024-04-22T01:16:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c0cc150 0x7f8c5c139590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c15e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c06e380 con 0x7f8c5c1455b0 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c06e200 con 0x7f8c5c0cc150 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c63832640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5c06e500 con 0x7f8c5c072060 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c15e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c15e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50552/0 (socket says 172.21.15.70:50552) 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 -- 172.21.15.70:0/1943091143 learned_addr learned my addr 172.21.15.70:0/1943091143 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c60dcd640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c0cc150 0x7f8c5c139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c615ce640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 -- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 msgr2=0x7f8c5c15da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 -- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c0cc150 msgr2=0x7f8c5c139590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c0cc150 0x7f8c5c139590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 -- 172.21.15.70:0/1943091143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5c1661e0 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c61dcf640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c15e8c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400cef0 tx=0x7f8c5400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.079+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5401a070 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c5400d840 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5c1664c0 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5400dc30 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c5c15df80 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c5c068210 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8c5401d070 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c51ffb640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c400678c0 0x7f8c40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f8c540865a0 con 0x7f8c5c1455b0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c615ce640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c400678c0 0x7f8c40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.083+0000 7f8c615ce640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c400678c0 0x7f8c40069d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0ae000 tx=0x7f8c4c0a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.087+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8c54056fe0 con 0x7f8c5c1455b0 2024-04-22T01:16:17.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.275+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8c5c15e570 con 0x7f8c5c1455b0 2024-04-22T01:16:17.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.275+0000 7f8c51ffb640 1 -- 172.21.15.70:0/1943091143 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f8c5405be20 con 0x7f8c5c1455b0 2024-04-22T01:16:17.276 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c400678c0 msgr2=0x7f8c40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c400678c0 0x7f8c40069d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0ae000 tx=0x7f8c4c0a6070 comp rx=0 tx=0).stop 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 msgr2=0x7f8c5c15e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c15e8c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400cef0 tx=0x7f8c5400a710 comp rx=0 tx=0).stop 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 shutdown_connections 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c400678c0 0x7f8c40069d80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c5c1455b0 0x7f8c5c15e8c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c5c0cc150 0x7f8c5c139590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 --2- 172.21.15.70:0/1943091143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c5c072060 0x7f8c5c15da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 >> 172.21.15.70:0/1943091143 conn(0x7f8c5c06b1f0 msgr2=0x7f8c5c0b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:17.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 shutdown_connections 2024-04-22T01:16:17.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17.279+0000 7f8c63832640 1 -- 172.21.15.70:0/1943091143 wait complete. 2024-04-22T01:16:17.292 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:16:17.293 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:16:17.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.294 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:17.300 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42445sec preferred_lft 42445sec 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:16:17.301 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:17.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.302 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:16:17.353 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:16:17.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.355 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:17.405 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:17.406 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:17.406 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:16:17.406 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:16:17.406 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:16:17.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:16:17.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:16:17.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:16:17.406 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:16:17.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:16:17.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:17.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.524 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:16:17.569 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:16:17.569 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:16:17.569 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:16:17.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:16:17.625 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:16:17.626 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:17.626 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:17.674 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:16:17.674 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:17.674 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:17.721 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:16:17.779 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:17.828 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:16:17.829 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:16:17.830 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:16:17.830 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:17.830 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:16:17.830 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:16:17.830 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:16:17.830 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:16:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:16:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:16:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:16:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:16:17.832 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:16:17.832 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:16:17.832 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:16:17.832 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:16:17.832 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:16:17.832 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:16:17.833 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:17.833 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:17.833 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:17.833 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:17.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.835 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:16:17.882 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:16:17.882 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:16:17.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:16:17.943 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:17.990 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:16:17.992 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:16:17.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:17.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:17.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:17.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:17.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:17.993 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:16:18.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:16:18.003+0000 7f8ad98fd580 -1 init, newargv = 0x560f25f07730 newargc=15 2024-04-22T01:16:18.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[38566]: starting ceph client 2024-04-22T01:16:18.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[38566]: starting fuse 2024-04-22T01:16:19.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:16:19.124 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:16:19.171 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:16:19.172 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:19.173 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:19.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:19.174 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:16:19.222 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:16:19.223 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:16:19.223 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:16:19.224 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:16:19.346 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.38566.asok 2024-04-22T01:16:19.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:19.358 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38566.asok 2024-04-22T01:16:19.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:19.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38566.asok status 2024-04-22T01:16:19.510 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "38566", 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5679, 2024-04-22T01:16:19.511 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 5679 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1836261735 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1836261735 2024-04-22T01:16:19.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.5679 192.168.0.1:0/1836261735", 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/1836261735", 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 17, 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 38, 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:16:19.513 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:16:19.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:19.524 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:16:19.524 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:19.530 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:16:19.530 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:19.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:19.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:19.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:16:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.751+0000 7f44dcc65640 1 Processor -- start 2024-04-22T01:16:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.751+0000 7f44dcc65640 1 -- start start 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44dcc65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d8159380 0x7f44d80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44dcc65640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f44d80b3a50 0x7f44d80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44dcc65640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d8162a60 con 0x7f44d8163500 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44dcc65640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f44d80b3f40 con 0x7f44d80b3a50 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44dcc65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f44d8162d00 con 0x7f44d8159380 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d6d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f44d8163500 0x7f44d806e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33524/0 (socket says 172.21.15.70:33524) 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d6d9d640 1 -- 172.21.15.70:0/3507797958 learned_addr learned my addr 172.21.15.70:0/3507797958 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d5d9b640 1 --2- 172.21.15.70:0/3507797958 >> v2:172.21.15.70:3300/0 conn(0x7f44d80b3a50 0x7f44d80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d659c640 1 --2- 172.21.15.70:0/3507797958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d8159380 0x7f44d80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d5d9b640 1 -- 172.21.15.70:0/3507797958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d8159380 msgr2=0x7f44d80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d5d9b640 1 --2- 172.21.15.70:0/3507797958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d8159380 0x7f44d80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d5d9b640 1 -- 172.21.15.70:0/3507797958 >> v1:172.21.15.70:6789/0 conn(0x7f44d8163500 legacy=0x7f44d806e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d559a640 1 -- 172.21.15.70:0/3507797958 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526637165 0 0) 0x7f44d8162a60 con 0x7f44d8163500 2024-04-22T01:16:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d5d9b640 1 -- 172.21.15.70:0/3507797958 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d8162fe0 con 0x7f44d80b3a50 2024-04-22T01:16:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d5d9b640 1 --2- 172.21.15.70:0/3507797958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d80b3a50 0x7f44d80cf750 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f44c80097c0 tx=0x7f44c801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa28039429688082 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d559a640 1 -- 172.21.15.70:0/3507797958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80b0070 con 0x7f44d80b3a50 2024-04-22T01:16:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d559a640 1 -- 172.21.15.70:0/3507797958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c80ab600 con 0x7f44d80b3a50 2024-04-22T01:16:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.755+0000 7f44d559a640 1 -- 172.21.15.70:0/3507797958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80ab9d0 con 0x7f44d80b3a50 2024-04-22T01:16:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/3507797958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d80b3a50 msgr2=0x7f44d80cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 --2- 172.21.15.70:0/3507797958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d80b3a50 0x7f44d80cf750 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f44c80097c0 tx=0x7f44c801c410 comp rx=0 tx=0).stop 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/3507797958 shutdown_connections 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 --2- 172.21.15.70:0/3507797958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d80b3a50 0x7f44d80cf750 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 --2- 172.21.15.70:0/3507797958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d8159380 0x7f44d80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/3507797958 >> 172.21.15.70:0/3507797958 conn(0x7f44d806b1f0 msgr2=0x7f44d806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/3507797958 shutdown_connections 2024-04-22T01:16:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/3507797958 wait complete. 2024-04-22T01:16:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 Processor -- start 2024-04-22T01:16:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- start start 2024-04-22T01:16:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d80b3a50 0x7f44d8154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 0x7f44d8150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d8163500 0x7f44d80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44d816f250 con 0x7f44d8159380 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44d8166e60 con 0x7f44d8163500 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f44d816f3d0 con 0x7f44d80b3a50 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 0x7f44d8150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 0x7f44d8150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50578/0 (socket says 172.21.15.70:50578) 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 -- 172.21.15.70:0/1537679255 learned_addr learned my addr 172.21.15.70:0/1537679255 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 -- 172.21.15.70:0/1537679255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d80b3a50 msgr2=0x7f44d8154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d80b3a50 0x7f44d8154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 -- 172.21.15.70:0/1537679255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d8163500 msgr2=0x7f44d80b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d8163500 0x7f44d80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 -- 172.21.15.70:0/1537679255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d814fa60 con 0x7f44d8159380 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44d5d9b640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 0x7f44d8150b00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f44c80ae040 tx=0x7f44d8159970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80b8020 con 0x7f44d8159380 2024-04-22T01:16:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c80b4410 con 0x7f44d8159380 2024-04-22T01:16:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d814fd40 con 0x7f44d8159380 2024-04-22T01:16:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80b0040 con 0x7f44d8159380 2024-04-22T01:16:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.759+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d8150300 con 0x7f44d8159380 2024-04-22T01:16:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.763+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d8068210 con 0x7f44d8159380 2024-04-22T01:16:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.763+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f44c80bb020 con 0x7f44d8159380 2024-04-22T01:16:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.763+0000 7f44c6ffd640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44b80678c0 0x7f44b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.763+0000 7f44d659c640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44b80678c0 0x7f44b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.763+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f44c811f8f0 con 0x7f44d8159380 2024-04-22T01:16:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.763+0000 7f44d659c640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44b80678c0 0x7f44b8069d80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f44cc006aa0 tx=0x7f44cc006a30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:19.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.767+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f44c80c2030 con 0x7f44d8159380 2024-04-22T01:16:19.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.963+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f44d806e040 con 0x7f44d8159380 2024-04-22T01:16:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.963+0000 7f44c6ffd640 1 -- 172.21.15.70:0/1537679255 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f44c8016090 con 0x7f44d8159380 2024-04-22T01:16:19.964 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44b80678c0 msgr2=0x7f44b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44b80678c0 0x7f44b8069d80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f44cc006aa0 tx=0x7f44cc006a30 comp rx=0 tx=0).stop 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 msgr2=0x7f44d8150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 0x7f44d8150b00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f44c80ae040 tx=0x7f44d8159970 comp rx=0 tx=0).stop 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 shutdown_connections 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44b80678c0 0x7f44b8069d80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d8163500 0x7f44d80b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d8159380 0x7f44d8150b00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 --2- 172.21.15.70:0/1537679255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44d80b3a50 0x7f44d8154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 >> 172.21.15.70:0/1537679255 conn(0x7f44d806b1f0 msgr2=0x7f44d806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 shutdown_connections 2024-04-22T01:16:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:19.967+0000 7f44dcc65640 1 -- 172.21.15.70:0/1537679255 wait complete. 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:16:19.983 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:16:19.984 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:16:19.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:19.984 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:16:19.991 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:16:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:16:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:16:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42446sec preferred_lft 42446sec 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.992 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:16:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:19.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:19.994 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:16:20.047 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:16:20.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:20.049 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:16:20.100 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:16:20.100 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:16:20.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:16:20.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:16:20.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:16:20.100 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:16:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:16:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:16:20.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:16:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:20.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:20.218 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:16:20.271 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:16:20.271 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:16:20.272 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:16:20.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:20.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:16:20.327 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:16:20.327 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:20.327 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:20.378 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:16:20.379 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:20.379 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:20.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:16:20.487 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:20.536 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:16:20.536 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:16:20.536 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:16:20.536 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:16:20.536 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:16:20.537 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:20.538 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:20.539 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:20.539 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:20.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:20.539 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:16:20.586 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:16:20.586 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:16:20.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:16:20.647 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:16:20.695 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:16:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:20.697 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:20.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:20.698 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:16:20.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:16:20.704+0000 7f168bb3f580 -1 init, newargv = 0x55f88e49e870 newargc=15 2024-04-22T01:16:20.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[33903]: starting ceph client 2024-04-22T01:16:20.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[33903]: starting fuse 2024-04-22T01:16:21.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:16:21.829 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:21.876 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:16:21.876 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:16:21.876 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:16:21.876 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:16:21.877 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:16:21.878 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:16:21.879 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:21.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:21.880 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:16:21.922 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:16:21.922 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:16:21.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:16:21.923 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:16:21.923 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:16:21.923 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:16:21.923 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:16:21.924 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:16:22.047 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.33903.asok 2024-04-22T01:16:22.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:16:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:22.059 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33903.asok 2024-04-22T01:16:22.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33903.asok status 2024-04-22T01:16:22.214 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "33903", 2024-04-22T01:16:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4769, 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4769 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1817535415 2024-04-22T01:16:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1817535415 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4769 192.168.0.1:0/1817535415", 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/1817535415", 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 17, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 38, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:16:22.217 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:16:22.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.228 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:16:22.228 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:22.234 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:16:22.234 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:22.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:22.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.288 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-22T01:16:22.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.304 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 0 ./subdir) 2024-04-22T01:16:22.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.355 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:16:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:22.404 INFO:teuthology.orchestra.run.smithi070.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-04-22T01:16:22.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.405 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1K ./subdir) 2024-04-22T01:16:22.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.457 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:16:22.506 INFO:teuthology.orchestra.run.smithi070.stdout:1024 2024-04-22T01:16:22.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.507 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 0M ./subdir) 2024-04-22T01:16:22.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:22.561 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:16:22.612 INFO:teuthology.orchestra.run.smithi070.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-04-22T01:16:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:22.613 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:16:22.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:16:22.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:16:22.760 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:16:22.760 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:22.760 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:16:22.760 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:22.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:22.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 Processor -- start 2024-04-22T01:16:22.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 -- start start 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83dc0cc150 0x7f83dc071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83dc072060 0x7f83dc06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83dc0cd7b0 con 0x7f83dc1455b0 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f83dc072490 con 0x7f83dc0cc150 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.939+0000 7f83e22fd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f83dc0cc710 con 0x7f83dc072060 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83dc0cc150 0x7f83dc071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f83dc0cc150 0x7f83dc071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50586/0 (socket says 172.21.15.70:50586) 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 -- 172.21.15.70:0/620845799 learned_addr learned my addr 172.21.15.70:0/620845799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83e089a640 1 --2- 172.21.15.70:0/620845799 >> v2:172.21.15.98:3300/0 conn(0x7f83dc072060 0x7f83dc06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 -- 172.21.15.70:0/620845799 >> v2:172.21.15.98:3300/0 conn(0x7f83dc072060 msgr2=0x7f83dc06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:22.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 --2- 172.21.15.70:0/620845799 >> v2:172.21.15.98:3300/0 conn(0x7f83dc072060 0x7f83dc06e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:22.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 -- 172.21.15.70:0/620845799 >> v1:172.21.15.98:6789/0 conn(0x7f83dc1455b0 legacy=0x7f83dc0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:22.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 -- 172.21.15.70:0/620845799 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc0cc9f0 con 0x7f83dc0cc150 2024-04-22T01:16:22.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83db7fe640 1 --2- 172.21.15.70:0/620845799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc0cc150 0x7f83dc071b20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f83dc06ffb0 tx=0x7f83d001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02eeddfd14dd932 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:22.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83daffd640 1 -- 172.21.15.70:0/620845799 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00b0070 con 0x7f83dc0cc150 2024-04-22T01:16:22.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83daffd640 1 -- 172.21.15.70:0/620845799 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d00a4e50 con 0x7f83dc0cc150 2024-04-22T01:16:22.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83daffd640 1 -- 172.21.15.70:0/620845799 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00ab820 con 0x7f83dc0cc150 2024-04-22T01:16:22.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83e22fd640 1 -- 172.21.15.70:0/620845799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc0cc150 msgr2=0x7f83dc071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.943+0000 7f83e22fd640 1 --2- 172.21.15.70:0/620845799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc0cc150 0x7f83dc071b20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f83dc06ffb0 tx=0x7f83d001c3c0 comp rx=0 tx=0).stop 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- 172.21.15.70:0/620845799 shutdown_connections 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 --2- 172.21.15.70:0/620845799 >> v2:172.21.15.98:3300/0 conn(0x7f83dc072060 0x7f83dc06e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 --2- 172.21.15.70:0/620845799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc0cc150 0x7f83dc071b20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- 172.21.15.70:0/620845799 >> 172.21.15.70:0/620845799 conn(0x7f83dc06b1f0 msgr2=0x7f83dc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- 172.21.15.70:0/620845799 shutdown_connections 2024-04-22T01:16:22.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- 172.21.15.70:0/620845799 wait complete. 2024-04-22T01:16:22.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 Processor -- start 2024-04-22T01:16:22.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- start start 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc072060 0x7f83dc1596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83dc0cc150 0x7f83dc15db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 0x7f83dc158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc06f2e0 con 0x7f83dc1455b0 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc06f160 con 0x7f83dc072060 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83dc168b40 con 0x7f83dc0cc150 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 0x7f83dc158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 0x7f83dc158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50600/0 (socket says 172.21.15.70:50600) 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 -- 172.21.15.70:0/1449915020 learned_addr learned my addr 172.21.15.70:0/1449915020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83dbfff640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc072060 0x7f83dc1596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:22.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 -- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83dc0cc150 msgr2=0x7f83dc15db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83dc0cc150 0x7f83dc15db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 -- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc072060 msgr2=0x7f83dc1596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc072060 0x7f83dc1596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 -- 172.21.15.70:0/1449915020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc1585a0 con 0x7f83dc1455b0 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e089a640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 0x7f83dc158060 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f83d4007e60 tx=0x7f83d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4014420 con 0x7f83dc1455b0 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d4019070 con 0x7f83dc1455b0 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83dc15e290 con 0x7f83dc1455b0 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4014820 con 0x7f83dc1455b0 2024-04-22T01:16:22.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.947+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc15e850 con 0x7f83dc1455b0 2024-04-22T01:16:22.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.951+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83dc068210 con 0x7f83dc1455b0 2024-04-22T01:16:22.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.951+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83d40149c0 con 0x7f83dc1455b0 2024-04-22T01:16:22.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.951+0000 7f83d8ff9640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83c80678f0 0x7f83c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:22.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.951+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f83d40865a0 con 0x7f83dc1455b0 2024-04-22T01:16:22.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.951+0000 7f83dbfff640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83c80678f0 0x7f83c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:22.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.951+0000 7f83dbfff640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83c80678f0 0x7f83c8069db0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00b810 tx=0x7f83cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:22.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:22.955+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f83d4056fe0 con 0x7f83dc1455b0 2024-04-22T01:16:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.115+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83dc15f510 con 0x7f83dc1455b0 2024-04-22T01:16:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83d8ff9640 1 -- 172.21.15.70:0/1449915020 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16906 (secure 0 0 0) 0x7f83d405be20 con 0x7f83dc1455b0 2024-04-22T01:16:23.118 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:23.118 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:12.187805+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83c80678f0 msgr2=0x7f83c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83c80678f0 0x7f83c8069db0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00b810 tx=0x7f83cc008040 comp rx=0 tx=0).stop 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 msgr2=0x7f83dc158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 0x7f83dc158060 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f83d4007e60 tx=0x7f83d400e4b0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 shutdown_connections 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83c80678f0 0x7f83c8069db0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83dc1455b0 0x7f83dc158060 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83dc0cc150 0x7f83dc15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 --2- 172.21.15.70:0/1449915020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc072060 0x7f83dc1596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 >> 172.21.15.70:0/1449915020 conn(0x7f83dc06b1f0 msgr2=0x7f83dc13c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 shutdown_connections 2024-04-22T01:16:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.119+0000 7f83e22fd640 1 -- 172.21.15.70:0/1449915020 wait complete. 2024-04-22T01:16:23.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:16:08.953522+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:16:09.533622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:23.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:23.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.259+0000 7f292842a640 1 Processor -- start 2024-04-22T01:16:23.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.259+0000 7f292842a640 1 -- start start 2024-04-22T01:16:23.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f292842a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f292015c010 0x7f29200b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f292842a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200af740 0x7f29200746b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f292842a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29200b2cd0 con 0x7f29200ccbb0 2024-04-22T01:16:23.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f292842a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f29201606a0 con 0x7f292015c010 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f292842a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2920160980 con 0x7f29200af740 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29261c6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f292015c010 0x7f29200b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29269c7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f29200ccbb0 0x7f29200cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33532/0 (socket says 172.21.15.70:33532) 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29269c7640 1 -- 172.21.15.70:0/2001404907 learned_addr learned my addr 172.21.15.70:0/2001404907 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29259c5640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200af740 0x7f29200746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29261c6640 1 -- 172.21.15.70:0/2001404907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200af740 msgr2=0x7f29200746b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29261c6640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200af740 0x7f29200746b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29261c6640 1 -- 172.21.15.70:0/2001404907 >> v1:172.21.15.70:6789/0 conn(0x7f29200ccbb0 legacy=0x7f29200cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29261c6640 1 -- 172.21.15.70:0/2001404907 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200b2680 con 0x7f292015c010 2024-04-22T01:16:23.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29259c5640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200af740 0x7f29200746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:23.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29261c6640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292015c010 0x7f29200b06b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f29180097c0 tx=0x7f291801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd09e91d375f1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29251c4640 1 -- 172.21.15.70:0/2001404907 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180b1070 con 0x7f292015c010 2024-04-22T01:16:23.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29251c4640 1 -- 172.21.15.70:0/2001404907 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29180ac410 con 0x7f292015c010 2024-04-22T01:16:23.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.263+0000 7f29251c4640 1 -- 172.21.15.70:0/2001404907 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180ac7e0 con 0x7f292015c010 2024-04-22T01:16:23.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/2001404907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292015c010 msgr2=0x7f29200b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292015c010 0x7f29200b06b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f29180097c0 tx=0x7f291801c4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/2001404907 shutdown_connections 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200af740 0x7f29200746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 --2- 172.21.15.70:0/2001404907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f292015c010 0x7f29200b06b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/2001404907 >> 172.21.15.70:0/2001404907 conn(0x7f2920099a60 msgr2=0x7f29200b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/2001404907 shutdown_connections 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/2001404907 wait complete. 2024-04-22T01:16:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 Processor -- start 2024-04-22T01:16:23.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- start start 2024-04-22T01:16:23.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 0x7f29200c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200ccbb0 0x7f29200c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292015c010 0x7f29200c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f292006c070 con 0x7f29200af740 2024-04-22T01:16:23.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f292006bef0 con 0x7f292015c010 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f292006c1f0 con 0x7f29200ccbb0 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 0x7f29200c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 0x7f29200c5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50624/0 (socket says 172.21.15.70:50624) 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 -- 172.21.15.70:0/611596838 learned_addr learned my addr 172.21.15.70:0/611596838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 -- 172.21.15.70:0/611596838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200ccbb0 msgr2=0x7f29200c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200ccbb0 0x7f29200c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 -- 172.21.15.70:0/611596838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292015c010 msgr2=0x7f29200c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292015c010 0x7f29200c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 -- 172.21.15.70:0/611596838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200c6360 con 0x7f29200af740 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f29261c6640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 0x7f29200c5e20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f291801a0d0 tx=0x7f2918019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180b1040 con 0x7f29200af740 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29180bb070 con 0x7f29200af740 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29200c7330 con 0x7f29200af740 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918006e20 con 0x7f29200af740 2024-04-22T01:16:23.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29200c78f0 con 0x7f29200af740 2024-04-22T01:16:23.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.267+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292009e450 con 0x7f29200af740 2024-04-22T01:16:23.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.271+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f29180b5630 con 0x7f29200af740 2024-04-22T01:16:23.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.271+0000 7f2916ffd640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f29040678c0 0x7f2904069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.271+0000 7f29259c5640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f29040678c0 0x7f2904069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.271+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f291811f960 con 0x7f29200af740 2024-04-22T01:16:23.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.271+0000 7f29259c5640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f29040678c0 0x7f2904069d80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f291000a890 tx=0x7f2910008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.275+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2918016090 con 0x7f29200af740 2024-04-22T01:16:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.427+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f292004f5f0 con 0x7f29200af740 2024-04-22T01:16:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f2916ffd640 1 -- 172.21.15.70:0/611596838 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f29180ace20 con 0x7f29200af740 2024-04-22T01:16:23.429 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f29040678c0 msgr2=0x7f2904069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f29040678c0 0x7f2904069d80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f291000a890 tx=0x7f2910008040 comp rx=0 tx=0).stop 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 msgr2=0x7f29200c5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 0x7f29200c5e20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f291801a0d0 tx=0x7f2918019240 comp rx=0 tx=0).stop 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 shutdown_connections 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f29040678c0 0x7f2904069d80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f292015c010 0x7f29200c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29200ccbb0 0x7f29200c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 --2- 172.21.15.70:0/611596838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29200af740 0x7f29200c5e20 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 >> 172.21.15.70:0/611596838 conn(0x7f2920099a60 msgr2=0x7f29200aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 shutdown_connections 2024-04-22T01:16:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.431+0000 7f292842a640 1 -- 172.21.15.70:0/611596838 wait complete. 2024-04-22T01:16:23.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:23.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 Processor -- start 2024-04-22T01:16:23.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 -- start start 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc0dc0b4450 0x7fc0dc071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc0dc0cd220 0x7fc0dc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0dc15c290 con 0x7fc0dc0720b0 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc0dc15c570 con 0x7fc0dc0cd220 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e2eec640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc0dc15c850 con 0x7fc0dc0b4450 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc0dc0cd220 0x7fc0dc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc0dc0cd220 0x7fc0dc0b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50636/0 (socket says 172.21.15.70:50636) 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 -- 172.21.15.70:0/3026764315 learned_addr learned my addr 172.21.15.70:0/3026764315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e0c88640 1 --2- 172.21.15.70:0/3026764315 >> v2:172.21.15.98:3300/0 conn(0x7fc0dc0b4450 0x7fc0dc071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 -- 172.21.15.70:0/3026764315 >> v2:172.21.15.98:3300/0 conn(0x7fc0dc0b4450 msgr2=0x7fc0dc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:23.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 --2- 172.21.15.70:0/3026764315 >> v2:172.21.15.98:3300/0 conn(0x7fc0dc0b4450 0x7fc0dc071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 -- 172.21.15.70:0/3026764315 >> v1:172.21.15.98:6789/0 conn(0x7fc0dc0720b0 legacy=0x7fc0dc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.571+0000 7fc0e1489640 1 -- 172.21.15.70:0/3026764315 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc15cb30 con 0x7fc0dc0cd220 2024-04-22T01:16:23.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e1489640 1 --2- 172.21.15.70:0/3026764315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc0b2a40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc06fce0 tx=0x7fc0d401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3adef3671b01121b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0db7fe640 1 -- 172.21.15.70:0/3026764315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40a56c0 con 0x7fc0dc0cd220 2024-04-22T01:16:23.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0db7fe640 1 -- 172.21.15.70:0/3026764315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d40a5860 con 0x7fc0dc0cd220 2024-04-22T01:16:23.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0db7fe640 1 -- 172.21.15.70:0/3026764315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40a5c30 con 0x7fc0dc0cd220 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 -- 172.21.15.70:0/3026764315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0cd220 msgr2=0x7fc0dc0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/3026764315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc0b2a40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc06fce0 tx=0x7fc0d401c9f0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 -- 172.21.15.70:0/3026764315 shutdown_connections 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/3026764315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc0b2a40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/3026764315 >> v2:172.21.15.98:3300/0 conn(0x7fc0dc0b4450 0x7fc0dc071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 -- 172.21.15.70:0/3026764315 >> 172.21.15.70:0/3026764315 conn(0x7fc0dc0b1d40 msgr2=0x7fc0dc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:23.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 -- 172.21.15.70:0/3026764315 shutdown_connections 2024-04-22T01:16:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 -- 172.21.15.70:0/3026764315 wait complete. 2024-04-22T01:16:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 Processor -- start 2024-04-22T01:16:23.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.575+0000 7fc0e2eec640 1 -- start start 2024-04-22T01:16:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0dc0720b0 0x7fc0dc06e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 0x7fc0dc145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b3910 con 0x7fc0dc0b4450 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b3790 con 0x7fc0dc0cd220 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc168b10 con 0x7fc0dc0720b0 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 0x7fc0dc145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 0x7fc0dc145d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50652/0 (socket says 172.21.15.70:50652) 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 -- 172.21.15.70:0/332359146 learned_addr learned my addr 172.21.15.70:0/332359146 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e0c88640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0dc0720b0 0x7fc0dc06e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e1489640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 -- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0dc0720b0 msgr2=0x7fc0dc06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0dc0720b0 0x7fc0dc06e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 -- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0dc0cd220 msgr2=0x7fc0dc15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc15ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 -- 172.21.15.70:0/332359146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc146280 con 0x7fc0dc0b4450 2024-04-22T01:16:23.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e0c88640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0dc0720b0 0x7fc0dc06e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e1489640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0dbfff640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 0x7fc0dc145d40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00c450 tx=0x7fc0cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc016020 con 0x7fc0dc0b4450 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc0052d0 con 0x7fc0dc0b4450 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc1551b0 con 0x7fc0dc0b4450 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc007260 con 0x7fc0dc0b4450 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc155770 con 0x7fc0dc0b4450 2024-04-22T01:16:23.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc068210 con 0x7fc0dc0b4450 2024-04-22T01:16:23.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc0cc007400 con 0x7fc0dc0b4450 2024-04-22T01:16:23.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0d97fa640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0c80678c0 0x7fc0c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0e0c88640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0c80678c0 0x7fc0c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.579+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fc0cc085740 con 0x7fc0dc0b4450 2024-04-22T01:16:23.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.583+0000 7fc0e0c88640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0c80678c0 0x7fc0c8069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009840 tx=0x7fc0d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.583+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc0cc020020 con 0x7fc0dc0b4450 2024-04-22T01:16:23.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.747+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0dc15e010 con 0x7fc0dc0b4450 2024-04-22T01:16:23.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.747+0000 7fc0d97fa640 1 -- 172.21.15.70:0/332359146 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7fc0cc005480 con 0x7fc0dc0b4450 2024-04-22T01:16:23.749 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0c80678c0 msgr2=0x7fc0c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0c80678c0 0x7fc0c8069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009840 tx=0x7fc0d0008040 comp rx=0 tx=0).stop 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 msgr2=0x7fc0dc145d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 0x7fc0dc145d40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00c450 tx=0x7fc0cc00c910 comp rx=0 tx=0).stop 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 shutdown_connections 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0c80678c0 0x7fc0c8069d80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0dc0cd220 0x7fc0dc15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0dc0b4450 0x7fc0dc145d40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 --2- 172.21.15.70:0/332359146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0dc0720b0 0x7fc0dc06e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 >> 172.21.15.70:0/332359146 conn(0x7fc0dc0b1d40 msgr2=0x7fc0dc0cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 shutdown_connections 2024-04-22T01:16:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.751+0000 7fc0e2eec640 1 -- 172.21.15.70:0/332359146 wait complete. 2024-04-22T01:16:23.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:23.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3752f15640 1 Processor -- start 2024-04-22T01:16:23.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3752f15640 1 -- start start 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3752f15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 0x7f374c0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3752f15640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374c0ccd40 con 0x7f374c09ae40 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3752f15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374c0cd020 con 0x7f374c15e030 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3752f15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f374c0cd300 con 0x7f374c09a440 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f374c09ae40 0x7f374c099d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33538/0 (socket says 172.21.15.70:33538) 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 -- 172.21.15.70:0/760928350 learned_addr learned my addr 172.21.15.70:0/760928350 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f37514b2640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 0x7f374c0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104424055 0 0) 0x7f374c0ccd40 con 0x7f374c09ae40 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3734003530 con 0x7f374c09ae40 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2434211084 0 0) 0x7f3734003530 con 0x7f374c09ae40 2024-04-22T01:16:23.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 msgr2=0x7f374c0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 0x7f374c0d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 >> v1:172.21.15.98:6789/0 conn(0x7f374c15e030 legacy=0x7f374c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c0cd5e0 con 0x7f374c09ae40 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f373c001680 con 0x7f374c09ae40 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 >> v1:172.21.15.70:6789/0 conn(0x7f374c09ae40 legacy=0x7f374c099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 0x7f37340039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3734003f70 0x7f3734004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3734004d50 0x7f37340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f374c0cd300 con 0x7f3734003f70 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f373400bab0 con 0x7f3734004d50 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f373400bcb0 con 0x7f374c09a440 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3734003f70 0x7f3734004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f37514b2640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3734004d50 0x7f37340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f3750cb1640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 0x7f37340039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 -- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 msgr2=0x7f37340039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:23.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f374c09a440 0x7f37340039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 -- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3734004d50 msgr2=0x7f37340051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:23.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3734004d50 0x7f37340051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.891+0000 7f374bfff640 1 -- 172.21.15.70:0/760928350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3734003530 con 0x7f3734003f70 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3734003f70 0x7f3734004810 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f374c072d60 tx=0x7f373c01f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb261784770f5d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c007d40 con 0x7f3734003f70 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c005b00 con 0x7f3734003f70 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374b7fe640 1 -- 172.21.15.70:0/760928350 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c007070 con 0x7f3734003f70 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- 172.21.15.70:0/760928350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3734003f70 msgr2=0x7f3734004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 --2- 172.21.15.70:0/760928350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3734003f70 0x7f3734004810 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f374c072d60 tx=0x7f373c01f430 comp rx=0 tx=0).stop 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f37514b2640 1 -- 172.21.15.70:0/760928350 reap_dead start 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- 172.21.15.70:0/760928350 shutdown_connections 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- 172.21.15.70:0/760928350 >> 172.21.15.70:0/760928350 conn(0x7f374c0b3700 msgr2=0x7f374c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- 172.21.15.70:0/760928350 shutdown_connections 2024-04-22T01:16:23.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- 172.21.15.70:0/760928350 wait complete. 2024-04-22T01:16:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 Processor -- start 2024-04-22T01:16:23.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- start start 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3734003f70 0x7f374c14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 0x7f374c0c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f374c0ca2e0 0x7f374c0b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f374c15e410 con 0x7f374c0cb8b0 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f374c0cc9c0 con 0x7f374c0ca2e0 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3752f15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f374c15e590 con 0x7f3734003f70 2024-04-22T01:16:23.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 0x7f374c0c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 0x7f374c0c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50682/0 (socket says 172.21.15.70:50682) 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 -- 172.21.15.70:0/503078139 learned_addr learned my addr 172.21.15.70:0/503078139 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f3750cb1640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3734003f70 0x7f374c14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f37514b2640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f374c0ca2e0 0x7f374c0b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 -- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3734003f70 msgr2=0x7f374c14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3734003f70 0x7f374c14e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 -- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f374c0ca2e0 msgr2=0x7f374c0b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f374c0ca2e0 0x7f374c0b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.895+0000 7f374bfff640 1 -- 172.21.15.70:0/503078139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c0ca810 con 0x7f374c0cb8b0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f374bfff640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 0x7f374c0c9da0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f373c000c10 tx=0x7f3734005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c001c90 con 0x7f374c0cb8b0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c005cb0 con 0x7f374c0cb8b0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374c0caaf0 con 0x7f374c0cb8b0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c005e60 con 0x7f374c0cb8b0 2024-04-22T01:16:23.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c0cb0b0 con 0x7f374c0cb8b0 2024-04-22T01:16:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f374c068920 con 0x7f374c0cb8b0 2024-04-22T01:16:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f373c0ba030 con 0x7f374c0cb8b0 2024-04-22T01:16:23.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f37497fa640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f37380678f0 0x7f3738069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f373c122810 con 0x7f374c0cb8b0 2024-04-22T01:16:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f3750cb1640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f37380678f0 0x7f3738069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:23.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.899+0000 7f3750cb1640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f37380678f0 0x7f3738069db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3740001720 tx=0x7f3740009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:23.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:23.903+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f373c019140 con 0x7f374c0cb8b0 2024-04-22T01:16:24.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f374c09a860 con 0x7f374c0cb8b0 2024-04-22T01:16:24.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f37497fa640 1 -- 172.21.15.70:0/503078139 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f373c0f31d0 con 0x7f374c0cb8b0 2024-04-22T01:16:24.050 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:16:24.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f37380678f0 msgr2=0x7f3738069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f37380678f0 0x7f3738069db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3740001720 tx=0x7f3740009040 comp rx=0 tx=0).stop 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 msgr2=0x7f374c0c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 0x7f374c0c9da0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f373c000c10 tx=0x7f3734005840 comp rx=0 tx=0).stop 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 shutdown_connections 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f37380678f0 0x7f3738069db0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f374c0ca2e0 0x7f374c0b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f374c0cb8b0 0x7f374c0c9da0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 --2- 172.21.15.70:0/503078139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3734003f70 0x7f374c14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 >> 172.21.15.70:0/503078139 conn(0x7f374c0b3700 msgr2=0x7f374c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 shutdown_connections 2024-04-22T01:16:24.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.051+0000 7f3752f15640 1 -- 172.21.15.70:0/503078139 wait complete. 2024-04-22T01:16:24.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:16:24.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 Processor -- start 2024-04-22T01:16:24.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 -- start start 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c1445b0 0x7f220c072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f220c0b4360 0x7f220c0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220c144990 con 0x7f220c0cdce0 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f220c0b3bd0 con 0x7f220c0b4360 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2210d40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f220c0b4d00 con 0x7f220c1445b0 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f220c0b4360 0x7f220c0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f220c0b4360 0x7f220c0b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50686/0 (socket says 172.21.15.70:50686) 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 -- 172.21.15.70:0/4157083043 learned_addr learned my addr 172.21.15.70:0/4157083043 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f220a59c640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c1445b0 0x7f220c072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 -- 172.21.15.70:0/4157083043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c1445b0 msgr2=0x7f220c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c1445b0 0x7f220c072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 -- 172.21.15.70:0/4157083043 >> v1:172.21.15.98:6789/0 conn(0x7f220c0cdce0 legacy=0x7f220c0ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:24.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f2209d9b640 1 -- 172.21.15.70:0/4157083043 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c0b2b50 con 0x7f220c0b4360 2024-04-22T01:16:24.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.175+0000 7f220a59c640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c1445b0 0x7f220c072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:24.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2209d9b640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0b4360 0x7f220c0b47c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009870 tx=0x7f21fc01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fd37e8d6d13052 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:24.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f220959a640 1 -- 172.21.15.70:0/4157083043 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0b0070 con 0x7f220c0b4360 2024-04-22T01:16:24.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f220959a640 1 -- 172.21.15.70:0/4157083043 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21fc0ab600 con 0x7f220c0b4360 2024-04-22T01:16:24.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f220959a640 1 -- 172.21.15.70:0/4157083043 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0ab9d0 con 0x7f220c0b4360 2024-04-22T01:16:24.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 -- 172.21.15.70:0/4157083043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0b4360 msgr2=0x7f220c0b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0b4360 0x7f220c0b47c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009870 tx=0x7f21fc01c310 comp rx=0 tx=0).stop 2024-04-22T01:16:24.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 -- 172.21.15.70:0/4157083043 shutdown_connections 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0b4360 0x7f220c0b47c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 --2- 172.21.15.70:0/4157083043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c1445b0 0x7f220c072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 -- 172.21.15.70:0/4157083043 >> 172.21.15.70:0/4157083043 conn(0x7f220c0cf9e0 msgr2=0x7f220c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 -- 172.21.15.70:0/4157083043 shutdown_connections 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.179+0000 7f2210d40640 1 -- 172.21.15.70:0/4157083043 wait complete. 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 Processor -- start 2024-04-22T01:16:24.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- start start 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c0b4360 0x7f220c13bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 0x7f220c1371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f220c1445b0 0x7f220c1289b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f220c072a40 con 0x7f220c0cdce0 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f220c0b36b0 con 0x7f220c1445b0 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f220c06edb0 con 0x7f220c0b4360 2024-04-22T01:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 0x7f220c1371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 0x7f220c1371c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50690/0 (socket says 172.21.15.70:50690) 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 -- 172.21.15.70:0/1783932292 learned_addr learned my addr 172.21.15.70:0/1783932292 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 -- 172.21.15.70:0/1783932292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c0b4360 msgr2=0x7f220c13bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c0b4360 0x7f220c13bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 -- 172.21.15.70:0/1783932292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f220c1445b0 msgr2=0x7f220c1289b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f220c1445b0 0x7f220c1289b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 -- 172.21.15.70:0/1783932292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c15b0f0 con 0x7f220c0cdce0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2209d9b640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 0x7f220c1371c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0ae040 tx=0x7f21fc0190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0ba070 con 0x7f220c0cdce0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21fc0b4640 con 0x7f220c0cdce0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c129470 con 0x7f220c0cdce0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0b0040 con 0x7f220c0cdce0 2024-04-22T01:16:24.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c15b3b0 con 0x7f220c0cdce0 2024-04-22T01:16:24.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f220c04f510 con 0x7f220c0cdce0 2024-04-22T01:16:24.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21fc0bd070 con 0x7f220c0cdce0 2024-04-22T01:16:24.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f21faffd640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21ec0678c0 0x7f21ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f21fc11efb0 con 0x7f220c0cdce0 2024-04-22T01:16:24.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f220a59c640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21ec0678c0 0x7f21ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.183+0000 7f220a59c640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21ec0678c0 0x7f21ec069d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f220000a890 tx=0x7f2200008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:24.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.187+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f220c144990 con 0x7f220c0cdce0 2024-04-22T01:16:24.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.355+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f220c074bd0 con 0x7f220c0cdce0 2024-04-22T01:16:24.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.355+0000 7f21faffd640 1 -- 172.21.15.70:0/1783932292 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7f21fc0ef9f0 con 0x7f220c0cdce0 2024-04-22T01:16:24.357 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:16:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21ec0678c0 msgr2=0x7f21ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21ec0678c0 0x7f21ec069d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f220000a890 tx=0x7f2200008040 comp rx=0 tx=0).stop 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 msgr2=0x7f220c1371c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 0x7f220c1371c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0ae040 tx=0x7f21fc0190b0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 shutdown_connections 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21ec0678c0 0x7f21ec069d80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f220c1445b0 0x7f220c1289b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f220c0cdce0 0x7f220c1371c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 --2- 172.21.15.70:0/1783932292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f220c0b4360 0x7f220c13bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 >> 172.21.15.70:0/1783932292 conn(0x7f220c0cf9e0 msgr2=0x7f220c142de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 shutdown_connections 2024-04-22T01:16:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.359+0000 7f2210d40640 1 -- 172.21.15.70:0/1783932292 wait complete. 2024-04-22T01:16:24.374 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:16:24.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:24.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:24.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[38566]: fuse finished with error 0 and tester_r 0 2024-04-22T01:16:24.386 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:16:24.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:24.386 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:16:24.552 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:24.552 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:16:24.552 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:24.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:24.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 Processor -- start 2024-04-22T01:16:24.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 -- start start 2024-04-22T01:16:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f80b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50f80cea80 0x7f50f80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f50f80b5730 0x7f50f80b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f50f80cf420 con 0x7f50f80b5730 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f50f815e390 con 0x7f50f80cea80 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50fe1d1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50f8161e00 con 0x7f50f815dfb0 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f50f80b5730 0x7f50f80b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f50f80b5730 0x7f50f80b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50694/0 (socket says 172.21.15.70:50694) 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 -- 172.21.15.70:0/38599533 learned_addr learned my addr 172.21.15.70:0/38599533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50effff640 1 --2- 172.21.15.70:0/38599533 >> v2:172.21.15.98:3300/0 conn(0x7f50f80cea80 0x7f50f80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f77fe640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f80b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 -- 172.21.15.70:0/38599533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 msgr2=0x7f50f80b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f80b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 -- 172.21.15.70:0/38599533 >> v2:172.21.15.98:3300/0 conn(0x7f50f80cea80 msgr2=0x7f50f80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:24.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 --2- 172.21.15.70:0/38599533 >> v2:172.21.15.98:3300/0 conn(0x7f50f80cea80 0x7f50f80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f7fff640 1 -- 172.21.15.70:0/38599533 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f81620e0 con 0x7f50f80b5730 2024-04-22T01:16:24.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.719+0000 7f50f77fe640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f80b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50f7fff640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80b5730 0x7f50f80b5b90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f50f80d4ce0 tx=0x7f50f001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b56c41c2d715e90 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50f6ffd640 1 -- 172.21.15.70:0/38599533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f00a5780 con 0x7f50f80b5730 2024-04-22T01:16:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50f6ffd640 1 -- 172.21.15.70:0/38599533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f00a5920 con 0x7f50f80b5730 2024-04-22T01:16:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50f6ffd640 1 -- 172.21.15.70:0/38599533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f00a5cf0 con 0x7f50f80b5730 2024-04-22T01:16:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- 172.21.15.70:0/38599533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80b5730 msgr2=0x7f50f80b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80b5730 0x7f50f80b5b90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f50f80d4ce0 tx=0x7f50f001c9f0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- 172.21.15.70:0/38599533 shutdown_connections 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80b5730 0x7f50f80b5b90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/38599533 >> v2:172.21.15.98:3300/0 conn(0x7f50f80cea80 0x7f50f80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/38599533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f80b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- 172.21.15.70:0/38599533 >> 172.21.15.70:0/38599533 conn(0x7f50f806c9b0 msgr2=0x7f50f806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- 172.21.15.70:0/38599533 shutdown_connections 2024-04-22T01:16:24.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- 172.21.15.70:0/38599533 wait complete. 2024-04-22T01:16:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 Processor -- start 2024-04-22T01:16:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- start start 2024-04-22T01:16:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f80b5730 0x7f50f815e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 0x7f50f815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50f8161a90 con 0x7f50f80cea80 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50f80b9b10 con 0x7f50f80b5730 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50fe1d1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50f80d2900 con 0x7f50f815dfb0 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 0x7f50f815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 0x7f50f815bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50698/0 (socket says 172.21.15.70:50698) 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.723+0000 7f50effff640 1 -- 172.21.15.70:0/27894949 learned_addr learned my addr 172.21.15.70:0/27894949 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f77fe640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f80b5730 0x7f50f815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f7fff640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50effff640 1 -- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 msgr2=0x7f50f815c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50effff640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f815c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50effff640 1 -- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f80b5730 msgr2=0x7f50f815e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50effff640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f80b5730 0x7f50f815e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50effff640 1 -- 172.21.15.70:0/27894949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f815ef10 con 0x7f50f80cea80 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f77fe640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f80b5730 0x7f50f815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50effff640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 0x7f50f815bc10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f50e000d800 tx=0x7f50e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e0018020 con 0x7f50f80cea80 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e00096a0 con 0x7f50f80cea80 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f8158140 con 0x7f50f80cea80 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e0009a50 con 0x7f50f80cea80 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f8158700 con 0x7f50f80cea80 2024-04-22T01:16:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f8068210 con 0x7f50f80cea80 2024-04-22T01:16:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50e001b020 con 0x7f50f80cea80 2024-04-22T01:16:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f4ff9640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678c0 0x7f50d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f50e0086300 con 0x7f50f80cea80 2024-04-22T01:16:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f77fe640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678c0 0x7f50d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.727+0000 7f50f77fe640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678c0 0x7f50d8069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f50e8008890 tx=0x7f50e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.731+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f50e0022020 con 0x7f50f80cea80 2024-04-22T01:16:24.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.891+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50f80b93f0 con 0x7f50f80cea80 2024-04-22T01:16:24.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50f4ff9640 1 -- 172.21.15.70:0/27894949 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16906 (secure 0 0 0) 0x7f50e0056d40 con 0x7f50f80cea80 2024-04-22T01:16:24.894 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:24.894 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:12.187805+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:24.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678c0 msgr2=0x7f50d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678c0 0x7f50d8069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f50e8008890 tx=0x7f50e8008040 comp rx=0 tx=0).stop 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 msgr2=0x7f50f815bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 0x7f50f815bc10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f50e000d800 tx=0x7f50e000dcc0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 shutdown_connections 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678c0 0x7f50d8069d80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f815dfb0 0x7f50f815c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f80cea80 0x7f50f815bc10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 --2- 172.21.15.70:0/27894949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f80b5730 0x7f50f815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 >> 172.21.15.70:0/27894949 conn(0x7f50f806c9b0 msgr2=0x7f50f80b81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 shutdown_connections 2024-04-22T01:16:24.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:24.895+0000 7f50fe1d1640 1 -- 172.21.15.70:0/27894949 wait complete. 2024-04-22T01:16:24.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:16:08.953522+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:16:09.533622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:24.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:25.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.035+0000 7fbe424a4640 1 Processor -- start 2024-04-22T01:16:25.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.035+0000 7fbe424a4640 1 -- start start 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe424a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 0x7fbe3c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe424a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe3c0720b0 0x7fbe3c0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe424a4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe3c15c290 con 0x7fbe3c0cd050 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe424a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbe3c072510 con 0x7fbe3c0720b0 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe424a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c0b3a50 con 0x7fbe3c0b4450 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe40a41640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbe3c0cd050 0x7fbe3c15f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33546/0 (socket says 172.21.15.70:33546) 2024-04-22T01:16:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe40a41640 1 -- 172.21.15.70:0/3817377375 learned_addr learned my addr 172.21.15.70:0/3817377375 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270781655 0 0) 0x7fbe3c15c290 con 0x7fbe3c0cd050 2024-04-22T01:16:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 --2- 172.21.15.70:0/3817377375 >> v2:172.21.15.98:3300/0 conn(0x7fbe3c0720b0 0x7fbe3c0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3bfff640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 0x7fbe3c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe24003530 con 0x7fbe3c0cd050 2024-04-22T01:16:25.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3932248893 0 0) 0x7fbe24003530 con 0x7fbe3c0cd050 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 msgr2=0x7fbe3c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 0x7fbe3c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 >> v2:172.21.15.98:3300/0 conn(0x7fbe3c0720b0 msgr2=0x7fbe3c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 --2- 172.21.15.70:0/3817377375 >> v2:172.21.15.98:3300/0 conn(0x7fbe3c0720b0 0x7fbe3c0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c15c570 con 0x7fbe3c0cd050 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fbe34001a00 con 0x7fbe3c0cd050 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 >> v1:172.21.15.70:6789/0 conn(0x7fbe3c0cd050 legacy=0x7fbe3c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 0x7fbe240039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe24003f40 0x7fbe24004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24004d50 0x7fbe240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c0b3a50 con 0x7fbe24003f40 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2400b9d0 con 0x7fbe24004d50 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2400bb50 con 0x7fbe3c0b4450 2024-04-22T01:16:25.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe24003f40 0x7fbe24004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3bfff640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 0x7fbe240039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 -- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 msgr2=0x7fbe240039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0b4450 0x7fbe240039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 -- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24004d50 msgr2=0x7fbe240051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24004d50 0x7fbe240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.039+0000 7fbe3b7fe640 1 -- 172.21.15.70:0/3817377375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe24003530 con 0x7fbe24003f40 2024-04-22T01:16:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe3b7fe640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe24003f40 0x7fbe24004810 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009ae0 tx=0x7fbe3001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57514298d292eb64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe300b2070 con 0x7fbe24003f40 2024-04-22T01:16:25.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe300a56a0 con 0x7fbe24003f40 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe3affd640 1 -- 172.21.15.70:0/3817377375 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe300096b0 con 0x7fbe24003f40 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/3817377375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe24003f40 msgr2=0x7fbe24004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 --2- 172.21.15.70:0/3817377375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe24003f40 0x7fbe24004810 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009ae0 tx=0x7fbe3001ca50 comp rx=0 tx=0).stop 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 -- 172.21.15.70:0/3817377375 reap_dead start 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/3817377375 shutdown_connections 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/3817377375 >> 172.21.15.70:0/3817377375 conn(0x7fbe3c0b1d40 msgr2=0x7fbe3c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/3817377375 shutdown_connections 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/3817377375 wait complete. 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 Processor -- start 2024-04-22T01:16:25.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- start start 2024-04-22T01:16:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24003f40 0x7fbe3c1443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0ab100 0x7fbe3c0ab580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 0x7fbe3c159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15d050 con 0x7fbe3c1599f0 2024-04-22T01:16:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15cb30 con 0x7fbe24003f40 2024-04-22T01:16:25.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c0cd4c0 con 0x7fbe3c0ab100 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 0x7fbe3c159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 0x7fbe3c159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50724/0 (socket says 172.21.15.70:50724) 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 -- 172.21.15.70:0/2335749082 learned_addr learned my addr 172.21.15.70:0/2335749082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe3bfff640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24003f40 0x7fbe3c1443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe3b7fe640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0ab100 0x7fbe3c0ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 -- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0ab100 msgr2=0x7fbe3c0ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0ab100 0x7fbe3c0ab580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 -- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24003f40 msgr2=0x7fbe3c1443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24003f40 0x7fbe3c1443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 -- 172.21.15.70:0/2335749082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c0abac0 con 0x7fbe3c1599f0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe40a41640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 0x7fbe3c159e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0cd430 tx=0x7fbe3400ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3400da20 con 0x7fbe3c1599f0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe3400dbc0 con 0x7fbe3c1599f0 2024-04-22T01:16:25.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3c15a760 con 0x7fbe3c1599f0 2024-04-22T01:16:25.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3400a0d0 con 0x7fbe3c1599f0 2024-04-22T01:16:25.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.043+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3c15ad20 con 0x7fbe3c1599f0 2024-04-22T01:16:25.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.047+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3c068210 con 0x7fbe3c1599f0 2024-04-22T01:16:25.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.047+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe34014070 con 0x7fbe3c1599f0 2024-04-22T01:16:25.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.047+0000 7fbe38ff9640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbe280678c0 0x7fbe28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.047+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fbe34088c20 con 0x7fbe3c1599f0 2024-04-22T01:16:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.047+0000 7fbe3bfff640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbe280678c0 0x7fbe28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.047+0000 7fbe3bfff640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbe280678c0 0x7fbe28069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009950 tx=0x7fbe2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.051+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbe3401d060 con 0x7fbe3c1599f0 2024-04-22T01:16:25.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.199+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbe3c0d01e0 con 0x7fbe3c1599f0 2024-04-22T01:16:25.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe38ff9640 1 -- 172.21.15.70:0/2335749082 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7fbe34059720 con 0x7fbe3c1599f0 2024-04-22T01:16:25.202 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbe280678c0 msgr2=0x7fbe28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbe280678c0 0x7fbe28069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009950 tx=0x7fbe2c008040 comp rx=0 tx=0).stop 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 msgr2=0x7fbe3c159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 0x7fbe3c159e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0cd430 tx=0x7fbe3400ff00 comp rx=0 tx=0).stop 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 shutdown_connections 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbe280678c0 0x7fbe28069d80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe3c1599f0 0x7fbe3c159e70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe3c0ab100 0x7fbe3c0ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 --2- 172.21.15.70:0/2335749082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe24003f40 0x7fbe3c1443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 >> 172.21.15.70:0/2335749082 conn(0x7fbe3c0b1d40 msgr2=0x7fbe3c0cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 shutdown_connections 2024-04-22T01:16:25.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.203+0000 7fbe424a4640 1 -- 172.21.15.70:0/2335749082 wait complete. 2024-04-22T01:16:25.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 Processor -- start 2024-04-22T01:16:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 -- start start 2024-04-22T01:16:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe3f00b4450 0x7fe3f0071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00720b0 0x7fe3f00ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3f015c290 con 0x7fe3f00cd050 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f0072510 con 0x7fe3f00b4450 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f6baa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f00b3a50 con 0x7fe3f00720b0 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe3f00b4450 0x7fe3f0071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe3f00b4450 0x7fe3f0071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50726/0 (socket says 172.21.15.70:50726) 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f5147640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe3f00cd050 0x7fe3f015f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33554/0 (socket says 172.21.15.70:33554) 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 -- 172.21.15.70:0/42617719 learned_addr learned my addr 172.21.15.70:0/42617719 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3effff640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00720b0 0x7fe3f00ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 -- 172.21.15.70:0/42617719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00720b0 msgr2=0x7fe3f00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00720b0 0x7fe3f00ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 -- 172.21.15.70:0/42617719 >> v1:172.21.15.70:6789/0 conn(0x7fe3f00cd050 legacy=0x7fe3f015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3ef7fe640 1 -- 172.21.15.70:0/42617719 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424037253 0 0) 0x7fe3f015c290 con 0x7fe3f00cd050 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3f4946640 1 -- 172.21.15.70:0/42617719 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f015c570 con 0x7fe3f00b4450 2024-04-22T01:16:25.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.343+0000 7fe3effff640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00720b0 0x7fe3f00ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:25.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00b4450 0x7fe3f0071b70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009870 tx=0x7fe3e401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c95fd91a457a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3ef7fe640 1 -- 172.21.15.70:0/42617719 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40ab650 con 0x7fe3f00b4450 2024-04-22T01:16:25.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3ef7fe640 1 -- 172.21.15.70:0/42617719 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e40b0070 con 0x7fe3f00b4450 2024-04-22T01:16:25.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3ef7fe640 1 -- 172.21.15.70:0/42617719 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40aba10 con 0x7fe3f00b4450 2024-04-22T01:16:25.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/42617719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00b4450 msgr2=0x7fe3f0071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00b4450 0x7fe3f0071b70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009870 tx=0x7fe3e401c530 comp rx=0 tx=0).stop 2024-04-22T01:16:25.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/42617719 shutdown_connections 2024-04-22T01:16:25.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00720b0 0x7fe3f00ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/42617719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00b4450 0x7fe3f0071b70 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/42617719 >> 172.21.15.70:0/42617719 conn(0x7fe3f00b1d40 msgr2=0x7fe3f00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/42617719 shutdown_connections 2024-04-22T01:16:25.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/42617719 wait complete. 2024-04-22T01:16:25.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 Processor -- start 2024-04-22T01:16:25.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- start start 2024-04-22T01:16:25.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 0x7fe3f0147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f00b4450 0x7fe3f014c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00cd050 0x7fe3f014a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f00b4830 con 0x7fe3f00720b0 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f00cd480 con 0x7fe3f00b4450 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f00b49b0 con 0x7fe3f00cd050 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 0x7fe3f0147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 0x7fe3f0147710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50728/0 (socket says 172.21.15.70:50728) 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 -- 172.21.15.70:0/1064646148 learned_addr learned my addr 172.21.15.70:0/1064646148 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 -- 172.21.15.70:0/1064646148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00cd050 msgr2=0x7fe3f014a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00cd050 0x7fe3f014a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 -- 172.21.15.70:0/1064646148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f00b4450 msgr2=0x7fe3f014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f00b4450 0x7fe3f014c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 -- 172.21.15.70:0/1064646148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f0148cd0 con 0x7fe3f00720b0 2024-04-22T01:16:25.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f4946640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 0x7fe3f0147710 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40ad040 tx=0x7fe3e401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40abe60 con 0x7fe3f00720b0 2024-04-22T01:16:25.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e40b4820 con 0x7fe3f00720b0 2024-04-22T01:16:25.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f0148fb0 con 0x7fe3f00720b0 2024-04-22T01:16:25.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40b4c10 con 0x7fe3f00720b0 2024-04-22T01:16:25.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.347+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f01497b0 con 0x7fe3f00720b0 2024-04-22T01:16:25.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.351+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f0150c50 con 0x7fe3f00720b0 2024-04-22T01:16:25.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.351+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe3e40c7390 con 0x7fe3f00720b0 2024-04-22T01:16:25.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.351+0000 7fe3ed7fa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3d40678c0 0x7fe3d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.351+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe3e4120760 con 0x7fe3f00720b0 2024-04-22T01:16:25.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.351+0000 7fe3effff640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3d40678c0 0x7fe3d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.351+0000 7fe3effff640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3d40678c0 0x7fe3d4069d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3f01548a0 tx=0x7fe3dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.355+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe3e40c4060 con 0x7fe3f00720b0 2024-04-22T01:16:25.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.511+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe3f0068210 con 0x7fe3f00720b0 2024-04-22T01:16:25.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3ed7fa640 1 -- 172.21.15.70:0/1064646148 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7fe3e4016150 con 0x7fe3f00720b0 2024-04-22T01:16:25.514 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3d40678c0 msgr2=0x7fe3d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3d40678c0 0x7fe3d4069d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3f01548a0 tx=0x7fe3dc008040 comp rx=0 tx=0).stop 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 msgr2=0x7fe3f0147710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 0x7fe3f0147710 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40ad040 tx=0x7fe3e401ad70 comp rx=0 tx=0).stop 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 shutdown_connections 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3d40678c0 0x7fe3d4069d80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3f00cd050 0x7fe3f014a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f00b4450 0x7fe3f014c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 --2- 172.21.15.70:0/1064646148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3f00720b0 0x7fe3f0147710 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 >> 172.21.15.70:0/1064646148 conn(0x7fe3f00b1d40 msgr2=0x7fe3f014f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 shutdown_connections 2024-04-22T01:16:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.515+0000 7fe3f6baa640 1 -- 172.21.15.70:0/1064646148 wait complete. 2024-04-22T01:16:25.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 Processor -- start 2024-04-22T01:16:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 -- start start 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a0154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe6a00ceda0 0x7fe6a01551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6a00b3800 con 0x7fe6a015e6e0 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe6a00b3ae0 con 0x7fe6a00ceda0 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a86d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a00cde40 con 0x7fe6a00ce920 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a646e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe6a015e6e0 0x7fe6a015eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33570/0 (socket says 172.21.15.70:33570) 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a646e640 1 -- 172.21.15.70:0/2561544007 learned_addr learned my addr 172.21.15.70:0/2561544007 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a6c6f640 1 --2- 172.21.15.70:0/2561544007 >> v2:172.21.15.70:3300/0 conn(0x7fe6a00ceda0 0x7fe6a01551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a5c6d640 1 --2- 172.21.15.70:0/2561544007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a0154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2561544007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 msgr2=0x7fe6a0154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a6c6f640 1 --2- 172.21.15.70:0/2561544007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a0154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2561544007 >> v1:172.21.15.70:6789/0 conn(0x7fe6a015e6e0 legacy=0x7fe6a015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a546c640 1 -- 172.21.15.70:0/2561544007 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435771203 0 0) 0x7fe6a00b3800 con 0x7fe6a015e6e0 2024-04-22T01:16:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.655+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2561544007 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a00ce120 con 0x7fe6a00ceda0 2024-04-22T01:16:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 --2- 172.21.15.70:0/2561544007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a00ceda0 0x7fe6a01551a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe698009d90 tx=0x7fe69801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4c9dafaedb8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a546c640 1 -- 172.21.15.70:0/2561544007 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980b2070 con 0x7fe6a00ceda0 2024-04-22T01:16:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a546c640 1 -- 172.21.15.70:0/2561544007 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6980a5a20 con 0x7fe6a00ceda0 2024-04-22T01:16:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a546c640 1 -- 172.21.15.70:0/2561544007 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980a5df0 con 0x7fe6a00ceda0 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2561544007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a00ceda0 msgr2=0x7fe6a01551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2561544007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a00ceda0 0x7fe6a01551a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe698009d90 tx=0x7fe69801c940 comp rx=0 tx=0).stop 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2561544007 shutdown_connections 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2561544007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a00ceda0 0x7fe6a01551a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2561544007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a0154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2561544007 >> 172.21.15.70:0/2561544007 conn(0x7fe6a00b1d40 msgr2=0x7fe6a00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2561544007 shutdown_connections 2024-04-22T01:16:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2561544007 wait complete. 2024-04-22T01:16:25.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 Processor -- start 2024-04-22T01:16:25.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- start start 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a01480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a00ceda0 0x7fe6a0147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 0x7fe6a0071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a00b2e80 con 0x7fe6a015e6e0 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a00b2d00 con 0x7fe6a00ceda0 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a86d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a00b3000 con 0x7fe6a00ce920 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 0x7fe6a0071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 0x7fe6a0071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50744/0 (socket says 172.21.15.70:50744) 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2608721927 learned_addr learned my addr 172.21.15.70:0/2608721927 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a646e640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a01480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 msgr2=0x7fe6a01480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a5c6d640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a00ceda0 0x7fe6a0147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a01480a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a00ceda0 msgr2=0x7fe6a0147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a00ceda0 0x7fe6a0147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 -- 172.21.15.70:0/2608721927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a00b28e0 con 0x7fe6a015e6e0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.659+0000 7fe6a6c6f640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 0x7fe6a0071b70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe69801aff0 tx=0x7fe6980196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980b2050 con 0x7fe6a015e6e0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6980be070 con 0x7fe6a015e6e0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a0155dc0 con 0x7fe6a015e6e0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980adad0 con 0x7fe6a015e6e0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a0156380 con 0x7fe6a015e6e0 2024-04-22T01:16:25.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a0068210 con 0x7fe6a015e6e0 2024-04-22T01:16:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe6980adc70 con 0x7fe6a015e6e0 2024-04-22T01:16:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe696ffd640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6840678c0 0x7fe684069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe698121310 con 0x7fe6a015e6e0 2024-04-22T01:16:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe6a646e640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6840678c0 0x7fe684069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.663+0000 7fe6a646e640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6840678c0 0x7fe684069d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe69c000f70 tx=0x7fe69c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.667+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe698016560 con 0x7fe6a015e6e0 2024-04-22T01:16:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.827+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe6a0159390 con 0x7fe6a015e6e0 2024-04-22T01:16:25.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.827+0000 7fe696ffd640 1 -- 172.21.15.70:0/2608721927 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7fe6980f1c30 con 0x7fe6a015e6e0 2024-04-22T01:16:25.828 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6840678c0 msgr2=0x7fe684069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6840678c0 0x7fe684069d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe69c000f70 tx=0x7fe69c009040 comp rx=0 tx=0).stop 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 msgr2=0x7fe6a0071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 0x7fe6a0071b70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe69801aff0 tx=0x7fe6980196e0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 shutdown_connections 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6840678c0 0x7fe684069d80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6a015e6e0 0x7fe6a0071b70 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6a00ceda0 0x7fe6a0147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 --2- 172.21.15.70:0/2608721927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6a00ce920 0x7fe6a01480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 >> 172.21.15.70:0/2608721927 conn(0x7fe6a00b1d40 msgr2=0x7fe6a0159670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 shutdown_connections 2024-04-22T01:16:25.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.831+0000 7fe6a86d2640 1 -- 172.21.15.70:0/2608721927 wait complete. 2024-04-22T01:16:25.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:16:25.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 Processor -- start 2024-04-22T01:16:25.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 -- start start 2024-04-22T01:16:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdabc0cecf0 0x7fdabc0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdabc0b48a0 0x7fdabc1646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc144990 con 0x7fdabc1445b0 2024-04-22T01:16:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc0b3bd0 con 0x7fdabc0cecf0 2024-04-22T01:16:25.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdac151b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc0b4cd0 con 0x7fdabc0b48a0 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdaba7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdabc0cecf0 0x7fdabc0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdabaffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdabc1445b0 0x7fdabc072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33572/0 (socket says 172.21.15.70:33572) 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdabaffd640 1 -- 172.21.15.70:0/4248669562 learned_addr learned my addr 172.21.15.70:0/4248669562 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdabb7fe640 1 --2- 172.21.15.70:0/4248669562 >> v2:172.21.15.98:3300/0 conn(0x7fdabc0b48a0 0x7fdabc1646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdaba7fc640 1 -- 172.21.15.70:0/4248669562 >> v2:172.21.15.98:3300/0 conn(0x7fdabc0b48a0 msgr2=0x7fdabc1646a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdaba7fc640 1 --2- 172.21.15.70:0/4248669562 >> v2:172.21.15.98:3300/0 conn(0x7fdabc0b48a0 0x7fdabc1646a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdaba7fc640 1 -- 172.21.15.70:0/4248669562 >> v1:172.21.15.70:6789/0 conn(0x7fdabc1445b0 legacy=0x7fdabc072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.963+0000 7fdaba7fc640 1 -- 172.21.15.70:0/4248669562 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc0b2b50 con 0x7fdabc0cecf0 2024-04-22T01:16:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdaba7fc640 1 --2- 172.21.15.70:0/4248669562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc0b4360 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdabc06fd00 tx=0x7fdab001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c09c39a19ed05c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdab9ffb640 1 -- 172.21.15.70:0/4248669562 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab00b0070 con 0x7fdabc0cecf0 2024-04-22T01:16:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdab9ffb640 1 -- 172.21.15.70:0/4248669562 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab0004230 con 0x7fdabc0cecf0 2024-04-22T01:16:25.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdab9ffb640 1 -- 172.21.15.70:0/4248669562 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0003620 con 0x7fdabc0cecf0 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 -- 172.21.15.70:0/4248669562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 msgr2=0x7fdabc0b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 --2- 172.21.15.70:0/4248669562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc0b4360 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdabc06fd00 tx=0x7fdab001c460 comp rx=0 tx=0).stop 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 -- 172.21.15.70:0/4248669562 shutdown_connections 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 --2- 172.21.15.70:0/4248669562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc0b4360 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 --2- 172.21.15.70:0/4248669562 >> v2:172.21.15.98:3300/0 conn(0x7fdabc0b48a0 0x7fdabc1646a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 -- 172.21.15.70:0/4248669562 >> 172.21.15.70:0/4248669562 conn(0x7fdabc0cf9e0 msgr2=0x7fdabc0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 -- 172.21.15.70:0/4248669562 shutdown_connections 2024-04-22T01:16:25.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 -- 172.21.15.70:0/4248669562 wait complete. 2024-04-22T01:16:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 Processor -- start 2024-04-22T01:16:25.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.967+0000 7fdac151b640 1 -- start start 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc0b48a0 0x7fdabc1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc1445b0 0x7fdabc15d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc06edb0 con 0x7fdabc0cecf0 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc0b36b0 con 0x7fdabc0b48a0 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc06ef30 con 0x7fdabc1445b0 2024-04-22T01:16:25.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50774/0 (socket says 172.21.15.70:50774) 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 -- 172.21.15.70:0/2160375953 learned_addr learned my addr 172.21.15.70:0/2160375953 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdabaffd640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc0b48a0 0x7fdabc1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdabb7fe640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc1445b0 0x7fdabc15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 -- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc1445b0 msgr2=0x7fdabc15d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc1445b0 0x7fdabc15d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 -- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc0b48a0 msgr2=0x7fdabc1592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc0b48a0 0x7fdabc1592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 -- 172.21.15.70:0/2160375953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc158a60 con 0x7fdabc0cecf0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdabaffd640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc0b48a0 0x7fdabc1592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaba7fc640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc142f40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdab00ae040 tx=0x7fdab0019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab00b9020 con 0x7fdabc0cecf0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab00043e0 con 0x7fdabc0cecf0 2024-04-22T01:16:25.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc1579c0 con 0x7fdabc0cecf0 2024-04-22T01:16:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab00b0040 con 0x7fdabc0cecf0 2024-04-22T01:16:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc157f80 con 0x7fdabc0cecf0 2024-04-22T01:16:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc04f510 con 0x7fdabc0cecf0 2024-04-22T01:16:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdab00bc020 con 0x7fdabc0cecf0 2024-04-22T01:16:25.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaa37fe640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdaa80678c0 0x7fdaa8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:25.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fdab011fef0 con 0x7fdabc0cecf0 2024-04-22T01:16:25.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdabaffd640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdaa80678c0 0x7fdaa8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:25.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.971+0000 7fdabaffd640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdaa80678c0 0x7fdaa8069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdabc1f8230 tx=0x7fdaac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:25.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:25.975+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdab00c3030 con 0x7fdabc0cecf0 2024-04-22T01:16:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdabc072430 con 0x7fdabc0cecf0 2024-04-22T01:16:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdaa37fe640 1 -- 172.21.15.70:0/2160375953 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7fdabc158a60 con 0x7fdabc0cecf0 2024-04-22T01:16:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:16:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdaa80678c0 msgr2=0x7fdaa8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdaa80678c0 0x7fdaa8069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdabc1f8230 tx=0x7fdaac009040 comp rx=0 tx=0).stop 2024-04-22T01:16:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 msgr2=0x7fdabc142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc142f40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdab00ae040 tx=0x7fdab0019200 comp rx=0 tx=0).stop 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 shutdown_connections 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdaa80678c0 0x7fdaa8069d80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc1445b0 0x7fdabc15d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdabc0cecf0 0x7fdabc142f40 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 --2- 172.21.15.70:0/2160375953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc0b48a0 0x7fdabc1592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 >> 172.21.15.70:0/2160375953 conn(0x7fdabc0cf9e0 msgr2=0x7fdabc06bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 shutdown_connections 2024-04-22T01:16:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.139+0000 7fdac151b640 1 -- 172.21.15.70:0/2160375953 wait complete. 2024-04-22T01:16:26.156 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:16:26.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:26.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:26.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[33903]: fuse finished with error 0 and tester_r 0 2024-04-22T01:16:26.171 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:16:26.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:26.171 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:16:26.308 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:26.309 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:16:26.309 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:26.315 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:16:26.315 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:16:26.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:26.315 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:16:26.361 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-22T01:16:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:26.363 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:26.363 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:16:26.363 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:26.369 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:16:26.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:16:26.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:26.369 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:16:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-22T01:16:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:26.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:26.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 Processor -- start 2024-04-22T01:16:26.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 -- start start 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11b00cea80 0x7f11b00ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11b00d2110 0x7f11b00d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b015e390 con 0x7f11b015dfb0 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f11b00cf420 con 0x7f11b00cea80 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b7ee1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f11b0161e00 con 0x7f11b00d2110 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b547c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11b00cea80 0x7f11b00ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b5c7d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f11b015dfb0 0x7f11b00b95d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33948/0 (socket says 172.21.15.70:33948) 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b5c7d640 1 -- 172.21.15.70:0/3980782004 learned_addr learned my addr 172.21.15.70:0/3980782004 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:26.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b4c7b640 1 -- 172.21.15.70:0/3980782004 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 517500829 0 0) 0x7f11b015e390 con 0x7f11b015dfb0 2024-04-22T01:16:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b4c7b640 1 -- 172.21.15.70:0/3980782004 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1198003530 con 0x7f11b015dfb0 2024-04-22T01:16:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b547c640 1 -- 172.21.15.70:0/3980782004 >> v2:172.21.15.98:3300/0 conn(0x7f11b00d2110 msgr2=0x7f11b00d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b547c640 1 --2- 172.21.15.70:0/3980782004 >> v2:172.21.15.98:3300/0 conn(0x7f11b00d2110 0x7f11b00d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b547c640 1 -- 172.21.15.70:0/3980782004 >> v1:172.21.15.70:6789/0 conn(0x7f11b015dfb0 legacy=0x7f11b00b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b547c640 1 -- 172.21.15.70:0/3980782004 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b01620e0 con 0x7f11b00cea80 2024-04-22T01:16:26.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.543+0000 7f11b547c640 1 --2- 172.21.15.70:0/3980782004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b00cea80 0x7f11b00ceee0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f11b00ce4b0 tx=0x7f11a401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171ee176fc4d6ac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:26.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b4c7b640 1 -- 172.21.15.70:0/3980782004 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a40b1070 con 0x7f11b00cea80 2024-04-22T01:16:26.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b4c7b640 1 -- 172.21.15.70:0/3980782004 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a40ac5f0 con 0x7f11b00cea80 2024-04-22T01:16:26.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b4c7b640 1 -- 172.21.15.70:0/3980782004 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a40ac9c0 con 0x7f11b00cea80 2024-04-22T01:16:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3980782004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b00cea80 msgr2=0x7f11b00ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3980782004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b00cea80 0x7f11b00ceee0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f11b00ce4b0 tx=0x7f11a401c480 comp rx=0 tx=0).stop 2024-04-22T01:16:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3980782004 shutdown_connections 2024-04-22T01:16:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3980782004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b00cea80 0x7f11b00ceee0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3980782004 >> v2:172.21.15.98:3300/0 conn(0x7f11b00d2110 0x7f11b00d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3980782004 >> 172.21.15.70:0/3980782004 conn(0x7f11b006c9b0 msgr2=0x7f11b006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3980782004 shutdown_connections 2024-04-22T01:16:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3980782004 wait complete. 2024-04-22T01:16:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 Processor -- start 2024-04-22T01:16:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- start start 2024-04-22T01:16:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b00cea80 0x7f11b0160e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b00d2110 0x7f11b0155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 0x7f11b00c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11b016af50 con 0x7f11b015dfb0 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11b016add0 con 0x7f11b00d2110 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f11b016b0d0 con 0x7f11b00cea80 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 0x7f11b00c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 0x7f11b00c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47508/0 (socket says 172.21.15.70:47508) 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 -- 172.21.15.70:0/3073849679 learned_addr learned my addr 172.21.15.70:0/3073849679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b547c640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b00d2110 0x7f11b0155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b5c7d640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b00cea80 0x7f11b0160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 -- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b00cea80 msgr2=0x7f11b0160e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b00cea80 0x7f11b0160e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 -- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b00d2110 msgr2=0x7f11b0155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b00d2110 0x7f11b0155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 -- 172.21.15.70:0/3073849679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b00c8c90 con 0x7f11b015dfb0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b647e640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 0x7f11b00c9d30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f11a000b390 tx=0x7f11a000b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a0015020 con 0x7f11b015dfb0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a0011830 con 0x7f11b015dfb0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b00c8f70 con 0x7f11b015dfb0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a0011c20 con 0x7f11b015dfb0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.547+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b00c9530 con 0x7f11b015dfb0 2024-04-22T01:16:26.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.551+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b0068210 con 0x7f11b015dfb0 2024-04-22T01:16:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.551+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11a0018020 con 0x7f11b015dfb0 2024-04-22T01:16:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.551+0000 7f11ae7fc640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f119c0678c0 0x7f119c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.551+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f11a0085db0 con 0x7f11b015dfb0 2024-04-22T01:16:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.551+0000 7f11b5c7d640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f119c0678c0 0x7f119c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.551+0000 7f11b5c7d640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f119c0678c0 0x7f119c069d80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f11a80012f0 tx=0x7f11a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:26.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.555+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f11a001f020 con 0x7f11b015dfb0 2024-04-22T01:16:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.747+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11b00d1950 con 0x7f11b015dfb0 2024-04-22T01:16:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.747+0000 7f11ae7fc640 1 -- 172.21.15.70:0/3073849679 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5178 (secure 0 0 0) 0x7f11a00172c0 con 0x7f11b015dfb0 2024-04-22T01:16:26.749 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:26.749 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:10.524071+0000","modified":"2024-04-22T01:16:13.536979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5478":{"gid":5478,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3248824947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3248824947},{"type":"v1","addr":"172.21.15.70:6837","nonce":3248824947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5478,"qdb_cluster":[5478]},"id":2}]} 2024-04-22T01:16:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f119c0678c0 msgr2=0x7f119c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f119c0678c0 0x7f119c069d80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f11a80012f0 tx=0x7f11a8009040 comp rx=0 tx=0).stop 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 msgr2=0x7f11b00c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 0x7f11b00c9d30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f11a000b390 tx=0x7f11a000b850 comp rx=0 tx=0).stop 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 shutdown_connections 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f119c0678c0 0x7f119c069d80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11b015dfb0 0x7f11b00c9d30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b00d2110 0x7f11b0155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 --2- 172.21.15.70:0/3073849679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b00cea80 0x7f11b0160e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 >> 172.21.15.70:0/3073849679 conn(0x7f11b006c9b0 msgr2=0x7f11b015d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 shutdown_connections 2024-04-22T01:16:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.751+0000 7f11b7ee1640 1 -- 172.21.15.70:0/3073849679 wait complete. 2024-04-22T01:16:26.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:26.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c5040d640 1 Processor -- start 2024-04-22T01:16:26.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c5040d640 1 -- start start 2024-04-22T01:16:26.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c5040d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6c4815bc70 0x7f6c4815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c5040d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c4815fd10 con 0x7f6c480cd740 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c5040d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c4815c050 con 0x7f6c4806b900 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c5040d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6c480cfd10 con 0x7f6c4815bc70 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c4d9a8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6c480cd740 0x7f6c4806b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33950/0 (socket says 172.21.15.70:33950) 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c4d9a8640 1 -- 172.21.15.70:0/56436035 learned_addr learned my addr 172.21.15.70:0/56436035 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/56436035 >> v2:172.21.15.98:3300/0 conn(0x7f6c4815bc70 0x7f6c4815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114536969 0 0) 0x7f6c44001120 con 0x7f6c480cd740 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.887+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c34003530 con 0x7f6c480cd740 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 964450614 0 0) 0x7f6c34003530 con 0x7f6c480cd740 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 >> v2:172.21.15.98:3300/0 conn(0x7f6c4815bc70 msgr2=0x7f6c4815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 --2- 172.21.15.70:0/56436035 >> v2:172.21.15.98:3300/0 conn(0x7f6c4815bc70 0x7f6c4815f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 >> v1:172.21.15.98:6789/0 conn(0x7f6c4806b900 legacy=0x7f6c480d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4815ff90 con 0x7f6c480cd740 2024-04-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6c44001850 con 0x7f6c480cd740 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 >> v1:172.21.15.70:6789/0 conn(0x7f6c480cd740 legacy=0x7f6c4806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c340039c0 0x7f6c34003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c34004810 0x7f6c34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340051d0 0x7f6c3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c480cfd10 con 0x7f6c340039c0 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3400c070 con 0x7f6c340051d0 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3400c270 con 0x7f6c34004810 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c340039c0 0x7f6c34003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d9a8640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c34004810 0x7f6c34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 -- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c34004810 msgr2=0x7f6c34004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c34004810 0x7f6c34004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 -- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340051d0 msgr2=0x7f6c3400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340051d0 0x7f6c3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 -- 172.21.15.70:0/56436035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c34003530 con 0x7f6c340039c0 2024-04-22T01:16:26.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e9aa640 1 -- 172.21.15.70:0/56436035 reap_dead start 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c340039c0 0x7f6c34003de0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009ae0 tx=0x7f6c3801ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921c8462d3ff4f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c380b2070 con 0x7f6c340039c0 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c380a5560 con 0x7f6c340039c0 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c4d1a7640 1 -- 172.21.15.70:0/56436035 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c380096b0 con 0x7f6c340039c0 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 -- 172.21.15.70:0/56436035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c340039c0 msgr2=0x7f6c34003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c340039c0 0x7f6c34003de0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009ae0 tx=0x7f6c3801ca80 comp rx=0 tx=0).stop 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 -- 172.21.15.70:0/56436035 shutdown_connections 2024-04-22T01:16:26.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 --2- 172.21.15.70:0/56436035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c340039c0 0x7f6c34003de0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 -- 172.21.15.70:0/56436035 >> 172.21.15.70:0/56436035 conn(0x7f6c48099a60 msgr2=0x7f6c48099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 -- 172.21.15.70:0/56436035 shutdown_connections 2024-04-22T01:16:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.891+0000 7f6c5040d640 1 -- 172.21.15.70:0/56436035 wait complete. 2024-04-22T01:16:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 Processor -- start 2024-04-22T01:16:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- start start 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340039c0 0x7f6c480c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 0x7f6c480c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c48159d20 0x7f6c4815a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4806b780 con 0x7f6c4806ec50 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4806b600 con 0x7f6c340039c0 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6c480cedb0 con 0x7f6c48159d20 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 0x7f6c480c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 0x7f6c480c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47538/0 (socket says 172.21.15.70:47538) 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 -- 172.21.15.70:0/1356261684 learned_addr learned my addr 172.21.15.70:0/1356261684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340039c0 0x7f6c480c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4e9aa640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c48159d20 0x7f6c4815a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 -- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c48159d20 msgr2=0x7f6c4815a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c48159d20 0x7f6c4815a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 -- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340039c0 msgr2=0x7f6c480c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340039c0 0x7f6c480c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 -- 172.21.15.70:0/1356261684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4815a6e0 con 0x7f6c4806ec50 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340039c0 0x7f6c480c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4e9aa640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c48159d20 0x7f6c4815a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c4d9a8640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 0x7f6c480c8a00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6c44007fd0 tx=0x7f6c4400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c44019070 con 0x7f6c4806ec50 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c440072d0 con 0x7f6c4806ec50 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c440047f0 con 0x7f6c4806ec50 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4806f4a0 con 0x7f6c4806ec50 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c4806fa60 con 0x7f6c4806ec50 2024-04-22T01:16:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.895+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c48068920 con 0x7f6c4806ec50 2024-04-22T01:16:26.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.899+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c44004990 con 0x7f6c4806ec50 2024-04-22T01:16:26.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.899+0000 7f6c3effd640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c2c0678f0 0x7f6c2c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:26.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.899+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f6c440855b0 con 0x7f6c4806ec50 2024-04-22T01:16:26.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.899+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c2c0678f0 0x7f6c2c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:26.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.899+0000 7f6c4e1a9640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c2c0678f0 0x7f6c2c069db0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6c380af000 tx=0x7f6c38008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:26.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:26.903+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6c4815a6e0 con 0x7f6c4806ec50 2024-04-22T01:16:27.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.051+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c480cd520 con 0x7f6c4806ec50 2024-04-22T01:16:27.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c3effd640 1 -- 172.21.15.70:0/1356261684 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16906 (secure 0 0 0) 0x7f6c4815fd10 con 0x7f6c4806ec50 2024-04-22T01:16:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:27.055 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:12.187805+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c2c0678f0 msgr2=0x7f6c2c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c2c0678f0 0x7f6c2c069db0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6c380af000 tx=0x7f6c38008040 comp rx=0 tx=0).stop 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 msgr2=0x7f6c480c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 0x7f6c480c8a00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6c44007fd0 tx=0x7f6c4400dc00 comp rx=0 tx=0).stop 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 shutdown_connections 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c2c0678f0 0x7f6c2c069db0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c48159d20 0x7f6c4815a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c4806ec50 0x7f6c480c8a00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 --2- 172.21.15.70:0/1356261684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c340039c0 0x7f6c480c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 >> 172.21.15.70:0/1356261684 conn(0x7f6c48099a60 msgr2=0x7f6c4815af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 shutdown_connections 2024-04-22T01:16:27.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.055+0000 7f6c5040d640 1 -- 172.21.15.70:0/1356261684 wait complete. 2024-04-22T01:16:27.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:16:08.953522+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:16:09.533622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:27.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:27.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.183+0000 7fd66651d640 1 Processor -- start 2024-04-22T01:16:27.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.183+0000 7fd66651d640 1 -- start start 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd66651d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd660158650 0x7fd66015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd66651d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd66006d8f0 0x7fd66006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd66651d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd66006de60 con 0x7fd66006f440 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd66651d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd66006e140 con 0x7fd66006d8f0 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd66651d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd66006e420 con 0x7fd660158650 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd66006d8f0 0x7fd66006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd66006d8f0 0x7fd66006ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47552/0 (socket says 172.21.15.70:47552) 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 -- 172.21.15.70:0/4188217772 learned_addr learned my addr 172.21.15.70:0/4188217772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:27.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd65ffff640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd660158650 0x7fd66015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 -- 172.21.15.70:0/4188217772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd660158650 msgr2=0x7fd66015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:27.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd660158650 0x7fd66015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 -- 172.21.15.70:0/4188217772 >> v1:172.21.15.98:6789/0 conn(0x7fd66006f440 legacy=0x7fd66015f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 -- 172.21.15.70:0/4188217772 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66006e700 con 0x7fd66006d8f0 2024-04-22T01:16:27.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd65ffff640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd660158650 0x7fd66015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:27.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd6577fe640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66006d8f0 0x7fd66006ef00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd648009870 tx=0x7fd64801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f623cab9799cff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd65f7fe640 1 -- 172.21.15.70:0/4188217772 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6480b0070 con 0x7fd66006d8f0 2024-04-22T01:16:27.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd65f7fe640 1 -- 172.21.15.70:0/4188217772 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6480ab600 con 0x7fd66006d8f0 2024-04-22T01:16:27.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.187+0000 7fd65f7fe640 1 -- 172.21.15.70:0/4188217772 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6480ab9d0 con 0x7fd66006d8f0 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/4188217772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66006d8f0 msgr2=0x7fd66006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66006d8f0 0x7fd66006ef00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd648009870 tx=0x7fd64801c4d0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/4188217772 shutdown_connections 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd66006d8f0 0x7fd66006ef00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 --2- 172.21.15.70:0/4188217772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd660158650 0x7fd66015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/4188217772 >> 172.21.15.70:0/4188217772 conn(0x7fd6600b1d40 msgr2=0x7fd6600cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/4188217772 shutdown_connections 2024-04-22T01:16:27.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/4188217772 wait complete. 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 Processor -- start 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- start start 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66006d8f0 0x7fd66014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66006f440 0x7fd6601563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 0x7fd6600b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd66016ac50 con 0x7fd660158650 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd66015fee0 con 0x7fd66006f440 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd66016add0 con 0x7fd66006d8f0 2024-04-22T01:16:27.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 0x7fd6600b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 0x7fd6600b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47554/0 (socket says 172.21.15.70:47554) 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 -- 172.21.15.70:0/2078249528 learned_addr learned my addr 172.21.15.70:0/2078249528 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd65ffff640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66006d8f0 0x7fd66014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd6577fe640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66006f440 0x7fd6601563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 -- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66006d8f0 msgr2=0x7fd66014f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66006d8f0 0x7fd66014f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 -- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66006f440 msgr2=0x7fd6601563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66006f440 0x7fd6601563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 -- 172.21.15.70:0/2078249528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66014fc90 con 0x7fd660158650 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd6577fe640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66006f440 0x7fd6601563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd664aba640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 0x7fd6600b3170 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c770 tx=0x7fd65800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658003b80 con 0x7fd660158650 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd658003d20 con 0x7fd660158650 2024-04-22T01:16:27.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd660156be0 con 0x7fd660158650 2024-04-22T01:16:27.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800aa30 con 0x7fd660158650 2024-04-22T01:16:27.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.191+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6601571a0 con 0x7fd660158650 2024-04-22T01:16:27.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.195+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd660068210 con 0x7fd660158650 2024-04-22T01:16:27.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.195+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd65800abd0 con 0x7fd660158650 2024-04-22T01:16:27.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.195+0000 7fd65d7fa640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6400678c0 0x7fd640069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.195+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd658085d30 con 0x7fd660158650 2024-04-22T01:16:27.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.195+0000 7fd65ffff640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6400678c0 0x7fd640069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.195+0000 7fd65ffff640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6400678c0 0x7fd640069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c900 tx=0x7fd650008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.199+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd6580566e0 con 0x7fd660158650 2024-04-22T01:16:27.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd66015fab0 con 0x7fd660158650 2024-04-22T01:16:27.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd65d7fa640 1 -- 172.21.15.70:0/2078249528 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7fd658055fe0 con 0x7fd660158650 2024-04-22T01:16:27.351 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6400678c0 msgr2=0x7fd640069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6400678c0 0x7fd640069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c900 tx=0x7fd650008040 comp rx=0 tx=0).stop 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 msgr2=0x7fd6600b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 0x7fd6600b3170 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c770 tx=0x7fd65800cc30 comp rx=0 tx=0).stop 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 shutdown_connections 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6400678c0 0x7fd640069d80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd660158650 0x7fd6600b3170 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66006f440 0x7fd6601563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 --2- 172.21.15.70:0/2078249528 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd66006d8f0 0x7fd66014f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 >> 172.21.15.70:0/2078249528 conn(0x7fd6600b1d40 msgr2=0x7fd660160b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 shutdown_connections 2024-04-22T01:16:27.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.351+0000 7fd66651d640 1 -- 172.21.15.70:0/2078249528 wait complete. 2024-04-22T01:16:27.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:27.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb71f3a640 1 Processor -- start 2024-04-22T01:16:27.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb71f3a640 1 -- start start 2024-04-22T01:16:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb71f3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 0x7fcb6c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb71f3a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6c0c5720 con 0x7fcb6c15f3e0 2024-04-22T01:16:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb71f3a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6c0cd710 con 0x7fcb6c0cd280 2024-04-22T01:16:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb71f3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6c0cd9f0 con 0x7fcb6c0b43e0 2024-04-22T01:16:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb6b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcb6c15f3e0 0x7fcb6c099d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33962/0 (socket says 172.21.15.70:33962) 2024-04-22T01:16:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.487+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/3800586327 learned_addr learned my addr 172.21.15.70:0/3800586327 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6bfff640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 0x7fcb6c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603109293 0 0) 0x7fcb6c0c5720 con 0x7fcb6c15f3e0 2024-04-22T01:16:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb58003530 con 0x7fcb6c15f3e0 2024-04-22T01:16:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2019569423 0 0) 0x7fcb6c0cd710 con 0x7fcb6c0cd280 2024-04-22T01:16:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb6c0c5720 con 0x7fcb6c0cd280 2024-04-22T01:16:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 293099932 0 0) 0x7fcb58003530 con 0x7fcb6c15f3e0 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 msgr2=0x7fcb6c0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 0x7fcb6c0cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 >> v1:172.21.15.98:6789/0 conn(0x7fcb6c0cd280 legacy=0x7fcb6c0cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c0cdcd0 con 0x7fcb6c15f3e0 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fcb5c0016a0 con 0x7fcb6c15f3e0 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 >> v1:172.21.15.70:6789/0 conn(0x7fcb6c15f3e0 legacy=0x7fcb6c099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb58003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 0x7fcb58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb58004d50 0x7fcb580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c0cd9f0 con 0x7fcb58003b80 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5800bab0 con 0x7fcb58004d50 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5800bcb0 con 0x7fcb6c0b43e0 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb58003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6affd640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 0x7fcb58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6bfff640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb58004d50 0x7fcb580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 msgr2=0x7fcb58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c0b43e0 0x7fcb58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb58004d50 msgr2=0x7fcb580051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb58004d50 0x7fcb580051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/3800586327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb58003530 con 0x7fcb58003b80 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb58003f80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c160 tx=0x7fcb5c01cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9e291d3ec3c420 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c007c50 con 0x7fcb58003b80 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0a9070 con 0x7fcb58003b80 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6a7fc640 1 -- 172.21.15.70:0/3800586327 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c005d20 con 0x7fcb58003b80 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb71f3a640 1 -- 172.21.15.70:0/3800586327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 msgr2=0x7fcb58003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/3800586327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb58003f80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c160 tx=0x7fcb5c01cb90 comp rx=0 tx=0).stop 2024-04-22T01:16:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb6bfff640 1 -- 172.21.15.70:0/3800586327 reap_dead start 2024-04-22T01:16:27.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb71f3a640 1 -- 172.21.15.70:0/3800586327 shutdown_connections 2024-04-22T01:16:27.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb71f3a640 1 -- 172.21.15.70:0/3800586327 >> 172.21.15.70:0/3800586327 conn(0x7fcb6c0b36c0 msgr2=0x7fcb6c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb71f3a640 1 -- 172.21.15.70:0/3800586327 shutdown_connections 2024-04-22T01:16:27.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.491+0000 7fcb71f3a640 1 -- 172.21.15.70:0/3800586327 wait complete. 2024-04-22T01:16:27.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 Processor -- start 2024-04-22T01:16:27.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- start start 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb6c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c154800 0x7fcb6c15c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c154c80 0x7fcb6c15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c0c6c30 con 0x7fcb58003b80 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c09a690 con 0x7fcb6c154c80 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6c0c6db0 con 0x7fcb6c154800 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb6c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb6c1509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47574/0 (socket says 172.21.15.70:47574) 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/940746502 learned_addr learned my addr 172.21.15.70:0/940746502 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6bfff640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c154c80 0x7fcb6c15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6affd640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c154800 0x7fcb6c15c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c154800 msgr2=0x7fcb6c15c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c154800 0x7fcb6c15c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c154c80 msgr2=0x7fcb6c15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c154c80 0x7fcb6c15cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 -- 172.21.15.70:0/940746502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c1577c0 con 0x7fcb58003b80 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6bfff640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c154c80 0x7fcb6c15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb6b7fe640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb6c1509a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c098520 tx=0x7fcb58005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c005d00 con 0x7fcb58003b80 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0014a0 con 0x7fcb58003b80 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c157aa0 con 0x7fcb58003b80 2024-04-22T01:16:27.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0a5a40 con 0x7fcb58003b80 2024-04-22T01:16:27.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c158060 con 0x7fcb58003b80 2024-04-22T01:16:27.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.495+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c068920 con 0x7fcb58003b80 2024-04-22T01:16:27.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.499+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcb5c0a5be0 con 0x7fcb58003b80 2024-04-22T01:16:27.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.499+0000 7fcb63fff640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb4c0678c0 0x7fcb4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.499+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fcb5c11f6b0 con 0x7fcb58003b80 2024-04-22T01:16:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.499+0000 7fcb6affd640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb4c0678c0 0x7fcb4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.499+0000 7fcb6affd640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb4c0678c0 0x7fcb4c069d80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06e4e0 tx=0x7fcb54009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.503+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcb5c0168d0 con 0x7fcb58003b80 2024-04-22T01:16:27.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.659+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcb6c0b2bc0 con 0x7fcb58003b80 2024-04-22T01:16:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb63fff640 1 -- 172.21.15.70:0/940746502 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7fcb5c0f0070 con 0x7fcb58003b80 2024-04-22T01:16:27.661 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb4c0678c0 msgr2=0x7fcb4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb4c0678c0 0x7fcb4c069d80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06e4e0 tx=0x7fcb54009040 comp rx=0 tx=0).stop 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 msgr2=0x7fcb6c1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb6c1509a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c098520 tx=0x7fcb58005840 comp rx=0 tx=0).stop 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 shutdown_connections 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcb4c0678c0 0x7fcb4c069d80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c154c80 0x7fcb6c15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb6c154800 0x7fcb6c15c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 --2- 172.21.15.70:0/940746502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb58003b80 0x7fcb6c1509a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 >> 172.21.15.70:0/940746502 conn(0x7fcb6c0b36c0 msgr2=0x7fcb6c0c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 shutdown_connections 2024-04-22T01:16:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.663+0000 7fcb71f3a640 1 -- 172.21.15.70:0/940746502 wait complete. 2024-04-22T01:16:27.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:27.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 Processor -- start 2024-04-22T01:16:27.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 -- start start 2024-04-22T01:16:27.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280b4450 0x7f6728071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67280720b0 0x7f67280ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672815c290 con 0x7f67280cd050 2024-04-22T01:16:27.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6728072510 con 0x7f67280720b0 2024-04-22T01:16:27.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672ebd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f67280b3a50 con 0x7f67280b4450 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67280720b0 0x7f67280ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67280720b0 0x7f67280ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47580/0 (socket says 172.21.15.70:47580) 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 -- 172.21.15.70:0/3131365697 learned_addr learned my addr 172.21.15.70:0/3131365697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672d16d640 1 --1- 172.21.15.70:0/3131365697 >> v1:172.21.15.70:6789/0 conn(0x7f67280cd050 0x7f672815f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33970/0 (socket says 172.21.15.70:33970) 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672c96c640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280b4450 0x7f6728071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 -- 172.21.15.70:0/3131365697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280b4450 msgr2=0x7f6728071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280b4450 0x7f6728071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 -- 172.21.15.70:0/3131365697 >> v1:172.21.15.70:6789/0 conn(0x7f67280cd050 legacy=0x7f672815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f67277fe640 1 -- 172.21.15.70:0/3131365697 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000998625 0 0) 0x7f672815c290 con 0x7f67280cd050 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 -- 172.21.15.70:0/3131365697 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672815c570 con 0x7f67280720b0 2024-04-22T01:16:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f672c96c640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280b4450 0x7f6728071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.803+0000 7f6727fff640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280720b0 0x7f67280ccb10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6710009870 tx=0x7f671001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e05a264afbb110e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f67277fe640 1 -- 172.21.15.70:0/3131365697 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100ab650 con 0x7f67280720b0 2024-04-22T01:16:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f67277fe640 1 -- 172.21.15.70:0/3131365697 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67100b0070 con 0x7f67280720b0 2024-04-22T01:16:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f67277fe640 1 -- 172.21.15.70:0/3131365697 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100aba10 con 0x7f67280720b0 2024-04-22T01:16:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- 172.21.15.70:0/3131365697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280720b0 msgr2=0x7f67280ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280720b0 0x7f67280ccb10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6710009870 tx=0x7f671001c530 comp rx=0 tx=0).stop 2024-04-22T01:16:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- 172.21.15.70:0/3131365697 shutdown_connections 2024-04-22T01:16:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280720b0 0x7f67280ccb10 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 --2- 172.21.15.70:0/3131365697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280b4450 0x7f6728071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- 172.21.15.70:0/3131365697 >> 172.21.15.70:0/3131365697 conn(0x7f67280b1d40 msgr2=0x7f67280cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- 172.21.15.70:0/3131365697 shutdown_connections 2024-04-22T01:16:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- 172.21.15.70:0/3131365697 wait complete. 2024-04-22T01:16:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 Processor -- start 2024-04-22T01:16:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- start start 2024-04-22T01:16:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67280720b0 0x7f672814a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 0x7f6728148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280cd050 0x7f6728154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67280b4830 con 0x7f67280b4450 2024-04-22T01:16:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67280cd480 con 0x7f67280720b0 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672ebd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f67280b49b0 con 0x7f67280cd050 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 0x7f6728148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 0x7f6728148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47582/0 (socket says 172.21.15.70:47582) 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 -- 172.21.15.70:0/4039307538 learned_addr learned my addr 172.21.15.70:0/4039307538 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672d16d640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280cd050 0x7f6728154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672c96c640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67280720b0 0x7f672814a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 -- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280cd050 msgr2=0x7f6728154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280cd050 0x7f6728154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 -- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67280720b0 msgr2=0x7f672814a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67280720b0 0x7f672814a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 -- 172.21.15.70:0/4039307538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67280b3c30 con 0x7f67280b4450 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f672c96c640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67280720b0 0x7f672814a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f6727fff640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 0x7f6728148950 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f671001a160 tx=0x7f671001ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100ba070 con 0x7f67280b4450 2024-04-22T01:16:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67100abc40 con 0x7f67280b4450 2024-04-22T01:16:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.807+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710004020 con 0x7f67280b4450 2024-04-22T01:16:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672815e250 con 0x7f67280b4450 2024-04-22T01:16:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672815e810 con 0x7f67280b4450 2024-04-22T01:16:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6728068210 con 0x7f67280b4450 2024-04-22T01:16:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f67100041c0 con 0x7f67280b4450 2024-04-22T01:16:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f67257fa640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67080678c0 0x7f6708069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6130+0+0 (secure 0 0 0) 0x7f67101201d0 con 0x7f67280b4450 2024-04-22T01:16:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f672c96c640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67080678c0 0x7f6708069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.811+0000 7f672c96c640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67080678c0 0x7f6708069d80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f67281f7fe0 tx=0x7f6718008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.815+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6710016150 con 0x7f67280b4450 2024-04-22T01:16:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.963+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6728071e90 con 0x7f67280b4450 2024-04-22T01:16:27.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.963+0000 7f67257fa640 1 -- 172.21.15.70:0/4039307538 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7f67100044b0 con 0x7f67280b4450 2024-04-22T01:16:27.965 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67080678c0 msgr2=0x7f6708069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67080678c0 0x7f6708069d80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f67281f7fe0 tx=0x7f6718008040 comp rx=0 tx=0).stop 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 msgr2=0x7f6728148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 0x7f6728148950 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f671001a160 tx=0x7f671001ad70 comp rx=0 tx=0).stop 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 shutdown_connections 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f67080678c0 0x7f6708069d80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67280cd050 0x7f6728154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67280b4450 0x7f6728148950 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 --2- 172.21.15.70:0/4039307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67280720b0 0x7f672814a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 >> 172.21.15.70:0/4039307538 conn(0x7f67280b1d40 msgr2=0x7f67281526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 shutdown_connections 2024-04-22T01:16:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:27.967+0000 7f672ebd0640 1 -- 172.21.15.70:0/4039307538 wait complete. 2024-04-22T01:16:27.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:28.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.087+0000 7fda1c7f8640 1 Processor -- start 2024-04-22T01:16:28.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.087+0000 7fda1c7f8640 1 -- start start 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1c7f8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda141455b0 0x7fda140cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1c7f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda1406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1c7f8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda140d0250 con 0x7fda140cc150 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1c7f8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fda140cd7b0 con 0x7fda141455b0 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1c7f8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fda140cc710 con 0x7fda14072230 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1a594640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fda141455b0 0x7fda140cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1ad95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda1406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1ad95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda1406e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:37430/0 (socket says 172.21.15.70:37430) 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda19d93640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fda140cc150 0x7fda14071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:41844/0 (socket says 172.21.15.70:41844) 2024-04-22T01:16:28.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1ad95640 1 -- 172.21.15.70:0/306977571 learned_addr learned my addr 172.21.15.70:0/306977571 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:28.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1a594640 1 -- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 msgr2=0x7fda1406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:28.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1a594640 1 --2- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda1406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1a594640 1 -- 172.21.15.70:0/306977571 >> v1:172.21.15.98:6789/0 conn(0x7fda140cc150 legacy=0x7fda14071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1a594640 1 -- 172.21.15.70:0/306977571 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda140cc9f0 con 0x7fda141455b0 2024-04-22T01:16:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.091+0000 7fda1a594640 1 --2- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140cfd10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda100097c0 tx=0x7fda1001c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e2dc026c29e9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19592640 1 -- 172.21.15.70:0/306977571 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100b0070 con 0x7fda141455b0 2024-04-22T01:16:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19592640 1 -- 172.21.15.70:0/306977571 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda100ab600 con 0x7fda141455b0 2024-04-22T01:16:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19592640 1 -- 172.21.15.70:0/306977571 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100ab9d0 con 0x7fda141455b0 2024-04-22T01:16:28.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 msgr2=0x7fda140cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140cfd10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda100097c0 tx=0x7fda1001c380 comp rx=0 tx=0).stop 2024-04-22T01:16:28.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/306977571 shutdown_connections 2024-04-22T01:16:28.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda1406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/306977571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140cfd10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/306977571 >> 172.21.15.70:0/306977571 conn(0x7fda1406b1f0 msgr2=0x7fda1406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/306977571 shutdown_connections 2024-04-22T01:16:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/306977571 wait complete. 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 Processor -- start 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- start start 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda140b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 0x7fda1415c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fda1406f320 con 0x7fda140cc150 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda1406f1a0 con 0x7fda141455b0 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fda14168b40 con 0x7fda14072230 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 0x7fda1415c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 0x7fda1415c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47586/0 (socket says 172.21.15.70:47586) 2024-04-22T01:16:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 -- 172.21.15.70:0/1999077599 learned_addr learned my addr 172.21.15.70:0/1999077599 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1a594640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda140b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1ad95640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 -- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 msgr2=0x7fda140b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda140b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 -- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 msgr2=0x7fda140ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 -- 172.21.15.70:0/1999077599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda140ade20 con 0x7fda140cc150 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda19d93640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 0x7fda1415c840 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c3a0 tx=0x7fda0400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda040074e0 con 0x7fda140cc150 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda040076a0 con 0x7fda140cc150 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04004a30 con 0x7fda140cc150 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda140ae100 con 0x7fda140cc150 2024-04-22T01:16:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.095+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1415e850 con 0x7fda140cc150 2024-04-22T01:16:28.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.099+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda14068210 con 0x7fda140cc150 2024-04-22T01:16:28.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.099+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fda0401a020 con 0x7fda140cc150 2024-04-22T01:16:28.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.099+0000 7fda0affd640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd9f80678c0 0x7fd9f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.099+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6130+0+0 (secure 0 0 0) 0x7fda04085510 con 0x7fda140cc150 2024-04-22T01:16:28.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.099+0000 7fda1a594640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd9f80678c0 0x7fd9f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.099+0000 7fda1a594640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd9f80678c0 0x7fd9f8069d80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fda100ad000 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.103+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fda04055f30 con 0x7fda140cc150 2024-04-22T01:16:28.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda1415bc10 con 0x7fda140cc150 2024-04-22T01:16:28.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda0affd640 1 -- 172.21.15.70:0/1999077599 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5178 (secure 0 0 0) 0x7fda04055830 con 0x7fda140cc150 2024-04-22T01:16:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:28.283 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:10.524071+0000","modified":"2024-04-22T01:16:13.536979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5478":{"gid":5478,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3248824947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3248824947},{"type":"v1","addr":"172.21.15.70:6837","nonce":3248824947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5478,"qdb_cluster":[5478]},"id":2}]} 2024-04-22T01:16:28.283 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd9f80678c0 msgr2=0x7fd9f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd9f80678c0 0x7fd9f8069d80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fda100ad000 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 msgr2=0x7fda1415c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 0x7fda1415c840 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c3a0 tx=0x7fda0400c860 comp rx=0 tx=0).stop 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 shutdown_connections 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd9f80678c0 0x7fd9f8069d80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda141455b0 0x7fda140ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda140cc150 0x7fda1415c840 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 --2- 172.21.15.70:0/1999077599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda14072230 0x7fda140b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 >> 172.21.15.70:0/1999077599 conn(0x7fda1406b1f0 msgr2=0x7fda1415af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:28.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 shutdown_connections 2024-04-22T01:16:28.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.283+0000 7fda1c7f8640 1 -- 172.21.15.70:0/1999077599 wait complete. 2024-04-22T01:16:28.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:28.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 Processor -- start 2024-04-22T01:16:28.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 -- start start 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec0cc150 0x7fe9ec071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe9ec072060 0x7fe9ec06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9ec0cd7b0 con 0x7fe9ec1455b0 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe9ec0cc710 con 0x7fe9ec072060 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.403+0000 7fe9f08f2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ec0cc9f0 con 0x7fe9ec0cc150 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe9ec072060 0x7fe9ec06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe9ec072060 0x7fe9ec06e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47598/0 (socket says 172.21.15.70:47598) 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 -- 172.21.15.70:0/553602126 learned_addr learned my addr 172.21.15.70:0/553602126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9e9d9b640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec0cc150 0x7fe9ec071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 -- 172.21.15.70:0/553602126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec0cc150 msgr2=0x7fe9ec071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:28.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec0cc150 0x7fe9ec071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 -- 172.21.15.70:0/553602126 >> v1:172.21.15.98:6789/0 conn(0x7fe9ec1455b0 legacy=0x7fe9ec0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 -- 172.21.15.70:0/553602126 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec0cccd0 con 0x7fe9ec072060 2024-04-22T01:16:28.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9e9d9b640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec0cc150 0x7fe9ec071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:28.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9ead9d640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec072060 0x7fe9ec06e200 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009870 tx=0x7fe9dc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df00e84ced26027e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9e959a640 1 -- 172.21.15.70:0/553602126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc0b0070 con 0x7fe9ec072060 2024-04-22T01:16:28.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9e959a640 1 -- 172.21.15.70:0/553602126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc0ab600 con 0x7fe9ec072060 2024-04-22T01:16:28.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9e959a640 1 -- 172.21.15.70:0/553602126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc0ab9d0 con 0x7fe9ec072060 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 -- 172.21.15.70:0/553602126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec072060 msgr2=0x7fe9ec06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec072060 0x7fe9ec06e200 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009870 tx=0x7fe9dc01c440 comp rx=0 tx=0).stop 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 -- 172.21.15.70:0/553602126 shutdown_connections 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec072060 0x7fe9ec06e200 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/553602126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec0cc150 0x7fe9ec071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 -- 172.21.15.70:0/553602126 >> 172.21.15.70:0/553602126 conn(0x7fe9ec06b1f0 msgr2=0x7fe9ec06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 -- 172.21.15.70:0/553602126 shutdown_connections 2024-04-22T01:16:28.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.407+0000 7fe9f08f2640 1 -- 172.21.15.70:0/553602126 wait complete. 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 Processor -- start 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- start start 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec072060 0x7fe9ec13a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9ec0cc150 0x7fe9ec15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 0x7fe9ec158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec06f2e0 con 0x7fe9ec1455b0 2024-04-22T01:16:28.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec06f160 con 0x7fe9ec0cc150 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ec168b40 con 0x7fe9ec072060 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 0x7fe9ec158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 0x7fe9ec158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47608/0 (socket says 172.21.15.70:47608) 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 -- 172.21.15.70:0/650719895 learned_addr learned my addr 172.21.15.70:0/650719895 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ea59c640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec072060 0x7fe9ec13a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 -- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec072060 msgr2=0x7fe9ec13a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec072060 0x7fe9ec13a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 -- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9ec0cc150 msgr2=0x7fe9ec15b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9ec0cc150 0x7fe9ec15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 -- 172.21.15.70:0/650719895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec13b0f0 con 0x7fe9ec1455b0 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ead9d640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 0x7fe9ec158860 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0ae040 tx=0x7fe9dc0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc0b0040 con 0x7fe9ec1455b0 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc0b4610 con 0x7fe9ec1455b0 2024-04-22T01:16:28.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ec13b3d0 con 0x7fe9ec1455b0 2024-04-22T01:16:28.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc0b49c0 con 0x7fe9ec1455b0 2024-04-22T01:16:28.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ec13b990 con 0x7fe9ec1455b0 2024-04-22T01:16:28.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ec068210 con 0x7fe9ec1455b0 2024-04-22T01:16:28.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe9dc0b4b60 con 0x7fe9ec1455b0 2024-04-22T01:16:28.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9daffd640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9cc0678f0 0x7fe9cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9ea59c640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9cc0678f0 0x7fe9cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.411+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6130+0+0 (secure 0 0 0) 0x7fe9dc120000 con 0x7fe9ec1455b0 2024-04-22T01:16:28.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.415+0000 7fe9ea59c640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9cc0678f0 0x7fe9cc069db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec140270 tx=0x7fe9e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.415+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe9dc016150 con 0x7fe9ec1455b0 2024-04-22T01:16:28.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.567+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9ec15f510 con 0x7fe9ec1455b0 2024-04-22T01:16:28.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.567+0000 7fe9daffd640 1 -- 172.21.15.70:0/650719895 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16899 (secure 0 0 0) 0x7fe9dc0f0990 con 0x7fe9ec1455b0 2024-04-22T01:16:28.567 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:28.568 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:27.691380+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.19","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9cc0678f0 msgr2=0x7fe9cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9cc0678f0 0x7fe9cc069db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec140270 tx=0x7fe9e4009040 comp rx=0 tx=0).stop 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 msgr2=0x7fe9ec158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 0x7fe9ec158860 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0ae040 tx=0x7fe9dc0191e0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 shutdown_connections 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9cc0678f0 0x7fe9cc069db0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9ec1455b0 0x7fe9ec158860 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9ec0cc150 0x7fe9ec15b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 --2- 172.21.15.70:0/650719895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9ec072060 0x7fe9ec13a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 >> 172.21.15.70:0/650719895 conn(0x7fe9ec06b1f0 msgr2=0x7fe9ec0744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 shutdown_connections 2024-04-22T01:16:28.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.571+0000 7fe9f08f2640 1 -- 172.21.15.70:0/650719895 wait complete. 2024-04-22T01:16:28.585 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:16:28.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.691+0000 7f67096be640 1 Processor -- start 2024-04-22T01:16:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.691+0000 7f67096be640 1 -- start start 2024-04-22T01:16:28.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67096be640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f67040d0f00 0x7f670415e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67096be640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67040ce730 0x7f67040ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67096be640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67040cebc0 con 0x7f670415ac40 2024-04-22T01:16:28.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67096be640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f67040ad9b0 con 0x7f67040ce730 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67096be640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f670415eec0 con 0x7f67040d0f00 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67040ce730 0x7f67040ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67040ce730 0x7f67040ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47612/0 (socket says 172.21.15.70:47612) 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 -- 172.21.15.70:0/4031491650 learned_addr learned my addr 172.21.15.70:0/4031491650 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67027fc640 1 --2- 172.21.15.70:0/4031491650 >> v2:172.21.15.98:3300/0 conn(0x7f67040d0f00 0x7f670415e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 -- 172.21.15.70:0/4031491650 >> v2:172.21.15.98:3300/0 conn(0x7f67040d0f00 msgr2=0x7f670415e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 --2- 172.21.15.70:0/4031491650 >> v2:172.21.15.98:3300/0 conn(0x7f67040d0f00 0x7f670415e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 -- 172.21.15.70:0/4031491650 >> v1:172.21.15.98:6789/0 conn(0x7f670415ac40 legacy=0x7f67040d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.695+0000 7f67037fe640 1 -- 172.21.15.70:0/4031491650 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670415f180 con 0x7f67040ce730 2024-04-22T01:16:28.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67037fe640 1 --2- 172.21.15.70:0/4031491650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f67040ade40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f67040b3560 tx=0x7f66f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8380fc93848acf92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f6701ffb640 1 -- 172.21.15.70:0/4031491650 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40b0070 con 0x7f67040ce730 2024-04-22T01:16:28.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f6701ffb640 1 -- 172.21.15.70:0/4031491650 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f40ab600 con 0x7f67040ce730 2024-04-22T01:16:28.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f6701ffb640 1 -- 172.21.15.70:0/4031491650 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40ab9d0 con 0x7f67040ce730 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- 172.21.15.70:0/4031491650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 msgr2=0x7f67040ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 --2- 172.21.15.70:0/4031491650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f67040ade40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f67040b3560 tx=0x7f66f401c520 comp rx=0 tx=0).stop 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- 172.21.15.70:0/4031491650 shutdown_connections 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 --2- 172.21.15.70:0/4031491650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f67040ade40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 --2- 172.21.15.70:0/4031491650 >> v2:172.21.15.98:3300/0 conn(0x7f67040d0f00 0x7f670415e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- 172.21.15.70:0/4031491650 >> 172.21.15.70:0/4031491650 conn(0x7f6704099a60 msgr2=0x7f6704099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- 172.21.15.70:0/4031491650 shutdown_connections 2024-04-22T01:16:28.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- 172.21.15.70:0/4031491650 wait complete. 2024-04-22T01:16:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 Processor -- start 2024-04-22T01:16:28.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- start start 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f6704154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67040d0f00 0x7f6704167500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f670415ac40 0x7f670406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67040aef20 con 0x7f67040ce730 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67040aeda0 con 0x7f67040d0f00 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67096be640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f67040af0a0 con 0x7f670415ac40 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f6704154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f6702ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f6704154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47620/0 (socket says 172.21.15.70:47620) 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f6702ffd640 1 -- 172.21.15.70:0/4198745749 learned_addr learned my addr 172.21.15.70:0/4198745749 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:28.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.699+0000 7f67037fe640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f670415ac40 0x7f670406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f6702ffd640 1 -- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f670415ac40 msgr2=0x7f670406efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f6702ffd640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f670415ac40 0x7f670406efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f67027fc640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67040d0f00 0x7f6704167500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f6702ffd640 1 -- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67040d0f00 msgr2=0x7f6704167500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f6702ffd640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67040d0f00 0x7f6704167500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f6702ffd640 1 -- 172.21.15.70:0/4198745749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6704167a40 con 0x7f67040ce730 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f6702ffd640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f6704154da0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f66fc00c7f0 tx=0x7f66fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc003b80 con 0x7f67040ce730 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66fc003d20 con 0x7f67040ce730 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00aa90 con 0x7f67040ce730 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6704167ce0 con 0x7f67040ce730 2024-04-22T01:16:28.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67040af760 con 0x7f67040ce730 2024-04-22T01:16:28.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6704068920 con 0x7f67040ce730 2024-04-22T01:16:28.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f66fc00ac30 con 0x7f67040ce730 2024-04-22T01:16:28.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f66fb7fe640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f66e40678f0 0x7f66e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:28.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f67027fc640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f66e40678f0 0x7f66e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:28.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f66fc085de0 con 0x7f67040ce730 2024-04-22T01:16:28.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.703+0000 7f67027fc640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f66e40678f0 0x7f66e4069db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f67040b6820 tx=0x7f66ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:28.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.711+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f66fc056750 con 0x7f67040ce730 2024-04-22T01:16:28.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.899+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f670406ed90 con 0x7f67040ce730 2024-04-22T01:16:28.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.899+0000 7f66fb7fe640 1 -- 172.21.15.70:0/4198745749 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5178 (secure 0 0 0) 0x7f66fc056050 con 0x7f67040ce730 2024-04-22T01:16:28.900 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:28.900 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:10.524071+0000","modified":"2024-04-22T01:16:13.536979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5478":{"gid":5478,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3248824947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3248824947},{"type":"v1","addr":"172.21.15.70:6837","nonce":3248824947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5478,"qdb_cluster":[5478]},"id":2}]} 2024-04-22T01:16:28.900 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f66e40678f0 msgr2=0x7f66e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f66e40678f0 0x7f66e4069db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f67040b6820 tx=0x7f66ec008040 comp rx=0 tx=0).stop 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 msgr2=0x7f6704154da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f6704154da0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f66fc00c7f0 tx=0x7f66fc00ccb0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 shutdown_connections 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f66e40678f0 0x7f66e4069db0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f670415ac40 0x7f670406efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67040d0f00 0x7f6704167500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 --2- 172.21.15.70:0/4198745749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67040ce730 0x7f6704154da0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 >> 172.21.15.70:0/4198745749 conn(0x7f6704099a60 msgr2=0x7f67040cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 shutdown_connections 2024-04-22T01:16:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:28.903+0000 7f67096be640 1 -- 172.21.15.70:0/4198745749 wait complete. 2024-04-22T01:16:28.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:29.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 Processor -- start 2024-04-22T01:16:29.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 -- start start 2024-04-22T01:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00e415bc70 0x7f00e415f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e415fd10 con 0x7f00e406b730 2024-04-22T01:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f00e406bb80 con 0x7f00e415bc70 2024-04-22T01:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e93a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f00e40cfd10 con 0x7f00e40cd740 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00e415bc70 0x7f00e415f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f00e406b730 0x7f00e40ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:41876/0 (socket says 172.21.15.70:41876) 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00e415bc70 0x7f00e415f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:54882/0 (socket says 172.21.15.70:54882) 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e37fe640 1 -- 172.21.15.70:0/3112073337 learned_addr learned my addr 172.21.15.70:0/3112073337 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e2ffd640 1 -- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 msgr2=0x7f00e406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e2ffd640 1 --2- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e406b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e2ffd640 1 -- 172.21.15.70:0/3112073337 >> v1:172.21.15.98:6789/0 conn(0x7f00e406b730 legacy=0x7f00e40ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e2ffd640 1 -- 172.21.15.70:0/3112073337 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e415c050 con 0x7f00e415bc70 2024-04-22T01:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.031+0000 7f00e27fc640 1 --2- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e406b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e2ffd640 1 --2- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e415bc70 0x7f00e415f7d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0097c0 tx=0x7f00dc01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef00f8c495acc18 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e1ffb640 1 -- 172.21.15.70:0/3112073337 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0b2070 con 0x7f00e415bc70 2024-04-22T01:16:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e1ffb640 1 -- 172.21.15.70:0/3112073337 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00dc0ad680 con 0x7f00e415bc70 2024-04-22T01:16:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e1ffb640 1 -- 172.21.15.70:0/3112073337 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0ada50 con 0x7f00e415bc70 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 -- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e415bc70 msgr2=0x7f00e415f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 --2- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e415bc70 0x7f00e415f7d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0097c0 tx=0x7f00dc01c470 comp rx=0 tx=0).stop 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 -- 172.21.15.70:0/3112073337 shutdown_connections 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 --2- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e406b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 --2- 172.21.15.70:0/3112073337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e415bc70 0x7f00e415f7d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 -- 172.21.15.70:0/3112073337 >> 172.21.15.70:0/3112073337 conn(0x7f00e4099a60 msgr2=0x7f00e4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 -- 172.21.15.70:0/3112073337 shutdown_connections 2024-04-22T01:16:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.035+0000 7f00e93a6640 1 -- 172.21.15.70:0/3112073337 wait complete. 2024-04-22T01:16:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 Processor -- start 2024-04-22T01:16:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- start start 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e406b730 0x7f00e40b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e40ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 0x7f00e414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00e4168b70 con 0x7f00e415bc70 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40cff60 con 0x7f00e406b730 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f00e4168cf0 con 0x7f00e40cd740 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 0x7f00e414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 0x7f00e414e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47624/0 (socket says 172.21.15.70:47624) 2024-04-22T01:16:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 -- 172.21.15.70:0/2023797185 learned_addr learned my addr 172.21.15.70:0/2023797185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e27fc640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e40ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e2ffd640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e406b730 0x7f00e40b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 -- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 msgr2=0x7f00e40ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e40ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 -- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e406b730 msgr2=0x7f00e40b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e406b730 0x7f00e40b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 -- 172.21.15.70:0/2023797185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e40ca9d0 con 0x7f00e415bc70 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e37fe640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 0x7f00e414e7c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f00d400cbc0 tx=0x7f00d400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4017070 con 0x7f00e415bc70 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d400ad40 con 0x7f00e415bc70 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e40b51b0 con 0x7f00e415bc70 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4012600 con 0x7f00e415bc70 2024-04-22T01:16:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e40b3840 con 0x7f00e415bc70 2024-04-22T01:16:29.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.039+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e4068920 con 0x7f00e415bc70 2024-04-22T01:16:29.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.043+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00d401a070 con 0x7f00e415bc70 2024-04-22T01:16:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.043+0000 7f00db7fe640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00c40678c0 0x7f00c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.043+0000 7f00e2ffd640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00c40678c0 0x7f00c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.043+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f00d4085a40 con 0x7f00e415bc70 2024-04-22T01:16:29.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.043+0000 7f00e2ffd640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00c40678c0 0x7f00c4069d80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0af000 tx=0x7f00dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.047+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f00e415fd10 con 0x7f00e415bc70 2024-04-22T01:16:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.235+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00e40cd520 con 0x7f00e415bc70 2024-04-22T01:16:29.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.235+0000 7f00db7fe640 1 -- 172.21.15.70:0/2023797185 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5178 (secure 0 0 0) 0x7f00d4055e10 con 0x7f00e415bc70 2024-04-22T01:16:29.236 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:29.236 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:10.524071+0000","modified":"2024-04-22T01:16:13.536979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5478":{"gid":5478,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3248824947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3248824947},{"type":"v1","addr":"172.21.15.70:6837","nonce":3248824947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5478,"qdb_cluster":[5478]},"id":2}]} 2024-04-22T01:16:29.237 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00c40678c0 msgr2=0x7f00c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00c40678c0 0x7f00c4069d80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0af000 tx=0x7f00dc008040 comp rx=0 tx=0).stop 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 msgr2=0x7f00e414e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 0x7f00e414e7c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f00d400cbc0 tx=0x7f00d400a430 comp rx=0 tx=0).stop 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 shutdown_connections 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f00c40678c0 0x7f00c4069d80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00e415bc70 0x7f00e414e7c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00e40cd740 0x7f00e40ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 --2- 172.21.15.70:0/2023797185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e406b730 0x7f00e40b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 >> 172.21.15.70:0/2023797185 conn(0x7f00e4099a60 msgr2=0x7f00e40ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 shutdown_connections 2024-04-22T01:16:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.239+0000 7f00e93a6640 1 -- 172.21.15.70:0/2023797185 wait complete. 2024-04-22T01:16:29.254 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:29.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 Processor -- start 2024-04-22T01:16:29.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 -- start start 2024-04-22T01:16:29.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3cc0cea80 0x7fd3cc0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3cc0d2110 0x7fd3cc0d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc15e390 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc0cf420 con 0x7fd3cc0cea80 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d3e68640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc161e00 con 0x7fd3cc0d2110 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d1403640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3cc0cea80 0x7fd3cc0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d1403640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3cc0cea80 0x7fd3cc0ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47628/0 (socket says 172.21.15.70:47628) 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d1403640 1 -- 172.21.15.70:0/2904585716 learned_addr learned my addr 172.21.15.70:0/2904585716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.359+0000 7fd3d1c04640 1 --1- 172.21.15.70:0/2904585716 >> v1:172.21.15.70:6789/0 conn(0x7fd3cc15dfb0 0x7fd3cc0b95d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33984/0 (socket says 172.21.15.70:33984) 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d2405640 1 --2- 172.21.15.70:0/2904585716 >> v2:172.21.15.98:3300/0 conn(0x7fd3cc0d2110 0x7fd3cc0d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d1403640 1 -- 172.21.15.70:0/2904585716 >> v2:172.21.15.98:3300/0 conn(0x7fd3cc0d2110 msgr2=0x7fd3cc0d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d0c02640 1 -- 172.21.15.70:0/2904585716 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195795664 0 0) 0x7fd3cc15e390 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d1403640 1 --2- 172.21.15.70:0/2904585716 >> v2:172.21.15.98:3300/0 conn(0x7fd3cc0d2110 0x7fd3cc0d25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d1403640 1 -- 172.21.15.70:0/2904585716 >> v1:172.21.15.70:6789/0 conn(0x7fd3cc15dfb0 legacy=0x7fd3cc0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d1403640 1 -- 172.21.15.70:0/2904585716 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc1620e0 con 0x7fd3cc0cea80 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d2405640 1 --2- 172.21.15.70:0/2904585716 >> v2:172.21.15.98:3300/0 conn(0x7fd3cc0d2110 0x7fd3cc0d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d1403640 1 --2- 172.21.15.70:0/2904585716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc0ceee0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0ce4b0 tx=0x7fd3bc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8674fa56bd73a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d0c02640 1 -- 172.21.15.70:0/2904585716 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0ab430 con 0x7fd3cc0cea80 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d0c02640 1 -- 172.21.15.70:0/2904585716 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc0b0070 con 0x7fd3cc0cea80 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d0c02640 1 -- 172.21.15.70:0/2904585716 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0ab7f0 con 0x7fd3cc0cea80 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 -- 172.21.15.70:0/2904585716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc0cea80 msgr2=0x7fd3cc0ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/2904585716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc0ceee0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0ce4b0 tx=0x7fd3bc01c520 comp rx=0 tx=0).stop 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 -- 172.21.15.70:0/2904585716 shutdown_connections 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/2904585716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc0ceee0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/2904585716 >> v2:172.21.15.98:3300/0 conn(0x7fd3cc0d2110 0x7fd3cc0d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 -- 172.21.15.70:0/2904585716 >> 172.21.15.70:0/2904585716 conn(0x7fd3cc06c9b0 msgr2=0x7fd3cc06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 -- 172.21.15.70:0/2904585716 shutdown_connections 2024-04-22T01:16:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 -- 172.21.15.70:0/2904585716 wait complete. 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.363+0000 7fd3d3e68640 1 Processor -- start 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- start start 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3cc0d2110 0x7fd3cc154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 0x7fd3cc0b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc16af50 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc16add0 con 0x7fd3cc0cea80 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc16b0d0 con 0x7fd3cc0d2110 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 0x7fd3cc0b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 0x7fd3cc0b83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47642/0 (socket says 172.21.15.70:47642) 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 -- 172.21.15.70:0/1607732884 learned_addr learned my addr 172.21.15.70:0/1607732884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d1c04640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 -- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3cc0d2110 msgr2=0x7fd3cc154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3cc0d2110 0x7fd3cc154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 -- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc0cea80 msgr2=0x7fd3cc154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 -- 172.21.15.70:0/1607732884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc1554c0 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d2405640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 0x7fd3cc0b83c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400cd50 tx=0x7fd3c400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c400f680 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c400f840 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc1557a0 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c400fc30 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc155d60 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc155fa0 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3c40213f0 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3c27fc640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3b00678c0 0x7fd3b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7fd3c4086ca0 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d1c04640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3b00678c0 0x7fd3b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.367+0000 7fd3d1c04640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3b00678c0 0x7fd3b0069d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8000f70 tx=0x7fd3c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.371+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd3cc155fa0 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.523+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3cc068210 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.523+0000 7fd3c27fc640 1 -- 172.21.15.70:0/1607732884 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+16899 (secure 0 0 0) 0x7fd3c4014030 con 0x7fd3cc15dfb0 2024-04-22T01:16:29.525 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:29.525 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":40,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:28.693438+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:08.953522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:09.533622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.19","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:29.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3b00678c0 msgr2=0x7fd3b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3b00678c0 0x7fd3b0069d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8000f70 tx=0x7fd3c8009040 comp rx=0 tx=0).stop 2024-04-22T01:16:29.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 msgr2=0x7fd3cc0b83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 0x7fd3cc0b83c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400cd50 tx=0x7fd3c400a650 comp rx=0 tx=0).stop 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 shutdown_connections 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3b00678c0 0x7fd3b0069d80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3cc15dfb0 0x7fd3cc0b83c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3cc0d2110 0x7fd3cc154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 --2- 172.21.15.70:0/1607732884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3cc0cea80 0x7fd3cc154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 >> 172.21.15.70:0/1607732884 conn(0x7fd3cc06c9b0 msgr2=0x7fd3cc156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 shutdown_connections 2024-04-22T01:16:29.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.527+0000 7fd3d3e68640 1 -- 172.21.15.70:0/1607732884 wait complete. 2024-04-22T01:16:29.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:16:29.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 Processor -- start 2024-04-22T01:16:29.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 -- start start 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb38161a50 0x7fbb380ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb380ceef0 0x7fbb380d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb38161ee0 con 0x7fbb381614d0 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbb380d0170 con 0x7fbb380ceef0 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3e56e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbb380cf360 con 0x7fbb38161a50 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb380ceef0 0x7fbb380d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb380ceef0 0x7fbb380d2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47648/0 (socket says 172.21.15.70:47648) 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 -- 172.21.15.70:0/2683659715 learned_addr learned my addr 172.21.15.70:0/2683659715 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2683659715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb38161a50 0x7fbb380ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 -- 172.21.15.70:0/2683659715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb38161a50 msgr2=0x7fbb380ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 --2- 172.21.15.70:0/2683659715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb38161a50 0x7fbb380ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 -- 172.21.15.70:0/2683659715 >> v1:172.21.15.70:6789/0 conn(0x7fbb381614d0 legacy=0x7fbb380b9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2683659715 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294133033 0 0) 0x7fbb38161ee0 con 0x7fbb381614d0 2024-04-22T01:16:29.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.655+0000 7fbb3cb0b640 1 -- 172.21.15.70:0/2683659715 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb381621c0 con 0x7fbb380ceef0 2024-04-22T01:16:29.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3cb0b640 1 --2- 172.21.15.70:0/2683659715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb380d2570 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009e50 tx=0x7fbb3001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65a6e057c736654 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2683659715 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300a5700 con 0x7fbb380ceef0 2024-04-22T01:16:29.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2683659715 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb300a58a0 con 0x7fbb380ceef0 2024-04-22T01:16:29.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb36ffd640 1 -- 172.21.15.70:0/2683659715 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300a5c70 con 0x7fbb380ceef0 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2683659715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 msgr2=0x7fbb380d2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2683659715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb380d2570 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009e50 tx=0x7fbb3001ca30 comp rx=0 tx=0).stop 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2683659715 shutdown_connections 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2683659715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb380d2570 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2683659715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb38161a50 0x7fbb380ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2683659715 >> 172.21.15.70:0/2683659715 conn(0x7fbb3806c9d0 msgr2=0x7fbb3806cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2683659715 shutdown_connections 2024-04-22T01:16:29.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2683659715 wait complete. 2024-04-22T01:16:29.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 Processor -- start 2024-04-22T01:16:29.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- start start 2024-04-22T01:16:29.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb3814cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb381614d0 0x7fbb380c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb380ca420 0x7fbb3815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb380d3650 con 0x7fbb380ceef0 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb380d34d0 con 0x7fbb380ca420 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb3e56e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbb380d37d0 con 0x7fbb381614d0 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb3814cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb3814cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47656/0 (socket says 172.21.15.70:47656) 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.659+0000 7fbb37fff640 1 -- 172.21.15.70:0/2893927670 learned_addr learned my addr 172.21.15.70:0/2893927670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb3cb0b640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb380ca420 0x7fbb3815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb37fff640 1 -- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb381614d0 msgr2=0x7fbb380c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb37fff640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb381614d0 0x7fbb380c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb37fff640 1 -- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb380ca420 msgr2=0x7fbb3815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb37fff640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb380ca420 0x7fbb3815fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb37fff640 1 -- 172.21.15.70:0/2893927670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb380cdc20 con 0x7fbb380ceef0 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb37fff640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb3814cb20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800c4d0 tx=0x7fbb2800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28017070 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb28005480 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb380cd380 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28003840 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb380cd940 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3804f510 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb2801a070 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb34ff9640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb240678c0 0x7fbb24069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7fbb280851f0 con 0x7fbb380ceef0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb240678c0 0x7fbb24069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.663+0000 7fbb377fe640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb240678c0 0x7fbb24069d80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbb38156e00 tx=0x7fbb2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.667+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbb28055cd0 con 0x7fbb380ceef0 2024-04-22T01:16:29.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.851+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbb38160d00 con 0x7fbb380ceef0 2024-04-22T01:16:29.851 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:16:29.851+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:16:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb34ff9640 1 -- 172.21.15.70:0/2893927670 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7fbb280555d0 con 0x7fbb380ceef0 2024-04-22T01:16:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb240678c0 msgr2=0x7fbb24069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb240678c0 0x7fbb24069d80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbb38156e00 tx=0x7fbb2c008040 comp rx=0 tx=0).stop 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 msgr2=0x7fbb3814cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb3814cb20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800c4d0 tx=0x7fbb2800c990 comp rx=0 tx=0).stop 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 shutdown_connections 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb240678c0 0x7fbb24069d80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb380ca420 0x7fbb3815fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb381614d0 0x7fbb380c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 --2- 172.21.15.70:0/2893927670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb380ceef0 0x7fbb3814cb20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 >> 172.21.15.70:0/2893927670 conn(0x7fbb3806c9d0 msgr2=0x7fbb380b81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 shutdown_connections 2024-04-22T01:16:29.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.855+0000 7fbb3e56e640 1 -- 172.21.15.70:0/2893927670 wait complete. 2024-04-22T01:16:29.859 INFO:tasks.ceph.mds.c.smithi070.stderr: -2009> 2024-04-22T01:16:15.147+0000 7f2efe046640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:16:29.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:16:29.895 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:16:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 Processor -- start 2024-04-22T01:16:29.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 -- start start 2024-04-22T01:16:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f24f015de20 0x7f24f00b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24f00cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f00cf3c0 con 0x7f24f00b6db0 2024-04-22T01:16:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f24f00b75e0 con 0x7f24f015de20 2024-04-22T01:16:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f6814640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f24f00b78c0 con 0x7f24f00cea10 2024-04-22T01:16:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24ef7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f24f00b6db0 0x7f24f00d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34000/0 (socket says 172.21.15.70:34000) 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24ef7fe640 1 -- 172.21.15.70:0/3733950652 learned_addr learned my addr 172.21.15.70:0/3733950652 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24f00cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24effff640 1 --2- 172.21.15.70:0/3733950652 >> v2:172.21.15.98:3300/0 conn(0x7f24f015de20 0x7f24f00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240268183 0 0) 0x7f24f00cf3c0 con 0x7f24f00b6db0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24d8003530 con 0x7f24f00b6db0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 912687784 0 0) 0x7f24d8003530 con 0x7f24f00b6db0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 msgr2=0x7f24f00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24f00cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 >> v2:172.21.15.98:3300/0 conn(0x7f24f015de20 msgr2=0x7f24f00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 --2- 172.21.15.70:0/3733950652 >> v2:172.21.15.98:3300/0 conn(0x7f24f015de20 0x7f24f00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f00b7ba0 con 0x7f24f00b6db0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f24e4001680 con 0x7f24f00b6db0 2024-04-22T01:16:29.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 >> v1:172.21.15.70:6789/0 conn(0x7f24f00b6db0 legacy=0x7f24f00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24d80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d8003f40 0x7f24d8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24d80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00b75e0 con 0x7f24d8004d50 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00b78c0 con 0x7f24d8003f40 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f24d800ba00 con 0x7f24f00cea10 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24d80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24effff640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24d80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24ef7fe640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d8003f40 0x7f24d8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 -- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 msgr2=0x7f24d80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24d80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 -- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d8003f40 msgr2=0x7f24d8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d8003f40 0x7f24d8004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24f4db1640 1 -- 172.21.15.70:0/3733950652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d8003530 con 0x7f24d8004d50 2024-04-22T01:16:29.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.987+0000 7f24effff640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00cea10 0x7f24d80039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f4db1640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24d80051d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009ac0 tx=0x7f24e001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a29e1a0fe90f807 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e00b2070 con 0x7f24d8004d50 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e00a5640 con 0x7f24d8004d50 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24eeffd640 1 -- 172.21.15.70:0/3733950652 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0009690 con 0x7f24d8004d50 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 -- 172.21.15.70:0/3733950652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 msgr2=0x7f24d80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 --2- 172.21.15.70:0/3733950652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24d80051d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009ac0 tx=0x7f24e001ca50 comp rx=0 tx=0).stop 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f4db1640 1 -- 172.21.15.70:0/3733950652 reap_dead start 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 -- 172.21.15.70:0/3733950652 shutdown_connections 2024-04-22T01:16:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 -- 172.21.15.70:0/3733950652 >> 172.21.15.70:0/3733950652 conn(0x7f24f0072d20 msgr2=0x7f24f0073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 -- 172.21.15.70:0/3733950652 shutdown_connections 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 -- 172.21.15.70:0/3733950652 wait complete. 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 Processor -- start 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 -- start start 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24f014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.991+0000 7f24f6814640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24f01f2f50 0x7f24f01f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00d0c00 0x7f24f00d1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24f015e200 con 0x7f24d8004d50 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00b8160 con 0x7f24f01f2f50 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f24f00b71f0 con 0x7f24f00d0c00 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24f014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24f014c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47672/0 (socket says 172.21.15.70:47672) 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 -- 172.21.15.70:0/425198385 learned_addr learned my addr 172.21.15.70:0/425198385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ef7fe640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24f01f2f50 0x7f24f01f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 -- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00d0c00 msgr2=0x7f24f00d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00d0c00 0x7f24f00d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 -- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24f01f2f50 msgr2=0x7f24f01f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24f01f2f50 0x7f24f01f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 -- 172.21.15.70:0/425198385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f014cba0 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24effff640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24f014c660 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f24e800f430 tx=0x7f24e800f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e80039a0 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8003b40 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f01f3b20 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8004020 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f01f40e0 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f0068210 con 0x7f24d8004d50 2024-04-22T01:16:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f24e801d030 con 0x7f24d8004d50 2024-04-22T01:16:29.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ecff9640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24dc0678c0 0x7f24dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:29.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f24e8086910 con 0x7f24d8004d50 2024-04-22T01:16:29.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ef7fe640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24dc0678c0 0x7f24dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:29.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.995+0000 7f24ef7fe640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24dc0678c0 0x7f24dc069d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24e4000f70 tx=0x7f24e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:30.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:29.999+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f24e8024020 con 0x7f24d8004d50 2024-04-22T01:16:30.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.187+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f24f0162260 con 0x7f24d8004d50 2024-04-22T01:16:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.855+0000 7f24ecff9640 1 -- 172.21.15.70:0/425198385 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f24e80572e0 con 0x7f24d8004d50 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24dc0678c0 msgr2=0x7f24dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24dc0678c0 0x7f24dc069d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24e4000f70 tx=0x7f24e4009040 comp rx=0 tx=0).stop 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 msgr2=0x7f24f014c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24f014c660 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f24e800f430 tx=0x7f24e800f8f0 comp rx=0 tx=0).stop 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 shutdown_connections 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24dc0678c0 0x7f24dc069d80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24f00d0c00 0x7f24f00d1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24f01f2f50 0x7f24f01f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 --2- 172.21.15.70:0/425198385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24d8004d50 0x7f24f014c660 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 >> 172.21.15.70:0/425198385 conn(0x7f24f0072d20 msgr2=0x7f24f00b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 shutdown_connections 2024-04-22T01:16:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:30.859+0000 7f24f6814640 1 -- 172.21.15.70:0/425198385 wait complete. 2024-04-22T01:16:30.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:16:31.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 Processor -- start 2024-04-22T01:16:31.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 -- start start 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f702406b900 0x7f7024161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cd090 0x7f70240cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7024162040 con 0x7f70240cca20 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f70240b02b0 con 0x7f702406b900 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f702a1c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7024162320 con 0x7f70240cd090 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f70237fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f70240cca20 0x7f702406b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34016/0 (socket says 172.21.15.70:34016) 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f70237fe640 1 -- 172.21.15.70:0/452744070 learned_addr learned my addr 172.21.15.70:0/452744070 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7022ffd640 1 --2- 172.21.15.70:0/452744070 >> v2:172.21.15.70:3300/0 conn(0x7f702406b900 0x7f7024161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7023fff640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cd090 0x7f70240cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f70227fc640 1 -- 172.21.15.70:0/452744070 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 808813831 0 0) 0x7f7024162040 con 0x7f70240cca20 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f70227fc640 1 -- 172.21.15.70:0/452744070 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f700c003530 con 0x7f70240cca20 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7022ffd640 1 -- 172.21.15.70:0/452744070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cd090 msgr2=0x7f70240cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7022ffd640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cd090 0x7f70240cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7022ffd640 1 -- 172.21.15.70:0/452744070 >> v1:172.21.15.70:6789/0 conn(0x7f70240cca20 legacy=0x7f702406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f70227fc640 1 -- 172.21.15.70:0/452744070 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1266702792 0 0) 0x7f700c003530 con 0x7f70240cca20 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7022ffd640 1 -- 172.21.15.70:0/452744070 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7024162600 con 0x7f702406b900 2024-04-22T01:16:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.055+0000 7f7023fff640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cd090 0x7f70240cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f7022ffd640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b900 0x7f7024161a60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f70180097c0 tx=0x7f701801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2d62b45ea45803 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f70227fc640 1 -- 172.21.15.70:0/452744070 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70180b0070 con 0x7f702406b900 2024-04-22T01:16:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f70227fc640 1 -- 172.21.15.70:0/452744070 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70180ab400 con 0x7f702406b900 2024-04-22T01:16:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f70227fc640 1 -- 172.21.15.70:0/452744070 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c001a00 con 0x7f702406b900 2024-04-22T01:16:31.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- 172.21.15.70:0/452744070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b900 msgr2=0x7f7024161a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:31.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b900 0x7f7024161a60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f70180097c0 tx=0x7f701801c3c0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- 172.21.15.70:0/452744070 shutdown_connections 2024-04-22T01:16:31.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cd090 0x7f70240cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 --2- 172.21.15.70:0/452744070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b900 0x7f7024161a60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- 172.21.15.70:0/452744070 >> 172.21.15.70:0/452744070 conn(0x7f7024099a60 msgr2=0x7f7024099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:31.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- 172.21.15.70:0/452744070 shutdown_connections 2024-04-22T01:16:31.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- 172.21.15.70:0/452744070 wait complete. 2024-04-22T01:16:31.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 Processor -- start 2024-04-22T01:16:31.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- start start 2024-04-22T01:16:31.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f702406b900 0x7f70241508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:31.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cca20 0x7f702415fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:31.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 0x7f70240c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:31.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f702415f100 con 0x7f70240cd090 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f702415ef80 con 0x7f702406b900 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f702a1c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f702415f280 con 0x7f70240cca20 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.059+0000 7f7023fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 0x7f70240c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 0x7f70240c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47682/0 (socket says 172.21.15.70:47682) 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 -- 172.21.15.70:0/228092976 learned_addr learned my addr 172.21.15.70:0/228092976 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7022ffd640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cca20 0x7f702415fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f70237fe640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f702406b900 0x7f70241508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 -- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cca20 msgr2=0x7f702415fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cca20 0x7f702415fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 -- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f702406b900 msgr2=0x7f70241508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f702406b900 0x7f70241508f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 -- 172.21.15.70:0/228092976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7024154cd0 con 0x7f70240cd090 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f7023fff640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 0x7f70240c5f10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f701400e810 tx=0x7f701400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:31.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014017070 con 0x7f70240cd090 2024-04-22T01:16:31.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f701400c6d0 con 0x7f70240cd090 2024-04-22T01:16:31.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701400f040 con 0x7f70240cd090 2024-04-22T01:16:31.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7024154fb0 con 0x7f70240cd090 2024-04-22T01:16:31.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7024155570 con 0x7f70240cd090 2024-04-22T01:16:31.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7024068920 con 0x7f70240cd090 2024-04-22T01:16:31.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f701401a070 con 0x7f70240cd090 2024-04-22T01:16:31.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f700bfff640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70100678c0 0x7f7010069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:31.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f70237fe640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70100678c0 0x7f7010069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:31.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f7014085a90 con 0x7f70240cd090 2024-04-22T01:16:31.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.063+0000 7f70237fe640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70100678c0 0x7f7010069d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f701c00e1b0 tx=0x7f701c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:31.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.067+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f70140563d0 con 0x7f70240cd090 2024-04-22T01:16:31.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.219+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f70240cd6a0 con 0x7f70240cd090 2024-04-22T01:16:31.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.867+0000 7f700bfff640 1 -- 172.21.15.70:0/228092976 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v42) ==== 181+0+0 (secure 0 0 0) 0x7f7014055cd0 con 0x7f70240cd090 2024-04-22T01:16:31.866 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:16:31.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.871+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70100678c0 msgr2=0x7f7010069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:31.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.871+0000 7f702a1c6640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70100678c0 0x7f7010069d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f701c00e1b0 tx=0x7f701c009040 comp rx=0 tx=0).stop 2024-04-22T01:16:31.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.871+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 msgr2=0x7f70240c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:31.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.871+0000 7f702a1c6640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 0x7f70240c5f10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f701400e810 tx=0x7f701400ecd0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 shutdown_connections 2024-04-22T01:16:31.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70100678c0 0x7f7010069d80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70240cd090 0x7f70240c5f10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cca20 0x7f702415fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 --2- 172.21.15.70:0/228092976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f702406b900 0x7f70241508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:31.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 >> 172.21.15.70:0/228092976 conn(0x7f7024099a60 msgr2=0x7f7024161080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:31.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 shutdown_connections 2024-04-22T01:16:31.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:31.875+0000 7f702a1c6640 1 -- 172.21.15.70:0/228092976 wait complete. 2024-04-22T01:16:31.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:16:32.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eef34a640 1 Processor -- start 2024-04-22T01:16:32.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eef34a640 1 -- start start 2024-04-22T01:16:32.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eef34a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8ee80cd220 0x7f8ee8166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:32.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eef34a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee815c290 con 0x7f8ee8072280 2024-04-22T01:16:32.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eef34a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee815c570 con 0x7f8ee80b4450 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eef34a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee80b3a50 con 0x7f8ee80cd220 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eec8e5640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8ee8072280 0x7f8ee80ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34032/0 (socket says 172.21.15.70:34032) 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eec8e5640 1 -- 172.21.15.70:0/2870845796 learned_addr learned my addr 172.21.15.70:0/2870845796 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eed8e7640 1 --2- 172.21.15.70:0/2870845796 >> v2:172.21.15.70:3300/0 conn(0x7f8ee80cd220 0x7f8ee8166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eed8e7640 1 -- 172.21.15.70:0/2870845796 >> v1:172.21.15.98:6789/0 conn(0x7f8ee80b4450 legacy=0x7f8ee8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eed8e7640 1 -- 172.21.15.70:0/2870845796 >> v1:172.21.15.70:6789/0 conn(0x7f8ee8072280 legacy=0x7f8ee80ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8eed8e7640 1 -- 172.21.15.70:0/2870845796 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee815c850 con 0x7f8ee80cd220 2024-04-22T01:16:32.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.019+0000 7f8edffff640 1 -- 172.21.15.70:0/2870845796 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392637076 0 0) 0x7f8ee815c290 con 0x7f8ee8072280 2024-04-22T01:16:32.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eed8e7640 1 --2- 172.21.15.70:0/2870845796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee80cd220 0x7f8ee8166b00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009d90 tx=0x7f8ee001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd7ba653abe3114 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:32.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8edffff640 1 -- 172.21.15.70:0/2870845796 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee00b2070 con 0x7f8ee80cd220 2024-04-22T01:16:32.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8edffff640 1 -- 172.21.15.70:0/2870845796 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee00a5870 con 0x7f8ee80cd220 2024-04-22T01:16:32.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8edffff640 1 -- 172.21.15.70:0/2870845796 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee00a5c40 con 0x7f8ee80cd220 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 -- 172.21.15.70:0/2870845796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee80cd220 msgr2=0x7f8ee8166b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2870845796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee80cd220 0x7f8ee8166b00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009d90 tx=0x7f8ee001c930 comp rx=0 tx=0).stop 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 -- 172.21.15.70:0/2870845796 shutdown_connections 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2870845796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee80cd220 0x7f8ee8166b00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 -- 172.21.15.70:0/2870845796 >> 172.21.15.70:0/2870845796 conn(0x7f8ee80b1d40 msgr2=0x7f8ee80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 -- 172.21.15.70:0/2870845796 shutdown_connections 2024-04-22T01:16:32.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 -- 172.21.15.70:0/2870845796 wait complete. 2024-04-22T01:16:32.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 Processor -- start 2024-04-22T01:16:32.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.023+0000 7f8eef34a640 1 -- start start 2024-04-22T01:16:32.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 0x7f8ee815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ee80b4450 0x7f8ee8149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80cd220 0x7f8ee81f8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee80cd980 con 0x7f8ee8072280 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee80cd800 con 0x7f8ee80cd220 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee8169860 con 0x7f8ee80b4450 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 0x7f8ee815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 0x7f8ee815b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47714/0 (socket says 172.21.15.70:47714) 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 -- 172.21.15.70:0/2941192267 learned_addr learned my addr 172.21.15.70:0/2941192267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed8e7640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80cd220 0x7f8ee81f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 -- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ee80b4450 msgr2=0x7f8ee8149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ee80b4450 0x7f8ee8149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 -- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80cd220 msgr2=0x7f8ee81f8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80cd220 0x7f8ee81f8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 -- 172.21.15.70:0/2941192267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee8149db0 con 0x7f8ee8072280 2024-04-22T01:16:32.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eed0e6640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 0x7f8ee815b840 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400c450 tx=0x7f8ee400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:32.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4012830 con 0x7f8ee8072280 2024-04-22T01:16:32.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee4017070 con 0x7f8ee8072280 2024-04-22T01:16:32.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4012c30 con 0x7f8ee8072280 2024-04-22T01:16:32.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee814a090 con 0x7f8ee8072280 2024-04-22T01:16:32.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee814a650 con 0x7f8ee8072280 2024-04-22T01:16:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee8068210 con 0x7f8ee8072280 2024-04-22T01:16:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ee401c460 con 0x7f8ee8072280 2024-04-22T01:16:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eddffb640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ecc0678c0 0x7f8ecc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5778+0+0 (secure 0 0 0) 0x7f8ee4085ac0 con 0x7f8ee8072280 2024-04-22T01:16:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eec8e5640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ecc0678c0 0x7f8ecc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.027+0000 7f8eec8e5640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ecc0678c0 0x7f8ecc069d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80011f0 tx=0x7f8ed8007930 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:32.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.031+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8ee40565b0 con 0x7f8ee8072280 2024-04-22T01:16:32.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.191+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8ee80d01e0 con 0x7f8ee8072280 2024-04-22T01:16:32.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.875+0000 7f8eddffb640 1 -- 172.21.15.70:0/2941192267 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v43) ==== 169+0+0 (secure 0 0 0) 0x7f8ee4055eb0 con 0x7f8ee8072280 2024-04-22T01:16:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:16:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ecc0678c0 msgr2=0x7f8ecc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ecc0678c0 0x7f8ecc069d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80011f0 tx=0x7f8ed8007930 comp rx=0 tx=0).stop 2024-04-22T01:16:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 msgr2=0x7f8ee815b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:32.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 0x7f8ee815b840 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400c450 tx=0x7f8ee400c910 comp rx=0 tx=0).stop 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 shutdown_connections 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ecc0678c0 0x7f8ecc069d80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80cd220 0x7f8ee81f8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ee80b4450 0x7f8ee8149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 --2- 172.21.15.70:0/2941192267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ee8072280 0x7f8ee815b840 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.879+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 >> 172.21.15.70:0/2941192267 conn(0x7f8ee80b1d40 msgr2=0x7f8ee8071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.883+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 shutdown_connections 2024-04-22T01:16:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:32.883+0000 7f8eef34a640 1 -- 172.21.15.70:0/2941192267 wait complete. 2024-04-22T01:16:32.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-22T01:16:33.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 Processor -- start 2024-04-22T01:16:33.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 -- start start 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fefcc0ceda0 0x7fefcc1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefcc0b3800 con 0x7fefcc15e6e0 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fefcc0b3ae0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.023+0000 7fefd4548640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fefcc0cde40 con 0x7fefcc0ce920 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd22e4640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fefcc15e6e0 0x7fefcc15eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34034/0 (socket says 172.21.15.70:34034) 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd22e4640 1 -- 172.21.15.70:0/2558428560 learned_addr learned my addr 172.21.15.70:0/2558428560 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd2ae5640 1 --2- 172.21.15.70:0/2558428560 >> v2:172.21.15.70:3300/0 conn(0x7fefcc0ceda0 0x7fefcc1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd1ae3640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd2ae5640 1 -- 172.21.15.70:0/2558428560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 msgr2=0x7fefcc154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd2ae5640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd2ae5640 1 -- 172.21.15.70:0/2558428560 >> v1:172.21.15.70:6789/0 conn(0x7fefcc15e6e0 legacy=0x7fefcc15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd2ae5640 1 -- 172.21.15.70:0/2558428560 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc0ce120 con 0x7fefcc0ceda0 2024-04-22T01:16:33.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd12e2640 1 -- 172.21.15.70:0/2558428560 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2303152945 0 0) 0x7fefcc0b3800 con 0x7fefcc15e6e0 2024-04-22T01:16:33.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd1ae3640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:33.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd2ae5640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc1551a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009d90 tx=0x7fefc401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf93516b0399f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:33.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd12e2640 1 -- 172.21.15.70:0/2558428560 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40b2070 con 0x7fefcc0ceda0 2024-04-22T01:16:33.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd12e2640 1 -- 172.21.15.70:0/2558428560 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc40a5a20 con 0x7fefcc0ceda0 2024-04-22T01:16:33.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd12e2640 1 -- 172.21.15.70:0/2558428560 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40a5df0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 -- 172.21.15.70:0/2558428560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 msgr2=0x7fefcc1551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc1551a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009d90 tx=0x7fefc401c940 comp rx=0 tx=0).stop 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 -- 172.21.15.70:0/2558428560 shutdown_connections 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc1551a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 --2- 172.21.15.70:0/2558428560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 -- 172.21.15.70:0/2558428560 >> 172.21.15.70:0/2558428560 conn(0x7fefcc0b1d40 msgr2=0x7fefcc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 -- 172.21.15.70:0/2558428560 shutdown_connections 2024-04-22T01:16:33.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.027+0000 7fefd4548640 1 -- 172.21.15.70:0/2558428560 wait complete. 2024-04-22T01:16:33.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 Processor -- start 2024-04-22T01:16:33.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- start start 2024-04-22T01:16:33.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefcc15e6e0 0x7fefcc14cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc0b2e40 con 0x7fefcc0ceda0 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc0b2cc0 con 0x7fefcc15e6e0 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fefcc0b2fc0 con 0x7fefcc0ce920 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47734/0 (socket says 172.21.15.70:47734) 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 -- 172.21.15.70:0/696882144 learned_addr learned my addr 172.21.15.70:0/696882144 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 -- 172.21.15.70:0/696882144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 msgr2=0x7fefcc146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 -- 172.21.15.70:0/696882144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefcc15e6e0 msgr2=0x7fefcc14cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefcc15e6e0 0x7fefcc14cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 -- 172.21.15.70:0/696882144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc15dcf0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd1ae3640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc14c960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00c7c0 tx=0x7fefbc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:33.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc007a30 con 0x7fefcc0ceda0 2024-04-22T01:16:33.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefbc007bd0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefcc15dfd0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0113f0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefcc0b1260 con 0x7fefcc0ceda0 2024-04-22T01:16:33.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefcc068210 con 0x7fefcc0ceda0 2024-04-22T01:16:33.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.031+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fefbc011590 con 0x7fefcc0ceda0 2024-04-22T01:16:33.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.035+0000 7fefc2ffd640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefb00678c0 0x7fefb0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.035+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7fefbc085ca0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.035+0000 7fefd22e4640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefb00678c0 0x7fefb0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.035+0000 7fefd22e4640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefb00678c0 0x7fefb0069d80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fefc80015a0 tx=0x7fefc8007970 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:33.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.039+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fefbc015020 con 0x7fefcc0ceda0 2024-04-22T01:16:33.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.191+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fefcc0d08e0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.199+0000 7fefc2ffd640 1 -- 172.21.15.70:0/696882144 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fefbc0569e0 con 0x7fefcc0ceda0 2024-04-22T01:16:33.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.199+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefb00678c0 msgr2=0x7fefb0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefb00678c0 0x7fefb0069d80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fefc80015a0 tx=0x7fefc8007970 comp rx=0 tx=0).stop 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 msgr2=0x7fefcc14c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc14c960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00c7c0 tx=0x7fefbc00cc80 comp rx=0 tx=0).stop 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 shutdown_connections 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fefb00678c0 0x7fefb0069d80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefcc15e6e0 0x7fefcc14cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefcc0ceda0 0x7fefcc14c960 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 --2- 172.21.15.70:0/696882144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefcc0ce920 0x7fefcc146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 >> 172.21.15.70:0/696882144 conn(0x7fefcc0b1d40 msgr2=0x7fefcc153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 shutdown_connections 2024-04-22T01:16:33.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.203+0000 7fefd4548640 1 -- 172.21.15.70:0/696882144 wait complete. 2024-04-22T01:16:33.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values' 2024-04-22T01:16:33.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.323+0000 7fe6f0758640 1 Processor -- start 2024-04-22T01:16:33.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.323+0000 7fe6f0758640 1 -- start start 2024-04-22T01:16:33.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6f0758640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe6e81455b0 0x7fe6e80cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6f0758640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6e80d0250 con 0x7fe6e8072230 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6f0758640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6e80cd7b0 con 0x7fe6e80cc150 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6f0758640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe6e80cc710 con 0x7fe6e81455b0 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6eecf5640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe6e8072230 0x7fe6e806e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34036/0 (socket says 172.21.15.70:34036) 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6eecf5640 1 -- 172.21.15.70:0/1487811778 learned_addr learned my addr 172.21.15.70:0/1487811778 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ee4f4640 1 --2- 172.21.15.70:0/1487811778 >> v2:172.21.15.70:3300/0 conn(0x7fe6e81455b0 0x7fe6e80cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ed4f2640 1 -- 172.21.15.70:0/1487811778 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062773993 0 0) 0x7fe6e80d0250 con 0x7fe6e8072230 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ed4f2640 1 -- 172.21.15.70:0/1487811778 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6d0003530 con 0x7fe6e8072230 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ed4f2640 1 -- 172.21.15.70:0/1487811778 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1061782271 0 0) 0x7fe6d0003530 con 0x7fe6e8072230 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/1487811778 >> v1:172.21.15.98:6789/0 conn(0x7fe6e80cc150 legacy=0x7fe6e8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/1487811778 >> v1:172.21.15.70:6789/0 conn(0x7fe6e8072230 legacy=0x7fe6e806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.327+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/1487811778 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e80cc9f0 con 0x7fe6e81455b0 2024-04-22T01:16:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6ee4f4640 1 --2- 172.21.15.70:0/1487811778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81455b0 0x7fe6e80cfd10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097c0 tx=0x7fe6e001c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbac91458e3c62e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6ed4f2640 1 -- 172.21.15.70:0/1487811778 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b0070 con 0x7fe6e81455b0 2024-04-22T01:16:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6ed4f2640 1 -- 172.21.15.70:0/1487811778 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e00ab400 con 0x7fe6e81455b0 2024-04-22T01:16:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6ed4f2640 1 -- 172.21.15.70:0/1487811778 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc001680 con 0x7fe6e81455b0 2024-04-22T01:16:33.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 -- 172.21.15.70:0/1487811778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81455b0 msgr2=0x7fe6e80cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 --2- 172.21.15.70:0/1487811778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81455b0 0x7fe6e80cfd10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097c0 tx=0x7fe6e001c2a0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 -- 172.21.15.70:0/1487811778 shutdown_connections 2024-04-22T01:16:33.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 --2- 172.21.15.70:0/1487811778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81455b0 0x7fe6e80cfd10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 -- 172.21.15.70:0/1487811778 >> 172.21.15.70:0/1487811778 conn(0x7fe6e806b1f0 msgr2=0x7fe6e806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 -- 172.21.15.70:0/1487811778 shutdown_connections 2024-04-22T01:16:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.331+0000 7fe6f0758640 1 -- 172.21.15.70:0/1487811778 wait complete. 2024-04-22T01:16:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 Processor -- start 2024-04-22T01:16:33.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- start start 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 0x7fe6e8134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e80cc150 0x7fe6e8139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e81455b0 0x7fe6e8144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e8169ee0 con 0x7fe6e8072230 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e8169d60 con 0x7fe6e81455b0 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e816a060 con 0x7fe6e80cc150 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 0x7fe6e8134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 0x7fe6e8134bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47750/0 (socket says 172.21.15.70:47750) 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/634809669 learned_addr learned my addr 172.21.15.70:0/634809669 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:33.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/634809669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e80cc150 msgr2=0x7fe6e8139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e80cc150 0x7fe6e8139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/634809669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e81455b0 msgr2=0x7fe6e8144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e81455b0 0x7fe6e8144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 -- 172.21.15.70:0/634809669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e813d790 con 0x7fe6e8072230 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6ee4f4640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 0x7fe6e8134bc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0000c10 tx=0x7fe6e0019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b9020 con 0x7fe6e8072230 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e0003920 con 0x7fe6e8072230 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b0040 con 0x7fe6e8072230 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e81351a0 con 0x7fe6e8072230 2024-04-22T01:16:33.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e813d4b0 con 0x7fe6e8072230 2024-04-22T01:16:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.335+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6e8068210 con 0x7fe6e8072230 2024-04-22T01:16:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.339+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe6e00bc020 con 0x7fe6e8072230 2024-04-22T01:16:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.339+0000 7fe6e6ffd640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6d406bcd0 0x7fe6d406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.339+0000 7fe6edcf3640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6d406bcd0 0x7fe6d406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.339+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7fe6e011fbe0 con 0x7fe6e8072230 2024-04-22T01:16:33.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.339+0000 7fe6edcf3640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6d406bcd0 0x7fe6d406e190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8007fd0 tx=0x7fe6d8007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:33.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.343+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe6e00c3030 con 0x7fe6e8072230 2024-04-22T01:16:33.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.543+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]} v 0) -- 0x7fe6e815f510 con 0x7fe6e8072230 2024-04-22T01:16:33.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.879+0000 7fe6e6ffd640 1 -- 172.21.15.70:0/634809669 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]}]=0 v121) ==== 160+0+0 (secure 0 0 0) 0x7fe6e0015fb0 con 0x7fe6e8072230 2024-04-22T01:16:33.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6d406bcd0 msgr2=0x7fe6d406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6d406bcd0 0x7fe6d406e190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8007fd0 tx=0x7fe6d8007d40 comp rx=0 tx=0).stop 2024-04-22T01:16:33.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 msgr2=0x7fe6e8134bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:33.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 0x7fe6e8134bc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0000c10 tx=0x7fe6e0019040 comp rx=0 tx=0).stop 2024-04-22T01:16:33.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 shutdown_connections 2024-04-22T01:16:33.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe6d406bcd0 0x7fe6d406e190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e81455b0 0x7fe6e8144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e80cc150 0x7fe6e8139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 --2- 172.21.15.70:0/634809669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e8072230 0x7fe6e8134bc0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:33.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 >> 172.21.15.70:0/634809669 conn(0x7fe6e806b1f0 msgr2=0x7fe6e815c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:33.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 shutdown_connections 2024-04-22T01:16:33.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:33.883+0000 7fe6f0758640 1 -- 172.21.15.70:0/634809669 wait complete. 2024-04-22T01:16:33.903 INFO:tasks.cephfs_test_runner:test: ... ok 2024-04-22T01:16:33.903 INFO:tasks.cephfs_test_runner:Starting test: test_human_readable_quota_invalid_values (tasks.cephfs.test_quota.TestQuota) test invalid values for ceph.quota.max_bytes 2024-04-22T01:16:33.903 INFO:tasks.cephfs_test_runner:test_human_readable_quota_invalid_values (tasks.cephfs.test_quota.TestQuota) 2024-04-22T01:16:33.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:34.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.027+0000 7feed1f45640 1 Processor -- start 2024-04-22T01:16:34.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.027+0000 7feed1f45640 1 -- start start 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feecc15dfb0 0x7feecc0b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7feecc0d3120 0x7feecc0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feecc15e390 con 0x7feecc0cea80 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7feecc0cf420 con 0x7feecc0d3120 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7feecc161e00 con 0x7feecc15dfb0 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7feecc0d3120 0x7feecc0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7feecc0d3120 0x7feecc0ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47756/0 (socket says 172.21.15.70:47756) 2024-04-22T01:16:34.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 -- 172.21.15.70:0/2954084424 learned_addr learned my addr 172.21.15.70:0/2954084424 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:34.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecb7fe640 1 --2- 172.21.15.70:0/2954084424 >> v2:172.21.15.98:3300/0 conn(0x7feecc15dfb0 0x7feecc0b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 -- 172.21.15.70:0/2954084424 >> v2:172.21.15.98:3300/0 conn(0x7feecc15dfb0 msgr2=0x7feecc0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:34.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 --2- 172.21.15.70:0/2954084424 >> v2:172.21.15.98:3300/0 conn(0x7feecc15dfb0 0x7feecc0b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 -- 172.21.15.70:0/2954084424 >> v1:172.21.15.98:6789/0 conn(0x7feecc0cea80 legacy=0x7feecc0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:34.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 -- 172.21.15.70:0/2954084424 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc1620e0 con 0x7feecc0d3120 2024-04-22T01:16:34.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecbfff640 1 --2- 172.21.15.70:0/2954084424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0d3120 0x7feecc0ceee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7feecc0d4d80 tx=0x7feec401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10af0c6e6628361 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecaffd640 1 -- 172.21.15.70:0/2954084424 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec40a56a0 con 0x7feecc0d3120 2024-04-22T01:16:34.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecaffd640 1 -- 172.21.15.70:0/2954084424 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec40a5840 con 0x7feecc0d3120 2024-04-22T01:16:34.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feecaffd640 1 -- 172.21.15.70:0/2954084424 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec40a5c10 con 0x7feecc0d3120 2024-04-22T01:16:34.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- 172.21.15.70:0/2954084424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0d3120 msgr2=0x7feecc0ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 --2- 172.21.15.70:0/2954084424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0d3120 0x7feecc0ceee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7feecc0d4d80 tx=0x7feec401c9b0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- 172.21.15.70:0/2954084424 shutdown_connections 2024-04-22T01:16:34.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 --2- 172.21.15.70:0/2954084424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0d3120 0x7feecc0ceee0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 --2- 172.21.15.70:0/2954084424 >> v2:172.21.15.98:3300/0 conn(0x7feecc15dfb0 0x7feecc0b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- 172.21.15.70:0/2954084424 >> 172.21.15.70:0/2954084424 conn(0x7feecc06c9b0 msgr2=0x7feecc06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:34.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- 172.21.15.70:0/2954084424 shutdown_connections 2024-04-22T01:16:34.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.031+0000 7feed1f45640 1 -- 172.21.15.70:0/2954084424 wait complete. 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 Processor -- start 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- start start 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 0x7feecc153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feecc0d3120 0x7feecc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feecc15dfb0 0x7feecc15caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feecc16add0 con 0x7feecc0cea80 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feecc161a90 con 0x7feecc0d3120 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7feecc16af50 con 0x7feecc15dfb0 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 0x7feecc153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 0x7feecc153000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47766/0 (socket says 172.21.15.70:47766) 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 -- 172.21.15.70:0/3171057043 learned_addr learned my addr 172.21.15.70:0/3171057043 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 -- 172.21.15.70:0/3171057043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feecc15dfb0 msgr2=0x7feecc15caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feecc15dfb0 0x7feecc15caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 -- 172.21.15.70:0/3171057043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feecc0d3120 msgr2=0x7feecc0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feecc0d3120 0x7feecc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 -- 172.21.15.70:0/3171057043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc0ccb90 con 0x7feecc0cea80 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feecb7fe640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 0x7feecc153000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7feebc00d5f0 tx=0x7feebc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc0125f0 con 0x7feecc0cea80 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feebc019070 con 0x7feecc0cea80 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feecc0cad80 con 0x7feecc0cea80 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc0129b0 con 0x7feecc0cea80 2024-04-22T01:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feecc0cb340 con 0x7feecc0cea80 2024-04-22T01:16:34.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feecc068210 con 0x7feecc0cea80 2024-04-22T01:16:34.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.035+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feebc012b50 con 0x7feecc0cea80 2024-04-22T01:16:34.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.039+0000 7feec8ff9640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7feeac0678c0 0x7feeac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.039+0000 7feec2ffd640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7feeac0678c0 0x7feeac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.039+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7feebc085f60 con 0x7feecc0cea80 2024-04-22T01:16:34.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.039+0000 7feec2ffd640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7feeac0678c0 0x7feeac069d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7feecc15ff60 tx=0x7feeb4006bd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.043+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7feebc056c10 con 0x7feecc0cea80 2024-04-22T01:16:34.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.203+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feecc0d1950 con 0x7feecc0cea80 2024-04-22T01:16:34.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feec8ff9640 1 -- 172.21.15.70:0/3171057043 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+13257 (secure 0 0 0) 0x7feebc056510 con 0x7feecc0cea80 2024-04-22T01:16:34.206 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:34.206 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":43,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:32.868555+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:6837/3248824947":"2024-04-23T01:16:29.853239+0000","172.21.15.70:6836/3248824947":"2024-04-23T01:16:29.853239+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7feeac0678c0 msgr2=0x7feeac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7feeac0678c0 0x7feeac069d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7feecc15ff60 tx=0x7feeb4006bd0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 msgr2=0x7feecc153000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 0x7feecc153000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7feebc00d5f0 tx=0x7feebc00dab0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 shutdown_connections 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7feeac0678c0 0x7feeac069d80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feecc15dfb0 0x7feecc15caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feecc0d3120 0x7feecc0cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 --2- 172.21.15.70:0/3171057043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feecc0cea80 0x7feecc153000 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 >> 172.21.15.70:0/3171057043 conn(0x7feecc06c9b0 msgr2=0x7feecc0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:34.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 shutdown_connections 2024-04-22T01:16:34.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.207+0000 7feed1f45640 1 -- 172.21.15.70:0/3171057043 wait complete. 2024-04-22T01:16:34.225 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:34.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.351+0000 7fc8211dc640 1 Processor -- start 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.351+0000 7fc8211dc640 1 -- start start 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc8211dc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c162350 0x7fc81c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc8211dc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc81c06de30 0x7fc81c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc8211dc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c06e340 con 0x7fc81c06f3e0 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc8211dc640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c06e620 con 0x7fc81c06de30 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc8211dc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c06e900 con 0x7fc81c162350 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc81c06de30 0x7fc81c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc81c06de30 0x7fc81c071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47776/0 (socket says 172.21.15.70:47776) 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 -- 172.21.15.70:0/953672466 learned_addr learned my addr 172.21.15.70:0/953672466 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81ad9d640 1 --2- 172.21.15.70:0/953672466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c162350 0x7fc81c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 -- 172.21.15.70:0/953672466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c162350 msgr2=0x7fc81c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 --2- 172.21.15.70:0/953672466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c162350 0x7fc81c06d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 -- 172.21.15.70:0/953672466 >> v1:172.21.15.98:6789/0 conn(0x7fc81c06f3e0 legacy=0x7fc81c165d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 -- 172.21.15.70:0/953672466 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c06ebe0 con 0x7fc81c06de30 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc81a59c640 1 --2- 172.21.15.70:0/953672466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06de30 0x7fc81c071b70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc810009870 tx=0x7fc81001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e108a1c279dea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc819d9b640 1 -- 172.21.15.70:0/953672466 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100b1070 con 0x7fc81c06de30 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.355+0000 7fc819d9b640 1 -- 172.21.15.70:0/953672466 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8100ac620 con 0x7fc81c06de30 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc819d9b640 1 -- 172.21.15.70:0/953672466 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100ac9f0 con 0x7fc81c06de30 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/953672466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06de30 msgr2=0x7fc81c071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 --2- 172.21.15.70:0/953672466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06de30 0x7fc81c071b70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc810009870 tx=0x7fc81001c4f0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/953672466 shutdown_connections 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 --2- 172.21.15.70:0/953672466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06de30 0x7fc81c071b70 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 --2- 172.21.15.70:0/953672466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c162350 0x7fc81c06d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/953672466 >> 172.21.15.70:0/953672466 conn(0x7fc81c0b1d40 msgr2=0x7fc81c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/953672466 shutdown_connections 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/953672466 wait complete. 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 Processor -- start 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- start start 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c06de30 0x7fc81c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 0x7fc81c14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc81c162350 0x7fc81c1576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c16f4c0 con 0x7fc81c06f3e0 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c16f340 con 0x7fc81c162350 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c16f640 con 0x7fc81c06de30 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 0x7fc81c14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 0x7fc81c14fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47780/0 (socket says 172.21.15.70:47780) 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 -- 172.21.15.70:0/873660647 learned_addr learned my addr 172.21.15.70:0/873660647 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81b59e640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc81c162350 0x7fc81c1576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81ad9d640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c06de30 0x7fc81c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 -- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c06de30 msgr2=0x7fc81c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c06de30 0x7fc81c155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 -- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc81c162350 msgr2=0x7fc81c1576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc81c162350 0x7fc81c1576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 -- 172.21.15.70:0/873660647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c1563d0 con 0x7fc81c06f3e0 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81b59e640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc81c162350 0x7fc81c1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc81a59c640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 0x7fc81c14fcc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc81001cd90 tx=0x7fc810019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100ba070 con 0x7fc81c06f3e0 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8100b5620 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c156650 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100b1040 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.359+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c163a60 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.363+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81c068210 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.363+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8100bd070 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.363+0000 7fc8037fe640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8080678c0 0x7fc808069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.363+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7fc81011ee90 con 0x7fc81c06f3e0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.363+0000 7fc81ad9d640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8080678c0 0x7fc808069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.363+0000 7fc81ad9d640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8080678c0 0x7fc808069d80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc80c00b7c0 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.367+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc81c1563d0 con 0x7fc81c06f3e0 2024-04-22T01:16:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc81c161ae0 con 0x7fc81c06f3e0 2024-04-22T01:16:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8037fe640 1 -- 172.21.15.70:0/873660647 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7fc810016110 con 0x7fc81c06f3e0 2024-04-22T01:16:34.526 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8080678c0 msgr2=0x7fc808069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8080678c0 0x7fc808069d80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc80c00b7c0 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 msgr2=0x7fc81c14fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 0x7fc81c14fcc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc81001cd90 tx=0x7fc810019290 comp rx=0 tx=0).stop 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 shutdown_connections 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc8080678c0 0x7fc808069d80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc81c162350 0x7fc81c1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc81c06f3e0 0x7fc81c14fcc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 --2- 172.21.15.70:0/873660647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc81c06de30 0x7fc81c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 >> 172.21.15.70:0/873660647 conn(0x7fc81c0b1d40 msgr2=0x7fc81c0b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 shutdown_connections 2024-04-22T01:16:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.527+0000 7fc8211dc640 1 -- 172.21.15.70:0/873660647 wait complete. 2024-04-22T01:16:34.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values' 2024-04-22T01:16:34.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.667+0000 7f4364853640 1 Processor -- start 2024-04-22T01:16:34.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.667+0000 7f4364853640 1 -- start start 2024-04-22T01:16:34.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f4364853640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f436015e130 0x7f436009ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f4364853640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f436009a3d0 0x7f43600cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f4364853640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436015e510 con 0x7f43600b1140 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f4364853640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f436006cdd0 con 0x7f436009a3d0 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f4364853640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f436006c4a0 con 0x7f436015e130 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f436009a3d0 0x7f43600cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f436009a3d0 0x7f43600cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47796/0 (socket says 172.21.15.70:47796) 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 -- 172.21.15.70:0/1795602854 learned_addr learned my addr 172.21.15.70:0/1795602854 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435e59c640 1 --2- 172.21.15.70:0/1795602854 >> v2:172.21.15.98:3300/0 conn(0x7f436015e130 0x7f436009ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 -- 172.21.15.70:0/1795602854 >> v2:172.21.15.98:3300/0 conn(0x7f436015e130 msgr2=0x7f436009ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 --2- 172.21.15.70:0/1795602854 >> v2:172.21.15.98:3300/0 conn(0x7f436015e130 0x7f436009ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 -- 172.21.15.70:0/1795602854 >> v1:172.21.15.98:6789/0 conn(0x7f43600b1140 legacy=0x7f4360099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:34.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 -- 172.21.15.70:0/1795602854 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436006c780 con 0x7f436009a3d0 2024-04-22T01:16:34.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435ed9d640 1 --2- 172.21.15.70:0/1795602854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f436009a3d0 0x7f43600cc830 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f43600b46d0 tx=0x7f435801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4695925ef29b26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435d59a640 1 -- 172.21.15.70:0/1795602854 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43580a57d0 con 0x7f436009a3d0 2024-04-22T01:16:34.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435d59a640 1 -- 172.21.15.70:0/1795602854 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43580a5970 con 0x7f436009a3d0 2024-04-22T01:16:34.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.671+0000 7f435d59a640 1 -- 172.21.15.70:0/1795602854 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43580a5d40 con 0x7f436009a3d0 2024-04-22T01:16:34.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/1795602854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f436009a3d0 msgr2=0x7f43600cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 --2- 172.21.15.70:0/1795602854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f436009a3d0 0x7f43600cc830 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f43600b46d0 tx=0x7f435801c8f0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/1795602854 shutdown_connections 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 --2- 172.21.15.70:0/1795602854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f436009a3d0 0x7f43600cc830 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 --2- 172.21.15.70:0/1795602854 >> v2:172.21.15.98:3300/0 conn(0x7f436015e130 0x7f436009ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/1795602854 >> 172.21.15.70:0/1795602854 conn(0x7f43600b35d0 msgr2=0x7f43600b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/1795602854 shutdown_connections 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/1795602854 wait complete. 2024-04-22T01:16:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 Processor -- start 2024-04-22T01:16:34.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- start start 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436009a3d0 0x7f4360153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 0x7f436015c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f436015e130 0x7f43601f8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f436016b010 con 0x7f43600b1140 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f436016ae90 con 0x7f436009a3d0 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f436016b190 con 0x7f436015e130 2024-04-22T01:16:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 0x7f436015c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 0x7f436015c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47800/0 (socket says 172.21.15.70:47800) 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 -- 172.21.15.70:0/3263703912 learned_addr learned my addr 172.21.15.70:0/3263703912 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435ed9d640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f436015e130 0x7f43601f8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 -- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f436015e130 msgr2=0x7f43601f8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f436015e130 0x7f43601f8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 -- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436009a3d0 msgr2=0x7f4360153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435e59c640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436009a3d0 0x7f4360153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436009a3d0 0x7f4360153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 -- 172.21.15.70:0/3263703912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436015b3f0 con 0x7f43600b1140 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f435dd9b640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 0x7f436015c490 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f435000c450 tx=0x7f435000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350012830 con 0x7f43600b1140 2024-04-22T01:16:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4350017070 con 0x7f43600b1140 2024-04-22T01:16:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436015b6d0 con 0x7f43600b1140 2024-04-22T01:16:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350012c30 con 0x7f43600b1140 2024-04-22T01:16:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436015bf70 con 0x7f43600b1140 2024-04-22T01:16:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.675+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436015b8a0 con 0x7f43600b1140 2024-04-22T01:16:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.679+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f435001c460 con 0x7f43600b1140 2024-04-22T01:16:34.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.679+0000 7f434effd640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f43400678c0 0x7f4340069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:34.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.679+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f4350085950 con 0x7f43600b1140 2024-04-22T01:16:34.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.679+0000 7f435e59c640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f43400678c0 0x7f4340069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:34.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.679+0000 7f435e59c640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f43400678c0 0x7f4340069d80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f435400b800 tx=0x7f4354008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.683+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4350056600 con 0x7f43600b1140 2024-04-22T01:16:34.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:34.903+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]} v 0) -- 0x7f4360068210 con 0x7f43600b1140 2024-04-22T01:16:35.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.887+0000 7f434effd640 1 -- 172.21.15.70:0/3263703912 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]}]=0 v123) ==== 156+0+0 (secure 0 0 0) 0x7f4350055f00 con 0x7f43600b1140 2024-04-22T01:16:35.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f43400678c0 msgr2=0x7f4340069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f43400678c0 0x7f4340069d80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f435400b800 tx=0x7f4354008040 comp rx=0 tx=0).stop 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 msgr2=0x7f436015c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 0x7f436015c490 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f435000c450 tx=0x7f435000c910 comp rx=0 tx=0).stop 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 shutdown_connections 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f43400678c0 0x7f4340069d80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f436015e130 0x7f43601f8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43600b1140 0x7f436015c490 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 --2- 172.21.15.70:0/3263703912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436009a3d0 0x7f4360153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 >> 172.21.15.70:0/3263703912 conn(0x7f43600b35d0 msgr2=0x7f4360099cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 shutdown_connections 2024-04-22T01:16:35.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:35.895+0000 7f4364853640 1 -- 172.21.15.70:0/3263703912 wait complete. 2024-04-22T01:16:35.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-22T01:16:36.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 Processor -- start 2024-04-22T01:16:36.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 -- start start 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17f00cca20 0x7f17f0161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 0x7f17f00cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f0162040 con 0x7f17f00cd090 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f17f00b02b0 con 0x7f17f00cca20 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f71b9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17f0162320 con 0x7f17f006b900 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f17f00cd090 0x7f17f006b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34040/0 (socket says 172.21.15.70:34040) 2024-04-22T01:16:36.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 -- 172.21.15.70:0/3235840951 learned_addr learned my addr 172.21.15.70:0/3235840951 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f5756640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 0x7f17f00cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563445105 0 0) 0x7f17f0162040 con 0x7f17f00cd090 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f4f55640 1 --2- 172.21.15.70:0/3235840951 >> v2:172.21.15.98:3300/0 conn(0x7f17f00cca20 0x7f17f0161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d8003530 con 0x7f17f00cd090 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 331711781 0 0) 0x7f17d8003530 con 0x7f17f00cd090 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 msgr2=0x7f17f00cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 0x7f17f00cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 >> v2:172.21.15.98:3300/0 conn(0x7f17f00cca20 msgr2=0x7f17f0161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 --2- 172.21.15.70:0/3235840951 >> v2:172.21.15.98:3300/0 conn(0x7f17f00cca20 0x7f17f0161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f0162600 con 0x7f17f00cd090 2024-04-22T01:16:36.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f17e4001680 con 0x7f17f00cd090 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 >> v1:172.21.15.70:6789/0 conn(0x7f17f00cd090 legacy=0x7f17f006b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d80039c0 0x7f17d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d8004810 0x7f17d8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 0x7f17d80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0162320 con 0x7f17d8004810 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17d800b9d0 con 0x7f17d80039c0 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17d800bb50 con 0x7f17f006b900 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d8004810 0x7f17d8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f4f55640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d80039c0 0x7f17d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f5756640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 0x7f17d80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 -- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 msgr2=0x7f17d80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f006b900 0x7f17d80051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 -- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d80039c0 msgr2=0x7f17d8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:36.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d80039c0 0x7f17d8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17effff640 1 -- 172.21.15.70:0/3235840951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d8003530 con 0x7f17d8004810 2024-04-22T01:16:36.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.043+0000 7f17f4f55640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d80039c0 0x7f17d8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17effff640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d8004810 0x7f17d8004c90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f17e400c2c0 tx=0x7f17e401f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c89ef9a15b6cd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4007d60 con 0x7f17d8004810 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e4005b00 con 0x7f17d8004810 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17ef7fe640 1 -- 172.21.15.70:0/3235840951 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4007070 con 0x7f17d8004810 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 -- 172.21.15.70:0/3235840951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d8004810 msgr2=0x7f17d8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 --2- 172.21.15.70:0/3235840951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d8004810 0x7f17d8004c90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f17e400c2c0 tx=0x7f17e401f4b0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f5756640 1 -- 172.21.15.70:0/3235840951 reap_dead start 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 -- 172.21.15.70:0/3235840951 shutdown_connections 2024-04-22T01:16:36.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 -- 172.21.15.70:0/3235840951 >> 172.21.15.70:0/3235840951 conn(0x7f17f0099a60 msgr2=0x7f17f0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 -- 172.21.15.70:0/3235840951 shutdown_connections 2024-04-22T01:16:36.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 -- 172.21.15.70:0/3235840951 wait complete. 2024-04-22T01:16:36.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 Processor -- start 2024-04-22T01:16:36.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.047+0000 7f17f71b9640 1 -- start start 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d8004810 0x7f17f00c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f01f0d90 0x7f17f015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 0x7f17f01f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17f01628e0 con 0x7f17f015c710 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17f00cce00 con 0x7f17d8004810 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17f00cfd50 con 0x7f17f01f0d90 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 0x7f17f01f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 0x7f17f01f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47824/0 (socket says 172.21.15.70:47824) 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 -- 172.21.15.70:0/4236626621 learned_addr learned my addr 172.21.15.70:0/4236626621 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f4f55640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d8004810 0x7f17f00c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17effff640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f01f0d90 0x7f17f015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 -- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f01f0d90 msgr2=0x7f17f015c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f01f0d90 0x7f17f015c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 -- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d8004810 msgr2=0x7f17f00c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d8004810 0x7f17f00c6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 -- 172.21.15.70:0/4236626621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f01f4830 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f5756640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 0x7f17f01f3ac0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f17f00b6a30 tx=0x7f17e800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e8017400 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f00c5470 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80177f0 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f00c5a30 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f0068920 con 0x7f17f015c710 2024-04-22T01:16:36.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f17e801f070 con 0x7f17f015c710 2024-04-22T01:16:36.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17ed7fa640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17dc0678f0 0x7f17dc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f17e8085ad0 con 0x7f17f015c710 2024-04-22T01:16:36.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f4f55640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17dc0678f0 0x7f17dc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.051+0000 7f17f4f55640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17dc0678f0 0x7f17dc069db0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009a60 tx=0x7f17e0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.055+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f17e8012070 con 0x7f17f015c710 2024-04-22T01:16:36.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.219+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f17f00cd6a0 con 0x7f17f015c710 2024-04-22T01:16:36.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17ed7fa640 1 -- 172.21.15.70:0/4236626621 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f17e8056130 con 0x7f17f015c710 2024-04-22T01:16:36.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17dc0678f0 msgr2=0x7f17dc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17dc0678f0 0x7f17dc069db0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009a60 tx=0x7f17e0008040 comp rx=0 tx=0).stop 2024-04-22T01:16:36.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 msgr2=0x7f17f01f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 0x7f17f01f3ac0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f17f00b6a30 tx=0x7f17e800f4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 shutdown_connections 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17dc0678f0 0x7f17dc069db0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.223+0000 7f17f71b9640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f015c710 0x7f17f01f3ac0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.227+0000 7f17f71b9640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f01f0d90 0x7f17f015c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.227+0000 7f17f71b9640 1 --2- 172.21.15.70:0/4236626621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d8004810 0x7f17f00c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.227+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 >> 172.21.15.70:0/4236626621 conn(0x7f17f0099a60 msgr2=0x7f17f00ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.227+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 shutdown_connections 2024-04-22T01:16:36.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.227+0000 7f17f71b9640 1 -- 172.21.15.70:0/4236626621 wait complete. 2024-04-22T01:16:36.242 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:16:36.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:16:36.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:16:36.343 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:16:36.344 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:16:36.344 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:36.348 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:16:36.348 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:16:36.348 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:36.354 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:16:36.354 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:16:36.354 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:36.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:16:36.393 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:16:36.393 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:36.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:16:36.402 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-22T01:16:36.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:36.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 Processor -- start 2024-04-22T01:16:36.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 -- start start 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f976c0aee40 0x7f976c0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976c06dbd0 con 0x7f976c15bb50 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f976c06de70 con 0x7f976c0aee40 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f97728be640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f976c06e150 con 0x7f976c15aa30 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9770e5b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f976c0aee40 0x7f976c0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9763fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f976c15bb50 0x7f976c0ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35944/0 (socket says 172.21.15.70:35944) 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9763fff640 1 -- 172.21.15.70:0/2796893956 learned_addr learned my addr 172.21.15.70:0/2796893956 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:36.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f976bfff640 1 --2- 172.21.15.70:0/2796893956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9770e5b640 1 -- 172.21.15.70:0/2796893956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 msgr2=0x7f976c0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:36.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9770e5b640 1 --2- 172.21.15.70:0/2796893956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9770e5b640 1 -- 172.21.15.70:0/2796893956 >> v1:172.21.15.70:6789/0 conn(0x7f976c15bb50 legacy=0x7f976c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f976b7fe640 1 -- 172.21.15.70:0/2796893956 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161449104 0 0) 0x7f976c06dbd0 con 0x7f976c15bb50 2024-04-22T01:16:36.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.563+0000 7f9770e5b640 1 -- 172.21.15.70:0/2796893956 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c06e430 con 0x7f976c0aee40 2024-04-22T01:16:36.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f9770e5b640 1 --2- 172.21.15.70:0/2796893956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c0b28f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9764009d90 tx=0x7f976401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd240e0b9f9f02ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976b7fe640 1 -- 172.21.15.70:0/2796893956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97640b2070 con 0x7f976c0aee40 2024-04-22T01:16:36.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976b7fe640 1 -- 172.21.15.70:0/2796893956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97640a5a10 con 0x7f976c0aee40 2024-04-22T01:16:36.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976b7fe640 1 -- 172.21.15.70:0/2796893956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97640a5de0 con 0x7f976c0aee40 2024-04-22T01:16:36.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/2796893956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 msgr2=0x7f976c0b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 --2- 172.21.15.70:0/2796893956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c0b28f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9764009d90 tx=0x7f976401c970 comp rx=0 tx=0).stop 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/2796893956 shutdown_connections 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 --2- 172.21.15.70:0/2796893956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c0b28f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 --2- 172.21.15.70:0/2796893956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/2796893956 >> 172.21.15.70:0/2796893956 conn(0x7f976c06b1f0 msgr2=0x7f976c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/2796893956 shutdown_connections 2024-04-22T01:16:36.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/2796893956 wait complete. 2024-04-22T01:16:36.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 Processor -- start 2024-04-22T01:16:36.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- start start 2024-04-22T01:16:36.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c15bb50 0x7f976c1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f976c06ead0 con 0x7f976c0aee40 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f976c06e950 con 0x7f976c15bb50 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f976c167a90 con 0x7f976c15aa30 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48892/0 (socket says 172.21.15.70:48892) 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 -- 172.21.15.70:0/3129017997 learned_addr learned my addr 172.21.15.70:0/3129017997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 -- 172.21.15.70:0/3129017997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 msgr2=0x7f976c0b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 -- 172.21.15.70:0/3129017997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c15bb50 msgr2=0x7f976c1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c15bb50 0x7f976c1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 -- 172.21.15.70:0/3129017997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c152820 con 0x7f976c0aee40 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f976bfff640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c152160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f975c00c820 tx=0x7f975c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c015070 con 0x7f976c0aee40 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975c004240 con 0x7f976c0aee40 2024-04-22T01:16:36.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976c152b00 con 0x7f976c0aee40 2024-04-22T01:16:36.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c00ce10 con 0x7f976c0aee40 2024-04-22T01:16:36.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.567+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976c1530c0 con 0x7f976c0aee40 2024-04-22T01:16:36.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.571+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976c1533f0 con 0x7f976c0aee40 2024-04-22T01:16:36.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.571+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f975c007830 con 0x7f976c0aee40 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.571+0000 7f97697fa640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.571+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f975c085980 con 0x7f976c0aee40 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.571+0000 7f9763fff640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.571+0000 7f9763fff640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f976c0aea20 tx=0x7f9754007970 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.575+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f975c0566c0 con 0x7f976c0aee40 2024-04-22T01:16:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f976c068210 con 0x7f976c0aee40 2024-04-22T01:16:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97697fa640 1 -- 172.21.15.70:0/3129017997 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3863 (secure 0 0 0) 0x7f975c055fc0 con 0x7f976c0aee40 2024-04-22T01:16:36.767 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:36.767 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/725120472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":725120472},{"type":"v1","addr":"172.21.15.98:6833","nonce":725120472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1099640161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1099640161},{"type":"v1","addr":"172.21.15.98:6835","nonce":1099640161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2274760335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2274760335},{"type":"v1","addr":"172.21.15.70:6835","nonce":2274760335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5835,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/668931238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":668931238},{"type":"v1","addr":"172.21.15.70:6837","nonce":668931238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-04-22T01:16:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 19 2024-04-22T01:16:36.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 msgr2=0x7f974c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f976c0aea20 tx=0x7f9754007970 comp rx=0 tx=0).stop 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 msgr2=0x7f976c152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c152160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f975c00c820 tx=0x7f975c00cce0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 shutdown_connections 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f974c0678c0 0x7f974c069d80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c15bb50 0x7f976c1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f976c15aa30 0x7f976c0b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 --2- 172.21.15.70:0/3129017997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0aee40 0x7f976c152160 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 >> 172.21.15.70:0/3129017997 conn(0x7f976c06b1f0 msgr2=0x7f976c0cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 shutdown_connections 2024-04-22T01:16:36.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.767+0000 7f97728be640 1 -- 172.21.15.70:0/3129017997 wait complete. 2024-04-22T01:16:36.784 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:16:36.785 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-22T01:16:36.785 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:16:36.785 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:36.785 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:16:36.785 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-22T01:16:36.785 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:16:36.785 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:36.786 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:16:36.786 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-22T01:16:36.786 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:16:36.786 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:36.786 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:16:36.786 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-22T01:16:36.787 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:16:36.787 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:16:36.797 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:16:36.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:16:36.798 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:16:36.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:16:36.800 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:16:36.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:16:36.800 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:16:36.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:16:36.843 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:16:36.843 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:36.843 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:16:36.843 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:36.848 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:16:36.848 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:36.849 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:16:36.849 DEBUG:teuthology.parallel:result is None 2024-04-22T01:16:36.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-22T01:16:36.903 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:16:36.903 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:16:36.916 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:16:36.917 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:16:36.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 Processor -- start 2024-04-22T01:16:36.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 -- start start 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff3a01615a0 0x7ff3a00b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a00cf3e0 0x7ff3a0162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a006b450 con 0x7ff3a0161b90 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff3a0162550 con 0x7ff3a01615a0 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a6b81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a0161980 con 0x7ff3a00cf3e0 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff3a01615a0 0x7ff3a00b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff3a01615a0 0x7ff3a00b70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48930/0 (socket says 172.21.15.70:48930) 2024-04-22T01:16:36.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 -- 172.21.15.70:0/620830610 learned_addr learned my addr 172.21.15.70:0/620830610 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:36.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a511e640 1 --2- 172.21.15.70:0/620830610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a00cf3e0 0x7ff3a0162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 -- 172.21.15.70:0/620830610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a00cf3e0 msgr2=0x7ff3a0162010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:36.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 --2- 172.21.15.70:0/620830610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a00cf3e0 0x7ff3a0162010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 -- 172.21.15.70:0/620830610 >> v1:172.21.15.98:6789/0 conn(0x7ff3a0161b90 legacy=0x7ff3a00cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.963+0000 7ff3a491d640 1 -- 172.21.15.70:0/620830610 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a00ce830 con 0x7ff3a01615a0 2024-04-22T01:16:36.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a491d640 1 --2- 172.21.15.70:0/620830610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b70c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff390009870 tx=0x7ff39001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb9046650bd2e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff39f7fe640 1 -- 172.21.15.70:0/620830610 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3900ab650 con 0x7ff3a01615a0 2024-04-22T01:16:36.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff39f7fe640 1 -- 172.21.15.70:0/620830610 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3900b0070 con 0x7ff3a01615a0 2024-04-22T01:16:36.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff39f7fe640 1 -- 172.21.15.70:0/620830610 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3900aba10 con 0x7ff3a01615a0 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 -- 172.21.15.70:0/620830610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 msgr2=0x7ff3a00b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/620830610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b70c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff390009870 tx=0x7ff39001c560 comp rx=0 tx=0).stop 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 -- 172.21.15.70:0/620830610 shutdown_connections 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/620830610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a00cf3e0 0x7ff3a0162010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/620830610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b70c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 -- 172.21.15.70:0/620830610 >> 172.21.15.70:0/620830610 conn(0x7ff3a006c910 msgr2=0x7ff3a006ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 -- 172.21.15.70:0/620830610 shutdown_connections 2024-04-22T01:16:36.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 -- 172.21.15.70:0/620830610 wait complete. 2024-04-22T01:16:36.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 Processor -- start 2024-04-22T01:16:36.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.967+0000 7ff3a6b81640 1 -- start start 2024-04-22T01:16:36.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3a00cf3e0 0x7ff3a0159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a0161b90 0x7ff3a01587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a00b7600 con 0x7ff3a01615a0 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a00d2130 con 0x7ff3a00cf3e0 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a00b7780 con 0x7ff3a0161b90 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48932/0 (socket says 172.21.15.70:48932) 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 -- 172.21.15.70:0/842479591 learned_addr learned my addr 172.21.15.70:0/842479591 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a491d640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3a00cf3e0 0x7ff3a0159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a511e640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a0161b90 0x7ff3a01587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 -- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a0161b90 msgr2=0x7ff3a01587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a0161b90 0x7ff3a01587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 -- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3a00cf3e0 msgr2=0x7ff3a0159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3a00cf3e0 0x7ff3a0159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 -- 172.21.15.70:0/842479591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a0158d30 con 0x7ff3a01615a0 2024-04-22T01:16:36.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39ffff640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b3710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff39400c450 tx=0x7ff39400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394016020 con 0x7ff3a01615a0 2024-04-22T01:16:36.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3940052d0 con 0x7ff3a01615a0 2024-04-22T01:16:36.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a015a090 con 0x7ff3a01615a0 2024-04-22T01:16:36.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394007260 con 0x7ff3a01615a0 2024-04-22T01:16:36.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a00b7e30 con 0x7ff3a01615a0 2024-04-22T01:16:36.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a004f510 con 0x7ff3a01615a0 2024-04-22T01:16:36.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff394007400 con 0x7ff3a01615a0 2024-04-22T01:16:36.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39d7fa640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff38c0678c0 0x7ff38c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7ff394085310 con 0x7ff3a01615a0 2024-04-22T01:16:36.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a491d640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff38c0678c0 0x7ff38c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.971+0000 7ff3a491d640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff38c0678c0 0x7ff38c069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff3900af000 tx=0x7ff3900a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:36.975+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff394020020 con 0x7ff3a01615a0 2024-04-22T01:16:37.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:37.147+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff3a00d1b90 con 0x7ff3a01615a0 2024-04-22T01:16:38.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.247+0000 7ff39d7fa640 1 -- 172.21.15.70:0/842479591 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v48) ==== 97+0+0 (secure 0 0 0) 0x7ff394005480 con 0x7ff3a01615a0 2024-04-22T01:16:38.247 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-22T01:16:38.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff38c0678c0 msgr2=0x7ff38c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff38c0678c0 0x7ff38c069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff3900af000 tx=0x7ff3900a7000 comp rx=0 tx=0).stop 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 msgr2=0x7ff3a00b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b3710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff39400c450 tx=0x7ff39400c910 comp rx=0 tx=0).stop 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 shutdown_connections 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff38c0678c0 0x7ff38c069d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3a0161b90 0x7ff3a01587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3a01615a0 0x7ff3a00b3710 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 --2- 172.21.15.70:0/842479591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3a00cf3e0 0x7ff3a0159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 >> 172.21.15.70:0/842479591 conn(0x7ff3a006c910 msgr2=0x7ff3a00cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 shutdown_connections 2024-04-22T01:16:38.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.251+0000 7ff3a6b81640 1 -- 172.21.15.70:0/842479591 wait complete. 2024-04-22T01:16:38.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-22T01:16:38.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 Processor -- start 2024-04-22T01:16:38.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 -- start start 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55415be20 0x7fe5540cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe55409a360 0x7fe5540c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5540ce080 con 0x7fe5540d0570 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe55415c200 con 0x7fe55409a360 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe55b9d5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe5540ccaa0 con 0x7fe55415be20 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559f72640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe55409a360 0x7fe5540c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe558f70640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe5540d0570 0x7fe554099c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35970/0 (socket says 172.21.15.70:35970) 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559f72640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe55409a360 0x7fe5540c5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48938/0 (socket says 172.21.15.70:48938) 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe558f70640 1 -- 172.21.15.70:0/4242097365 learned_addr learned my addr 172.21.15.70:0/4242097365 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559771640 1 --2- 172.21.15.70:0/4242097365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55415be20 0x7fe5540cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559f72640 1 -- 172.21.15.70:0/4242097365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55415be20 msgr2=0x7fe5540cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:38.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559f72640 1 --2- 172.21.15.70:0/4242097365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55415be20 0x7fe5540cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559f72640 1 -- 172.21.15.70:0/4242097365 >> v1:172.21.15.70:6789/0 conn(0x7fe5540d0570 legacy=0x7fe554099c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe553fff640 1 -- 172.21.15.70:0/4242097365 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840023421 0 0) 0x7fe5540ce080 con 0x7fe5540d0570 2024-04-22T01:16:38.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.395+0000 7fe559f72640 1 -- 172.21.15.70:0/4242097365 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5540ccd80 con 0x7fe55409a360 2024-04-22T01:16:38.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe559f72640 1 --2- 172.21.15.70:0/4242097365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe55409a360 0x7fe5540c5600 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0097c0 tx=0x7fe54c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62245bfbf99637c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe553fff640 1 -- 172.21.15.70:0/4242097365 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c0ab430 con 0x7fe55409a360 2024-04-22T01:16:38.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe553fff640 1 -- 172.21.15.70:0/4242097365 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe54c0b0070 con 0x7fe55409a360 2024-04-22T01:16:38.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe553fff640 1 -- 172.21.15.70:0/4242097365 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c0ab7f0 con 0x7fe55409a360 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- 172.21.15.70:0/4242097365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe55409a360 msgr2=0x7fe5540c5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/4242097365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe55409a360 0x7fe5540c5600 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0097c0 tx=0x7fe54c01c470 comp rx=0 tx=0).stop 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- 172.21.15.70:0/4242097365 shutdown_connections 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/4242097365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe55409a360 0x7fe5540c5600 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/4242097365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55415be20 0x7fe5540cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- 172.21.15.70:0/4242097365 >> 172.21.15.70:0/4242097365 conn(0x7fe5540b35a0 msgr2=0x7fe5540b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- 172.21.15.70:0/4242097365 shutdown_connections 2024-04-22T01:16:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- 172.21.15.70:0/4242097365 wait complete. 2024-04-22T01:16:38.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 Processor -- start 2024-04-22T01:16:38.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- start start 2024-04-22T01:16:38.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55409a360 0x7fe554152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 0x7fe5540c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe55415be20 0x7fe5540c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5540cf9c0 con 0x7fe5540d0570 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5540cf840 con 0x7fe55415be20 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe55b9d5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe5540d0990 con 0x7fe55409a360 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.399+0000 7fe558f70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 0x7fe5540c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 0x7fe5540c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48946/0 (socket says 172.21.15.70:48946) 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 -- 172.21.15.70:0/469620083 learned_addr learned my addr 172.21.15.70:0/469620083 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe559771640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55409a360 0x7fe554152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe559f72640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe55415be20 0x7fe5540c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 -- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55409a360 msgr2=0x7fe554152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55409a360 0x7fe554152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 -- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe55415be20 msgr2=0x7fe5540c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe55415be20 0x7fe5540c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 -- 172.21.15.70:0/469620083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5540c9c50 con 0x7fe5540d0570 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe559f72640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe55415be20 0x7fe5540c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe558f70640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 0x7fe5540c7520 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5480011f0 tx=0x7fe54800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548018070 con 0x7fe5540d0570 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5480074e0 con 0x7fe5540d0570 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548004a10 con 0x7fe5540d0570 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5540c9f30 con 0x7fe5540d0570 2024-04-22T01:16:38.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5540ca4f0 con 0x7fe5540d0570 2024-04-22T01:16:38.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe554068210 con 0x7fe5540d0570 2024-04-22T01:16:38.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe548004bb0 con 0x7fe5540d0570 2024-04-22T01:16:38.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe551ffb640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5400678c0 0x7fe540069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe559771640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5400678c0 0x7fe540069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7fe548084de0 con 0x7fe5540d0570 2024-04-22T01:16:38.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.403+0000 7fe559771640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5400678c0 0x7fe540069d80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5540cfd10 tx=0x7fe54400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.407+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe548055b70 con 0x7fe5540d0570 2024-04-22T01:16:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe55409a780 con 0x7fe5540d0570 2024-04-22T01:16:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe551ffb640 1 -- 172.21.15.70:0/469620083 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7fe548055470 con 0x7fe5540d0570 2024-04-22T01:16:38.618 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:38.618 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:16:38.618 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-22T01:16:38.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.618 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw==", 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.619 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w==", 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ==", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA==", 2024-04-22T01:16:38.621 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-22T01:16:38.622 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA==", 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-22T01:16:38.623 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA==", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg==", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.624 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg==", 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong==", 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.626 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ==", 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA==", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ==", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:16:38.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw==", 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.629 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow r", 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-22T01:16:38.630 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ==", 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA==", 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.634 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw==", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:16:38.635 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm2O5GBRAA/JxaogwD4Hjmtr9eQvMY0g==", 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.636 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw==", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg==", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:16:38.637 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:16:38.638 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.638 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:38.638 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-22T01:16:38.638 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5400678c0 msgr2=0x7fe540069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5400678c0 0x7fe540069d80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5540cfd10 tx=0x7fe54400b040 comp rx=0 tx=0).stop 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 msgr2=0x7fe5540c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 0x7fe5540c7520 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5480011f0 tx=0x7fe54800c9b0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 shutdown_connections 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5400678c0 0x7fe540069d80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe55415be20 0x7fe5540c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5540d0570 0x7fe5540c7520 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 --2- 172.21.15.70:0/469620083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe55409a360 0x7fe554152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 >> 172.21.15.70:0/469620083 conn(0x7fe5540b35a0 msgr2=0x7fe55415b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 shutdown_connections 2024-04-22T01:16:38.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.619+0000 7fe55b9d5640 1 -- 172.21.15.70:0/469620083 wait complete. 2024-04-22T01:16:38.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 Processor -- start 2024-04-22T01:16:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 -- start start 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2d340b7150 0x7f2d340b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340d03e0 0x7f2d340d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3406e340 con 0x7f2d3415df90 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2d340b7ad0 con 0x7f2d340b7150 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3bc1f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2d340d2610 con 0x7f2d340d03e0 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d399bb640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2d3415df90 0x7f2d340d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35986/0 (socket says 172.21.15.70:35986) 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d391ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2d340b7150 0x7f2d340b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d399bb640 1 -- 172.21.15.70:0/2222620956 learned_addr learned my addr 172.21.15.70:0/2222620956 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d3a1bc640 1 --2- 172.21.15.70:0/2222620956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340d03e0 0x7f2d340d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d391ba640 1 -- 172.21.15.70:0/2222620956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340d03e0 msgr2=0x7f2d340d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d391ba640 1 --2- 172.21.15.70:0/2222620956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340d03e0 0x7f2d340d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d391ba640 1 -- 172.21.15.70:0/2222620956 >> v1:172.21.15.70:6789/0 conn(0x7f2d3415df90 legacy=0x7f2d340d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d391ba640 1 -- 172.21.15.70:0/2222620956 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d340d0db0 con 0x7f2d340b7150 2024-04-22T01:16:38.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d389b9640 1 -- 172.21.15.70:0/2222620956 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972731965 0 0) 0x7f2d3406e340 con 0x7f2d3415df90 2024-04-22T01:16:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.763+0000 7f2d391ba640 1 --2- 172.21.15.70:0/2222620956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340b7150 0x7f2d340b7590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2d240097c0 tx=0x7f2d2401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7812a0563fe08d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d389b9640 1 -- 172.21.15.70:0/2222620956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240b0070 con 0x7f2d340b7150 2024-04-22T01:16:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d389b9640 1 -- 172.21.15.70:0/2222620956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d240ab600 con 0x7f2d340b7150 2024-04-22T01:16:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d389b9640 1 -- 172.21.15.70:0/2222620956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240ab9d0 con 0x7f2d340b7150 2024-04-22T01:16:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/2222620956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340b7150 msgr2=0x7f2d340b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/2222620956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340b7150 0x7f2d340b7590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2d240097c0 tx=0x7f2d2401c410 comp rx=0 tx=0).stop 2024-04-22T01:16:38.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/2222620956 shutdown_connections 2024-04-22T01:16:38.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/2222620956 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340d03e0 0x7f2d340d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/2222620956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340b7150 0x7f2d340b7590 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/2222620956 >> 172.21.15.70:0/2222620956 conn(0x7f2d3406c9f0 msgr2=0x7f2d3406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/2222620956 shutdown_connections 2024-04-22T01:16:38.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/2222620956 wait complete. 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 Processor -- start 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- start start 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340b7150 0x7f2d341522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 0x7f2d34154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d3415df90 0x7f2d340cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2d340d3330 con 0x7f2d340d03e0 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d340d31b0 con 0x7f2d3415df90 2024-04-22T01:16:38.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3415e370 con 0x7f2d340b7150 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 0x7f2d34154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 0x7f2d34154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48952/0 (socket says 172.21.15.70:48952) 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 -- 172.21.15.70:0/1024306848 learned_addr learned my addr 172.21.15.70:0/1024306848 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d399bb640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340b7150 0x7f2d341522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3a1bc640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d3415df90 0x7f2d340cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 -- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340b7150 msgr2=0x7f2d341522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340b7150 0x7f2d341522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 -- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d3415df90 msgr2=0x7f2d340cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d3415df90 0x7f2d340cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 -- 172.21.15.70:0/1024306848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d34152aa0 con 0x7f2d340d03e0 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d391ba640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 0x7f2d34154b90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2d2401a040 tx=0x7f2d240191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240b8030 con 0x7f2d340d03e0 2024-04-22T01:16:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d240ab400 con 0x7f2d340d03e0 2024-04-22T01:16:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d34155290 con 0x7f2d340d03e0 2024-04-22T01:16:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240b0040 con 0x7f2d340d03e0 2024-04-22T01:16:38.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.767+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d34153220 con 0x7f2d340d03e0 2024-04-22T01:16:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.771+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d34068920 con 0x7f2d340d03e0 2024-04-22T01:16:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.771+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d240bb070 con 0x7f2d340d03e0 2024-04-22T01:16:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.771+0000 7f2d2a7fc640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2d180678c0 0x7f2d18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.771+0000 7f2d399bb640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2d180678c0 0x7f2d18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.771+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f2d2411f170 con 0x7f2d340d03e0 2024-04-22T01:16:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.771+0000 7f2d399bb640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2d180678c0 0x7f2d18069d80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d30000f70 tx=0x7f2d30007b80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.775+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2d240c2070 con 0x7f2d340d03e0 2024-04-22T01:16:38.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.943+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d340d1c00 con 0x7f2d340d03e0 2024-04-22T01:16:38.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.943+0000 7f2d2a7fc640 1 -- 172.21.15.70:0/1024306848 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+13128 (secure 0 0 0) 0x7f2d24016090 con 0x7f2d340d03e0 2024-04-22T01:16:38.945 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:38.945 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":48,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:38.246091+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2d180678c0 msgr2=0x7f2d18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2d180678c0 0x7f2d18069d80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d30000f70 tx=0x7f2d30007b80 comp rx=0 tx=0).stop 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 msgr2=0x7f2d34154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 0x7f2d34154b90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2d2401a040 tx=0x7f2d240191b0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 shutdown_connections 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2d180678c0 0x7f2d18069d80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d3415df90 0x7f2d340cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d340d03e0 0x7f2d34154b90 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 --2- 172.21.15.70:0/1024306848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d340b7150 0x7f2d341522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 >> 172.21.15.70:0/1024306848 conn(0x7f2d3406c9f0 msgr2=0x7f2d340cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 shutdown_connections 2024-04-22T01:16:38.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:38.947+0000 7f2d3bc1f640 1 -- 172.21.15.70:0/1024306848 wait complete. 2024-04-22T01:16:38.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:38.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:39.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 Processor -- start 2024-04-22T01:16:39.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 -- start start 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200b4450 0x7ff120071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1200cd220 0x7ff1200b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12015c290 con 0x7ff1200720b0 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff120072510 con 0x7ff1200cd220 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.071+0000 7ff1257c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff12015c570 con 0x7ff1200b4450 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1200cd220 0x7ff1200b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1200cd220 0x7ff1200b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48964/0 (socket says 172.21.15.70:48964) 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 -- 172.21.15.70:0/3042970450 learned_addr learned my addr 172.21.15.70:0/3042970450 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11effd640 1 --2- 172.21.15.70:0/3042970450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200b4450 0x7ff120071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 -- 172.21.15.70:0/3042970450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200b4450 msgr2=0x7ff120071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:39.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/3042970450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200b4450 0x7ff120071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 -- 172.21.15.70:0/3042970450 >> v1:172.21.15.98:6789/0 conn(0x7ff1200720b0 legacy=0x7ff1200ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 -- 172.21.15.70:0/3042970450 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12015c850 con 0x7ff1200cd220 2024-04-22T01:16:39.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/3042970450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff1200b2a40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff118009e50 tx=0x7ff11801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8526370b8b391ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11e7fc640 1 -- 172.21.15.70:0/3042970450 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1180a5860 con 0x7ff1200cd220 2024-04-22T01:16:39.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11e7fc640 1 -- 172.21.15.70:0/3042970450 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1180a5a00 con 0x7ff1200cd220 2024-04-22T01:16:39.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.075+0000 7ff11e7fc640 1 -- 172.21.15.70:0/3042970450 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1180a5dd0 con 0x7ff1200cd220 2024-04-22T01:16:39.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/3042970450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 msgr2=0x7ff1200b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 --2- 172.21.15.70:0/3042970450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff1200b2a40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff118009e50 tx=0x7ff11801ca00 comp rx=0 tx=0).stop 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/3042970450 shutdown_connections 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 --2- 172.21.15.70:0/3042970450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff1200b2a40 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 --2- 172.21.15.70:0/3042970450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200b4450 0x7ff120071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/3042970450 >> 172.21.15.70:0/3042970450 conn(0x7ff1200b1d40 msgr2=0x7ff1200cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/3042970450 shutdown_connections 2024-04-22T01:16:39.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/3042970450 wait complete. 2024-04-22T01:16:39.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 Processor -- start 2024-04-22T01:16:39.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- start start 2024-04-22T01:16:39.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200720b0 0x7ff120144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1200b4450 0x7ff1201447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff120153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1200cd980 con 0x7ff1200cd220 2024-04-22T01:16:39.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1200cd800 con 0x7ff1200b4450 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff1200b3790 con 0x7ff1200720b0 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff120153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff120153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48976/0 (socket says 172.21.15.70:48976) 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 -- 172.21.15.70:0/1393909334 learned_addr learned my addr 172.21.15.70:0/1393909334 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff117fff640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1200b4450 0x7ff1201447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11effd640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200720b0 0x7ff120144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 -- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200720b0 msgr2=0x7ff120144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200720b0 0x7ff120144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 -- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1200b4450 msgr2=0x7ff1201447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1200b4450 0x7ff1201447f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 -- 172.21.15.70:0/1393909334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff120147fd0 con 0x7ff1200cd220 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11effd640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200720b0 0x7ff120144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:39.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff11f7fe640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff120153140 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff1180af040 tx=0x7ff1180197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1180b8670 con 0x7ff1200cd220 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1180be070 con 0x7ff1200cd220 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1201482b0 con 0x7ff1200cd220 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1180b8a90 con 0x7ff1200cd220 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff120148870 con 0x7ff1200cd220 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.079+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff120147d20 con 0x7ff1200cd220 2024-04-22T01:16:39.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.083+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff1180b8c30 con 0x7ff1200cd220 2024-04-22T01:16:39.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.083+0000 7ff1177fe640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.083+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7ff118120cd0 con 0x7ff1200cd220 2024-04-22T01:16:39.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.083+0000 7ff11effd640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.083+0000 7ff11effd640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff110009840 tx=0x7ff110008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.087+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff118016620 con 0x7ff1200cd220 2024-04-22T01:16:39.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff120068210 con 0x7ff1200cd220 2024-04-22T01:16:39.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1177fe640 1 -- 172.21.15.70:0/1393909334 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7ff1180f1a60 con 0x7ff1200cd220 2024-04-22T01:16:39.234 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 msgr2=0x7ff100069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff110009840 tx=0x7ff110008040 comp rx=0 tx=0).stop 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 msgr2=0x7ff120153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff120153140 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff1180af040 tx=0x7ff1180197a0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 shutdown_connections 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1000678c0 0x7ff100069d80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1200cd220 0x7ff120153140 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1200b4450 0x7ff1201447f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 --2- 172.21.15.70:0/1393909334 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1200720b0 0x7ff120144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 >> 172.21.15.70:0/1393909334 conn(0x7ff1200b1d40 msgr2=0x7ff1200744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 shutdown_connections 2024-04-22T01:16:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.235+0000 7ff1257c3640 1 -- 172.21.15.70:0/1393909334 wait complete. 2024-04-22T01:16:39.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:16:39.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.371+0000 7f33a6813640 1 Processor -- start 2024-04-22T01:16:39.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.371+0000 7f33a6813640 1 -- start start 2024-04-22T01:16:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a6813640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f33a015aa30 0x7f33a00b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a6813640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33a00aee40 0x7f33a00b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a6813640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a006dbd0 con 0x7f33a015bb50 2024-04-22T01:16:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a6813640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f33a006de70 con 0x7f33a00aee40 2024-04-22T01:16:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a6813640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f33a006e150 con 0x7f33a015aa30 2024-04-22T01:16:39.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33a00aee40 0x7f33a00b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33a00aee40 0x7f33a00b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48984/0 (socket says 172.21.15.70:48984) 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 -- 172.21.15.70:0/1328622482 learned_addr learned my addr 172.21.15.70:0/1328622482 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f339ffff640 1 --2- 172.21.15.70:0/1328622482 >> v2:172.21.15.98:3300/0 conn(0x7f33a015aa30 0x7f33a00b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 -- 172.21.15.70:0/1328622482 >> v2:172.21.15.98:3300/0 conn(0x7f33a015aa30 msgr2=0x7f33a00b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 --2- 172.21.15.70:0/1328622482 >> v2:172.21.15.98:3300/0 conn(0x7f33a015aa30 0x7f33a00b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 -- 172.21.15.70:0/1328622482 >> v1:172.21.15.98:6789/0 conn(0x7f33a015bb50 legacy=0x7f33a00ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 -- 172.21.15.70:0/1328622482 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a006e430 con 0x7f33a00aee40 2024-04-22T01:16:39.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.375+0000 7f33a4db0640 1 --2- 172.21.15.70:0/1328622482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b28f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f33a006ecd0 tx=0x7f339801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20be0e10b85ee74 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339effd640 1 -- 172.21.15.70:0/1328622482 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33980aa670 con 0x7f33a00aee40 2024-04-22T01:16:39.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339effd640 1 -- 172.21.15.70:0/1328622482 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33980af070 con 0x7f33a00aee40 2024-04-22T01:16:39.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339effd640 1 -- 172.21.15.70:0/1328622482 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33980aaa30 con 0x7f33a00aee40 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- 172.21.15.70:0/1328622482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 msgr2=0x7f33a00b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 --2- 172.21.15.70:0/1328622482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b28f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f33a006ecd0 tx=0x7f339801c4f0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- 172.21.15.70:0/1328622482 shutdown_connections 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 --2- 172.21.15.70:0/1328622482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b28f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 --2- 172.21.15.70:0/1328622482 >> v2:172.21.15.98:3300/0 conn(0x7f33a015aa30 0x7f33a00b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- 172.21.15.70:0/1328622482 >> 172.21.15.70:0/1328622482 conn(0x7f33a006b1f0 msgr2=0x7f33a006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- 172.21.15.70:0/1328622482 shutdown_connections 2024-04-22T01:16:39.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- 172.21.15.70:0/1328622482 wait complete. 2024-04-22T01:16:39.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 Processor -- start 2024-04-22T01:16:39.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- start start 2024-04-22T01:16:39.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a015aa30 0x7f33a00affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a015bb50 0x7f33a0152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33a006ead0 con 0x7f33a00aee40 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33a006e950 con 0x7f33a015aa30 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33a0165a80 con 0x7f33a015bb50 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48990/0 (socket says 172.21.15.70:48990) 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 -- 172.21.15.70:0/1641333664 learned_addr learned my addr 172.21.15.70:0/1641333664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339f7fe640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a015aa30 0x7f33a00affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a4db0640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a015bb50 0x7f33a0152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 -- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a015bb50 msgr2=0x7f33a0152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a015bb50 0x7f33a0152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 -- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a015aa30 msgr2=0x7f33a00affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a015aa30 0x7f33a00affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 -- 172.21.15.70:0/1641333664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a00ccf10 con 0x7f33a00aee40 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339f7fe640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a015aa30 0x7f33a00affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339ffff640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b0c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3390007ef0 tx=0x7f339000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390014620 con 0x7f33a00aee40 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3390019070 con 0x7f33a00aee40 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390014a00 con 0x7f33a00aee40 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.379+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a00b12d0 con 0x7f33a00aee40 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a01f4c90 con 0x7f33a00aee40 2024-04-22T01:16:39.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a0068210 con 0x7f33a00aee40 2024-04-22T01:16:39.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3390014ba0 con 0x7f33a00aee40 2024-04-22T01:16:39.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f339cff9640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33800678c0 0x7f3380069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f33900863c0 con 0x7f33a00aee40 2024-04-22T01:16:39.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f339f7fe640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33800678c0 0x7f3380069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.383+0000 7f339f7fe640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33800678c0 0x7f3380069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f33a0166140 tx=0x7f3388009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.387+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3390057150 con 0x7f33a00aee40 2024-04-22T01:16:39.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f33a006f190 con 0x7f33a00aee40 2024-04-22T01:16:39.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f339cff9640 1 -- 172.21.15.70:0/1641333664 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) ==== 84+0+65 (secure 0 0 0) 0x7f3390056a50 con 0x7f33a00aee40 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:39.559 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33800678c0 msgr2=0x7f3380069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33800678c0 0x7f3380069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f33a0166140 tx=0x7f3388009040 comp rx=0 tx=0).stop 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 msgr2=0x7f33a00b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b0c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3390007ef0 tx=0x7f339000e4b0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 shutdown_connections 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33800678c0 0x7f3380069d80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a015bb50 0x7f33a0152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a015aa30 0x7f33a00affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 --2- 172.21.15.70:0/1641333664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a00aee40 0x7f33a00b0c40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 >> 172.21.15.70:0/1641333664 conn(0x7f33a006b1f0 msgr2=0x7f33a015d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 shutdown_connections 2024-04-22T01:16:39.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.559+0000 7f33a6813640 1 -- 172.21.15.70:0/1641333664 wait complete. 2024-04-22T01:16:39.577 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:16:39.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:16:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 Processor -- start 2024-04-22T01:16:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 -- start start 2024-04-22T01:16:39.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffb68072280 0x7ffb680ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb680b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6815c290 con 0x7ffb680b4450 2024-04-22T01:16:39.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6815c570 con 0x7ffb68072280 2024-04-22T01:16:39.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6e826640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6815c850 con 0x7ffb680cd050 2024-04-22T01:16:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6cdc3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb680b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb677fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffb68072280 0x7ffb680ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6cdc3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb680b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:60000/0 (socket says 172.21.15.70:60000) 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6cdc3640 1 -- 172.21.15.70:0/1037645559 learned_addr learned my addr 172.21.15.70:0/1037645559 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb67fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffb680b4450 0x7ffb68071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:40314/0 (socket says 172.21.15.70:40314) 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb66ffd640 1 -- 172.21.15.70:0/1037645559 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152954236 0 0) 0x7ffb6815c290 con 0x7ffb680b4450 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb66ffd640 1 -- 172.21.15.70:0/1037645559 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb50003530 con 0x7ffb680b4450 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6cdc3640 1 -- 172.21.15.70:0/1037645559 >> v2:172.21.15.98:3300/0 conn(0x7ffb68072280 msgr2=0x7ffb680ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.703+0000 7ffb6cdc3640 1 --2- 172.21.15.70:0/1037645559 >> v2:172.21.15.98:3300/0 conn(0x7ffb68072280 0x7ffb680ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb6cdc3640 1 -- 172.21.15.70:0/1037645559 >> v1:172.21.15.98:6789/0 conn(0x7ffb680b4450 legacy=0x7ffb68071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb6cdc3640 1 -- 172.21.15.70:0/1037645559 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6815cb30 con 0x7ffb680cd050 2024-04-22T01:16:39.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb6cdc3640 1 --2- 172.21.15.70:0/1037645559 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb680b2a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009e50 tx=0x7ffb6001c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5621b6382630f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb66ffd640 1 -- 172.21.15.70:0/1037645559 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600a5880 con 0x7ffb680cd050 2024-04-22T01:16:39.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb66ffd640 1 -- 172.21.15.70:0/1037645559 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb600a5a20 con 0x7ffb680cd050 2024-04-22T01:16:39.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb66ffd640 1 -- 172.21.15.70:0/1037645559 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600a5dd0 con 0x7ffb680cd050 2024-04-22T01:16:39.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb6e826640 1 -- 172.21.15.70:0/1037645559 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 msgr2=0x7ffb680b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.707+0000 7ffb6e826640 1 --2- 172.21.15.70:0/1037645559 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb680b2a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009e50 tx=0x7ffb6001c9a0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- 172.21.15.70:0/1037645559 shutdown_connections 2024-04-22T01:16:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 --2- 172.21.15.70:0/1037645559 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb680b2a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 --2- 172.21.15.70:0/1037645559 >> v2:172.21.15.98:3300/0 conn(0x7ffb68072280 0x7ffb680ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- 172.21.15.70:0/1037645559 >> 172.21.15.70:0/1037645559 conn(0x7ffb680b1d40 msgr2=0x7ffb680cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- 172.21.15.70:0/1037645559 shutdown_connections 2024-04-22T01:16:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- 172.21.15.70:0/1037645559 wait complete. 2024-04-22T01:16:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 Processor -- start 2024-04-22T01:16:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- start start 2024-04-22T01:16:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 0x7ffb6814c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb680b4450 0x7ffb6814caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb68144780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb680cd980 con 0x7ffb68072280 2024-04-22T01:16:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb680cd800 con 0x7ffb680b4450 2024-04-22T01:16:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffb680b3790 con 0x7ffb680cd050 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 0x7ffb6814c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 0x7ffb6814c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49004/0 (socket says 172.21.15.70:49004) 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 -- 172.21.15.70:0/283451655 learned_addr learned my addr 172.21.15.70:0/283451655 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6cdc3640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb68144780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb677fe640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb680b4450 0x7ffb6814caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 -- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 msgr2=0x7ffb68144780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb68144780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 -- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb680b4450 msgr2=0x7ffb6814caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb680b4450 0x7ffb6814caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 -- 172.21.15.70:0/283451655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb68154de0 con 0x7ffb68072280 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb677fe640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb680b4450 0x7ffb6814caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb67fff640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 0x7ffb6814c560 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800c460 tx=0x7ffb5800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58018070 con 0x7ffb68072280 2024-04-22T01:16:39.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb580074a0 con 0x7ffb68072280 2024-04-22T01:16:39.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb681550c0 con 0x7ffb68072280 2024-04-22T01:16:39.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58004a30 con 0x7ffb68072280 2024-04-22T01:16:39.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.711+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68155680 con 0x7ffb68072280 2024-04-22T01:16:39.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.715+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb68068210 con 0x7ffb68072280 2024-04-22T01:16:39.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.715+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb5801b070 con 0x7ffb68072280 2024-04-22T01:16:39.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.715+0000 7ffb64ff9640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb540678f0 0x7ffb54069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.715+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7ffb58085010 con 0x7ffb68072280 2024-04-22T01:16:39.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.715+0000 7ffb677fe640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb540678f0 0x7ffb54069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.715+0000 7ffb677fe640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb540678f0 0x7ffb54069db0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffb68074d90 tx=0x7ffb5c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.719+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ffb58055df0 con 0x7ffb68072280 2024-04-22T01:16:39.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:39.875+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffb680d01e0 con 0x7ffb68072280 2024-04-22T01:16:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.251+0000 7ffb64ff9640 1 -- 172.21.15.70:0/283451655 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) ==== 138+0+0 (secure 0 0 0) 0x7ffb580556f0 con 0x7ffb68072280 2024-04-22T01:16:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb540678f0 msgr2=0x7ffb54069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb540678f0 0x7ffb54069db0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffb68074d90 tx=0x7ffb5c008040 comp rx=0 tx=0).stop 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 msgr2=0x7ffb6814c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 0x7ffb6814c560 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800c460 tx=0x7ffb5800c920 comp rx=0 tx=0).stop 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 shutdown_connections 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb540678f0 0x7ffb54069db0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb680cd050 0x7ffb68144780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb680b4450 0x7ffb6814caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 --2- 172.21.15.70:0/283451655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb68072280 0x7ffb6814c560 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 >> 172.21.15.70:0/283451655 conn(0x7ffb680b1d40 msgr2=0x7ffb6814f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 shutdown_connections 2024-04-22T01:16:40.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.255+0000 7ffb6e826640 1 -- 172.21.15.70:0/283451655 wait complete. 2024-04-22T01:16:40.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:16:40.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.407+0000 7f4cdb264640 1 Processor -- start 2024-04-22T01:16:40.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.407+0000 7f4cdb264640 1 -- start start 2024-04-22T01:16:40.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4cd415e1e0 0x7f4cd40d3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b30e0 0x7f4cd40b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cd406d740 con 0x7f4cd40b5da0 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4cd406d2b0 con 0x7f4cd415e1e0 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd406c980 con 0x7f4cd40b30e0 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4cd415e1e0 0x7f4cd40d3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4cd415e1e0 0x7f4cd40d3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49006/0 (socket says 172.21.15.70:49006) 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 -- 172.21.15.70:0/2422671780 learned_addr learned my addr 172.21.15.70:0/2422671780 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd3fff640 1 --2- 172.21.15.70:0/2422671780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b30e0 0x7f4cd40b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 -- 172.21.15.70:0/2422671780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b30e0 msgr2=0x7f4cd40b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 --2- 172.21.15.70:0/2422671780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b30e0 0x7f4cd40b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 -- 172.21.15.70:0/2422671780 >> v1:172.21.15.98:6789/0 conn(0x7f4cd40b5da0 legacy=0x7f4cd40d2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:40.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 -- 172.21.15.70:0/2422671780 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd406cc60 con 0x7f4cd415e1e0 2024-04-22T01:16:40.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd9000640 1 --2- 172.21.15.70:0/2422671780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd40d3340 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009870 tx=0x7f4cc801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f637f785cfdaed6e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:40.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd37fe640 1 -- 172.21.15.70:0/2422671780 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc80b0070 con 0x7f4cd415e1e0 2024-04-22T01:16:40.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd37fe640 1 -- 172.21.15.70:0/2422671780 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc80ab600 con 0x7f4cd415e1e0 2024-04-22T01:16:40.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cd37fe640 1 -- 172.21.15.70:0/2422671780 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc80ab9d0 con 0x7f4cd415e1e0 2024-04-22T01:16:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 -- 172.21.15.70:0/2422671780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 msgr2=0x7f4cd40d3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.411+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2422671780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd40d3340 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009870 tx=0x7f4cc801c520 comp rx=0 tx=0).stop 2024-04-22T01:16:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2422671780 shutdown_connections 2024-04-22T01:16:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2422671780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b30e0 0x7f4cd40b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2422671780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd40d3340 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2422671780 >> 172.21.15.70:0/2422671780 conn(0x7f4cd4073110 msgr2=0x7f4cd40d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2422671780 shutdown_connections 2024-04-22T01:16:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2422671780 wait complete. 2024-04-22T01:16:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 Processor -- start 2024-04-22T01:16:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- start start 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cd40b30e0 0x7f4cd40c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b5da0 0x7f4cd41558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd415ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd416af30 con 0x7f4cd415e1e0 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd416adb0 con 0x7f4cd40b30e0 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd416b0b0 con 0x7f4cd40b5da0 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd415ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd415ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49012/0 (socket says 172.21.15.70:49012) 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 -- 172.21.15.70:0/2933640318 learned_addr learned my addr 172.21.15.70:0/2933640318 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd3fff640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b5da0 0x7f4cd41558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9000640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cd40b30e0 0x7f4cd40c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 -- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b5da0 msgr2=0x7f4cd41558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b5da0 0x7f4cd41558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 -- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cd40b30e0 msgr2=0x7f4cd40c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cd40b30e0 0x7f4cd40c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 -- 172.21.15.70:0/2933640318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd40cddc0 con 0x7f4cd415e1e0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9000640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cd40b30e0 0x7f4cd40c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd9801640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd415ce40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00c780 tx=0x7f4ccc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc004b70 con 0x7f4cd415e1e0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ccc004d10 con 0x7f4cd415e1e0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00aa90 con 0x7f4cd415e1e0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd40c8a20 con 0x7f4cd415e1e0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd40c8fe0 con 0x7f4cd415e1e0 2024-04-22T01:16:40.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.415+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd4068210 con 0x7f4cd415e1e0 2024-04-22T01:16:40.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.419+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ccc00ac30 con 0x7f4cd415e1e0 2024-04-22T01:16:40.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.419+0000 7f4cd17fa640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4cb80678c0 0x7f4cb8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.419+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5619+0+0 (secure 0 0 0) 0x7f4ccc085e50 con 0x7f4cd415e1e0 2024-04-22T01:16:40.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.419+0000 7f4cd9000640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4cb80678c0 0x7f4cb8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.419+0000 7f4cd9000640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4cb80678c0 0x7f4cb8069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80ad000 tx=0x7f4cc8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:40.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.423+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4ccc020070 con 0x7f4cd415e1e0 2024-04-22T01:16:40.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:40.571+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4cd40746b0 con 0x7f4cd415e1e0 2024-04-22T01:16:41.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cd17fa640 1 -- 172.21.15.70:0/2933640318 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) ==== 170+0+0 (secure 0 0 0) 0x7f4cd40cddc0 con 0x7f4cd415e1e0 2024-04-22T01:16:41.262 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:16:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4cb80678c0 msgr2=0x7f4cb8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4cb80678c0 0x7f4cb8069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80ad000 tx=0x7f4cc8008040 comp rx=0 tx=0).stop 2024-04-22T01:16:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 msgr2=0x7f4cd415ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd415ce40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00c780 tx=0x7f4ccc00cc40 comp rx=0 tx=0).stop 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 shutdown_connections 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4cb80678c0 0x7f4cb8069d80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4cd415e1e0 0x7f4cd415ce40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cd40b5da0 0x7f4cd41558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 --2- 172.21.15.70:0/2933640318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cd40b30e0 0x7f4cd40c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.262+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 >> 172.21.15.70:0/2933640318 conn(0x7f4cd4073110 msgr2=0x7f4cd40ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.266+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 shutdown_connections 2024-04-22T01:16:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.266+0000 7f4cdb264640 1 -- 172.21.15.70:0/2933640318 wait complete. 2024-04-22T01:16:41.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:16:41.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0e43a640 1 Processor -- start 2024-04-22T01:16:41.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0e43a640 1 -- start start 2024-04-22T01:16:41.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0e43a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d080cd220 0x7f9d08166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0e43a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d0815c290 con 0x7f9d080b4450 2024-04-22T01:16:41.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0e43a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d0815c570 con 0x7f9d08072280 2024-04-22T01:16:41.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0e43a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9d080b3a50 con 0x7f9d080cd220 2024-04-22T01:16:41.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d07fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9d080b4450 0x7f9d08071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35990/0 (socket says 172.21.15.70:35990) 2024-04-22T01:16:41.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d07fff640 1 -- 172.21.15.70:0/3708272705 learned_addr learned my addr 172.21.15.70:0/3708272705 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:41.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0c9d7640 1 --2- 172.21.15.70:0/3708272705 >> v2:172.21.15.98:3300/0 conn(0x7f9d080cd220 0x7f9d08166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286264184 0 0) 0x7f9d0815c290 con 0x7f9d080b4450 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cfc003530 con 0x7f9d080b4450 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3342881597 0 0) 0x7f9cfc003530 con 0x7f9d080b4450 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 >> v2:172.21.15.98:3300/0 conn(0x7f9d080cd220 msgr2=0x7f9d08166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 --2- 172.21.15.70:0/3708272705 >> v2:172.21.15.98:3300/0 conn(0x7f9d080cd220 0x7f9d08166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 >> v1:172.21.15.98:6789/0 conn(0x7f9d08072280 legacy=0x7f9d080ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0815c850 con 0x7f9d080b4450 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f9cf8001680 con 0x7f9d080b4450 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 >> v1:172.21.15.70:6789/0 conn(0x7f9d080b4450 legacy=0x7f9d08071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc0039c0 0x7f9cfc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0051d0 0x7f9cfc00fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d080b3a50 con 0x7f9cfc0039c0 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc010480 con 0x7f9cfc004810 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9cfc010680 con 0x7f9cfc0051d0 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc0039c0 0x7f9cfc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d077fe640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.430+0000 7f9d0c9d7640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0051d0 0x7f9cfc00fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d07fff640 1 -- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0051d0 msgr2=0x7f9cfc00fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0051d0 0x7f9cfc00fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d07fff640 1 -- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc004810 msgr2=0x7f9cfc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0c9d7640 1 -- 172.21.15.70:0/3708272705 reap_dead start 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d07fff640 1 -- 172.21.15.70:0/3708272705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cfc003530 con 0x7f9cfc0039c0 2024-04-22T01:16:41.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d077fe640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:41.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc0039c0 0x7f9cfc003de0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c1c0 tx=0x7f9cf801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c345ae971dcb49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:41.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8007d00 con 0x7f9cfc0039c0 2024-04-22T01:16:41.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf80a9020 con 0x7f9cfc0039c0 2024-04-22T01:16:41.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d06ffd640 1 -- 172.21.15.70:0/3708272705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8005ae0 con 0x7f9cfc0039c0 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3708272705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc0039c0 msgr2=0x7f9cfc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc0039c0 0x7f9cfc003de0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c1c0 tx=0x7f9cf801cbe0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3708272705 shutdown_connections 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3708272705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc0039c0 0x7f9cfc003de0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3708272705 >> 172.21.15.70:0/3708272705 conn(0x7f9d080b1d40 msgr2=0x7f9d080cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3708272705 shutdown_connections 2024-04-22T01:16:41.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.434+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3708272705 wait complete. 2024-04-22T01:16:41.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 Processor -- start 2024-04-22T01:16:41.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- start start 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0039c0 0x7f9d081419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 0x7f9d081e6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d08152980 0x7f9d08156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d08071f80 con 0x7f9d08152500 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0806f900 con 0x7f9d08152980 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9d08072100 con 0x7f9cfc0039c0 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 0x7f9d081e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 0x7f9d081e6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49024/0 (socket says 172.21.15.70:49024) 2024-04-22T01:16:41.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 -- 172.21.15.70:0/3826292813 learned_addr learned my addr 172.21.15.70:0/3826292813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0039c0 0x7f9d081419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0c9d7640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d08152980 0x7f9d08156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 -- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0039c0 msgr2=0x7f9d081419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0039c0 0x7f9d081419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 -- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d08152980 msgr2=0x7f9d08156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d08152980 0x7f9d08156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 -- 172.21.15.70:0/3826292813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0813e870 con 0x7f9d08152500 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0c9d7640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d08152980 0x7f9d08156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d077fe640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 0x7f9d081e6030 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000e480 tx=0x7f9cf000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00052e0 con 0x7f9d08152500 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf0007020 con 0x7f9d08152500 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0005480 con 0x7f9d08152500 2024-04-22T01:16:41.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0813eb50 con 0x7f9d08152500 2024-04-22T01:16:41.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0813f110 con 0x7f9d08152500 2024-04-22T01:16:41.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.438+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d08068210 con 0x7f9d08152500 2024-04-22T01:16:41.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.442+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9cf001a020 con 0x7f9d08152500 2024-04-22T01:16:41.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.442+0000 7f9d04ff9640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9ce80678f0 0x7f9ce8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.442+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9ce80678f0 0x7f9ce8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.442+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6001+0+0 (secure 0 0 0) 0x7f9cf0085de0 con 0x7f9d08152500 2024-04-22T01:16:41.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.442+0000 7f9d07fff640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9ce80678f0 0x7f9ce8069db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80aa000 tx=0x7f9d0815c460 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:41.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.446+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9cf0056860 con 0x7f9d08152500 2024-04-22T01:16:41.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:41.638+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9d080d01e0 con 0x7f9d08152500 2024-04-22T01:16:42.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.262+0000 7f9d04ff9640 1 -- 172.21.15.70:0/3826292813 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v21) ==== 172+0+0 (secure 0 0 0) 0x7f9cf0056160 con 0x7f9d08152500 2024-04-22T01:16:42.265 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 7 and data pool 8 2024-04-22T01:16:42.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.266+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9ce80678f0 msgr2=0x7f9ce8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9ce80678f0 0x7f9ce8069db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80aa000 tx=0x7f9d0815c460 comp rx=0 tx=0).stop 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 msgr2=0x7f9d081e6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 0x7f9d081e6030 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000e480 tx=0x7f9cf000e940 comp rx=0 tx=0).stop 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 shutdown_connections 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9ce80678f0 0x7f9ce8069db0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d08152980 0x7f9d08156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d08152500 0x7f9d081e6030 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 --2- 172.21.15.70:0/3826292813 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cfc0039c0 0x7f9d081419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 >> 172.21.15.70:0/3826292813 conn(0x7f9d080b1d40 msgr2=0x7f9d0815ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 shutdown_connections 2024-04-22T01:16:42.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.270+0000 7f9d0e43a640 1 -- 172.21.15.70:0/3826292813 wait complete. 2024-04-22T01:16:42.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:42.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 Processor -- start 2024-04-22T01:16:42.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 -- start start 2024-04-22T01:16:42.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70241455b0 0x7f70240cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f7024071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70240d0250 con 0x7f7024072060 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f70240724d0 con 0x7f70241455b0 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.426+0000 7f702bb2d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f70240cd7b0 con 0x7f70240cc150 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70241455b0 0x7f70240cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70241455b0 0x7f70240cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49032/0 (socket says 172.21.15.70:49032) 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 -- 172.21.15.70:0/1663924511 learned_addr learned my addr 172.21.15.70:0/1663924511 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702a0ca640 1 --1- 172.21.15.70:0/1663924511 >> v1:172.21.15.70:6789/0 conn(0x7f7024072060 0x7f702415f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36006/0 (socket says 172.21.15.70:36006) 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70290c8640 1 --2- 172.21.15.70:0/1663924511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f7024071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 -- 172.21.15.70:0/1663924511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 msgr2=0x7f7024071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 --2- 172.21.15.70:0/1663924511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f7024071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 -- 172.21.15.70:0/1663924511 >> v1:172.21.15.70:6789/0 conn(0x7f7024072060 legacy=0x7f702415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 -- 172.21.15.70:0/1663924511 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70240cc710 con 0x7f70241455b0 2024-04-22T01:16:42.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70298c9640 1 --2- 172.21.15.70:0/1663924511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f70240cfd10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7020009870 tx=0x7f702001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9486b877e21b19 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70288c7640 1 -- 172.21.15.70:0/1663924511 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70200b0070 con 0x7f70241455b0 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70288c7640 1 -- 172.21.15.70:0/1663924511 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70200ab600 con 0x7f70241455b0 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f70288c7640 1 -- 172.21.15.70:0/1663924511 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70200ab9d0 con 0x7f70241455b0 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 -- 172.21.15.70:0/1663924511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 msgr2=0x7f70240cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 --2- 172.21.15.70:0/1663924511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f70240cfd10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7020009870 tx=0x7f702001c440 comp rx=0 tx=0).stop 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 -- 172.21.15.70:0/1663924511 shutdown_connections 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 --2- 172.21.15.70:0/1663924511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f7024071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 --2- 172.21.15.70:0/1663924511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f70240cfd10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 -- 172.21.15.70:0/1663924511 >> 172.21.15.70:0/1663924511 conn(0x7f702406b1f0 msgr2=0x7f702406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 -- 172.21.15.70:0/1663924511 shutdown_connections 2024-04-22T01:16:42.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.430+0000 7f702bb2d640 1 -- 172.21.15.70:0/1663924511 wait complete. 2024-04-22T01:16:42.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 Processor -- start 2024-04-22T01:16:42.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- start start 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7024072060 0x7f702415de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f702413ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f702413b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7024168b40 con 0x7f70241455b0 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f702406f210 con 0x7f7024072060 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7024168cc0 con 0x7f70240cc150 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f702413b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f702413b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49048/0 (socket says 172.21.15.70:49048) 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 -- 172.21.15.70:0/3717632350 learned_addr learned my addr 172.21.15.70:0/3717632350 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:42.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f70298c9640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7024072060 0x7f702415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f70290c8640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f702413ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 -- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 msgr2=0x7f702413ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f702413ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 -- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7024072060 msgr2=0x7f702415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7024072060 0x7f702415de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 -- 172.21.15.70:0/3717632350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70241402d0 con 0x7f70241455b0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f70298c9640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7024072060 0x7f702415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702a0ca640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f702413b3a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f701c00c760 tx=0x7f701c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c018070 con 0x7f70241455b0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f701c004d70 con 0x7f70241455b0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70241405b0 con 0x7f70241455b0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c00a880 con 0x7f70241455b0 2024-04-22T01:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7024140b70 con 0x7f70241455b0 2024-04-22T01:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7024068210 con 0x7f70241455b0 2024-04-22T01:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.434+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f701c01b070 con 0x7f70241455b0 2024-04-22T01:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.438+0000 7f701a7fc640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70080678c0 0x7f7008069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.438+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f701c085250 con 0x7f70241455b0 2024-04-22T01:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.438+0000 7f70298c9640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70080678c0 0x7f7008069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.438+0000 7f70298c9640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70080678c0 0x7f7008069d80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f70200ad000 tx=0x7f7020008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.442+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f701c055c00 con 0x7f70241455b0 2024-04-22T01:16:42.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:16:42.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:16:42.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.602+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f702406eff0 con 0x7f70241455b0 2024-04-22T01:16:42.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f701a7fc640 1 -- 172.21.15.70:0/3717632350 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16919 (secure 0 0 0) 0x7f701c055500 con 0x7f70241455b0 2024-04-22T01:16:42.607 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:42.607 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":51,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:41.642420+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:42.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70080678c0 msgr2=0x7f7008069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70080678c0 0x7f7008069d80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f70200ad000 tx=0x7f7020008040 comp rx=0 tx=0).stop 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 msgr2=0x7f702413b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f702413b3a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f701c00c760 tx=0x7f701c00cb60 comp rx=0 tx=0).stop 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 shutdown_connections 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70080678c0 0x7f7008069d80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241455b0 0x7f702413b3a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70240cc150 0x7f702413ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 --2- 172.21.15.70:0/3717632350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7024072060 0x7f702415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 >> 172.21.15.70:0/3717632350 conn(0x7f702406b1f0 msgr2=0x7f702406e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 shutdown_connections 2024-04-22T01:16:42.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.606+0000 7f702bb2d640 1 -- 172.21.15.70:0/3717632350 wait complete. 2024-04-22T01:16:42.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:42.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 Processor -- start 2024-04-22T01:16:42.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 -- start start 2024-04-22T01:16:42.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f014815dfb0 0x7f01480b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480d3120 0x7f01480ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014815e390 con 0x7f01480cea80 2024-04-22T01:16:42.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f01480cf420 con 0x7f014815dfb0 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014fb67640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0148161e00 con 0x7f01480d3120 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f014815dfb0 0x7f01480b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f014815dfb0 0x7f01480b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49058/0 (socket says 172.21.15.70:49058) 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d102640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f01480cea80 0x7f01480b5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36008/0 (socket says 172.21.15.70:36008) 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 -- 172.21.15.70:0/4267896557 learned_addr learned my addr 172.21.15.70:0/4267896557 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 -- 172.21.15.70:0/4267896557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480d3120 msgr2=0x7f01480ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 --2- 172.21.15.70:0/4267896557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480d3120 0x7f01480ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 -- 172.21.15.70:0/4267896557 >> v1:172.21.15.70:6789/0 conn(0x7f01480cea80 legacy=0x7f01480b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014d903640 1 -- 172.21.15.70:0/4267896557 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01481620e0 con 0x7f014815dfb0 2024-04-22T01:16:42.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.790+0000 7f014c901640 1 -- 172.21.15.70:0/4267896557 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917480330 0 0) 0x7f014815e390 con 0x7f01480cea80 2024-04-22T01:16:42.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014d903640 1 --2- 172.21.15.70:0/4267896557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480b95d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0144009870 tx=0x7f014401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3988ece5b534d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014c901640 1 -- 172.21.15.70:0/4267896557 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440b2070 con 0x7f014815dfb0 2024-04-22T01:16:42.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014c901640 1 -- 172.21.15.70:0/4267896557 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01440ad640 con 0x7f014815dfb0 2024-04-22T01:16:42.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014c901640 1 -- 172.21.15.70:0/4267896557 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440ada10 con 0x7f014815dfb0 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- 172.21.15.70:0/4267896557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 msgr2=0x7f01480b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 --2- 172.21.15.70:0/4267896557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480b95d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0144009870 tx=0x7f014401c560 comp rx=0 tx=0).stop 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- 172.21.15.70:0/4267896557 shutdown_connections 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 --2- 172.21.15.70:0/4267896557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480d3120 0x7f01480ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 --2- 172.21.15.70:0/4267896557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480b95d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- 172.21.15.70:0/4267896557 >> 172.21.15.70:0/4267896557 conn(0x7f014806c9b0 msgr2=0x7f014806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- 172.21.15.70:0/4267896557 shutdown_connections 2024-04-22T01:16:42.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- 172.21.15.70:0/4267896557 wait complete. 2024-04-22T01:16:42.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 Processor -- start 2024-04-22T01:16:42.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- start start 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480cea80 0x7f0148163a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01480d3120 0x7f01480cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480d3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f014816add0 con 0x7f014815dfb0 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0148161a90 con 0x7f01480d3120 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014fb67640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f014816af50 con 0x7f01480cea80 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480d3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480d3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49066/0 (socket says 172.21.15.70:49066) 2024-04-22T01:16:42.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 -- 172.21.15.70:0/2152241229 learned_addr learned my addr 172.21.15.70:0/2152241229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014d903640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480cea80 0x7f0148163a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 -- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480cea80 msgr2=0x7f0148163a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480cea80 0x7f0148163a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014d102640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01480d3120 0x7f01480cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 -- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01480d3120 msgr2=0x7f01480cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01480d3120 0x7f01480cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 -- 172.21.15.70:0/2152241229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01480ccd20 con 0x7f014815dfb0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014d903640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480cea80 0x7f0148163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.794+0000 7f014e104640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480d3ef0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f014000b9e0 tx=0x7f014000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140019070 con 0x7f014815dfb0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014000cb20 con 0x7f014815dfb0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0148162cd0 con 0x7f014815dfb0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140014440 con 0x7f014815dfb0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0148163290 con 0x7f014815dfb0 2024-04-22T01:16:42.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0148068210 con 0x7f014815dfb0 2024-04-22T01:16:42.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f014001c070 con 0x7f014815dfb0 2024-04-22T01:16:42.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f013e7fc640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f012c0678c0 0x7f012c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f0140085b60 con 0x7f014815dfb0 2024-04-22T01:16:42.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f014d903640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f012c0678c0 0x7f012c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.798+0000 7f014d903640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f012c0678c0 0x7f012c069d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f01440af000 tx=0x7f0144008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.802+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0140056510 con 0x7f014815dfb0 2024-04-22T01:16:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.966+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01480ca3e0 con 0x7f014815dfb0 2024-04-22T01:16:42.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.966+0000 7f013e7fc640 1 -- 172.21.15.70:0/2152241229 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16919 (secure 0 0 0) 0x7f0140055e10 con 0x7f014815dfb0 2024-04-22T01:16:42.969 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:42.969 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":51,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:41.642420+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f012c0678c0 msgr2=0x7f012c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f012c0678c0 0x7f012c069d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f01440af000 tx=0x7f0144008040 comp rx=0 tx=0).stop 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 msgr2=0x7f01480d3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480d3ef0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f014000b9e0 tx=0x7f014000bea0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 shutdown_connections 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f012c0678c0 0x7f012c069d80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f014815dfb0 0x7f01480d3ef0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01480d3120 0x7f01480cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 --2- 172.21.15.70:0/2152241229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01480cea80 0x7f0148163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 >> 172.21.15.70:0/2152241229 conn(0x7f014806c9b0 msgr2=0x7f01480c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 shutdown_connections 2024-04-22T01:16:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:42.970+0000 7f014fb67640 1 -- 172.21.15.70:0/2152241229 wait complete. 2024-04-22T01:16:42.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:16:43.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.106+0000 7f648a555640 1 Processor -- start 2024-04-22T01:16:43.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.106+0000 7f648a555640 1 -- start start 2024-04-22T01:16:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f648a555640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f64840b4450 0x7f6484071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f648a555640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648415c290 con 0x7f64840720b0 2024-04-22T01:16:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f648a555640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648415c570 con 0x7f64840cd220 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f648a555640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f648415c850 con 0x7f64840b4450 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f6483fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f64840b4450 0x7f6484071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f64837fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f64840720b0 0x7f64840ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36022/0 (socket says 172.21.15.70:36022) 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f64837fe640 1 -- 172.21.15.70:0/3668866286 learned_addr learned my addr 172.21.15.70:0/3668866286 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f6483fff640 1 -- 172.21.15.70:0/3668866286 >> v1:172.21.15.98:6789/0 conn(0x7f64840cd220 legacy=0x7f64840b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f6483fff640 1 -- 172.21.15.70:0/3668866286 >> v1:172.21.15.70:6789/0 conn(0x7f64840720b0 legacy=0x7f64840ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f6483fff640 1 -- 172.21.15.70:0/3668866286 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648415cb30 con 0x7f64840b4450 2024-04-22T01:16:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.110+0000 7f6482ffd640 1 -- 172.21.15.70:0/3668866286 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792858663 0 0) 0x7f648415c290 con 0x7f64840720b0 2024-04-22T01:16:43.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6483fff640 1 --2- 172.21.15.70:0/3668866286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484071b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f64740097c0 tx=0x7f647401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b2af9e1586f25b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:43.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6482ffd640 1 -- 172.21.15.70:0/3668866286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64740b0070 con 0x7f64840b4450 2024-04-22T01:16:43.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6482ffd640 1 -- 172.21.15.70:0/3668866286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64740ab600 con 0x7f64840b4450 2024-04-22T01:16:43.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6482ffd640 1 -- 172.21.15.70:0/3668866286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64740ab9d0 con 0x7f64840b4450 2024-04-22T01:16:43.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3668866286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 msgr2=0x7f6484071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:43.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 --2- 172.21.15.70:0/3668866286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484071b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f64740097c0 tx=0x7f647401c460 comp rx=0 tx=0).stop 2024-04-22T01:16:43.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3668866286 shutdown_connections 2024-04-22T01:16:43.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 --2- 172.21.15.70:0/3668866286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484071b70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3668866286 >> 172.21.15.70:0/3668866286 conn(0x7f64840b1d40 msgr2=0x7f64840cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:43.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3668866286 shutdown_connections 2024-04-22T01:16:43.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3668866286 wait complete. 2024-04-22T01:16:43.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 Processor -- start 2024-04-22T01:16:43.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- start start 2024-04-22T01:16:43.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64840720b0 0x7f648414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:43.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:43.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64840cd220 0x7f648414ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:43.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6484169ca0 con 0x7f64840b4450 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6484169b20 con 0x7f64840cd220 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6484169e20 con 0x7f64840720b0 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49086/0 (socket says 172.21.15.70:49086) 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 -- 172.21.15.70:0/3987252126 learned_addr learned my addr 172.21.15.70:0/3987252126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 -- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64840720b0 msgr2=0x7f648414ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6483fff640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64840720b0 0x7f648414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64840720b0 0x7f648414ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 -- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64840cd220 msgr2=0x7f648414ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64840cd220 0x7f648414ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 -- 172.21.15.70:0/3987252126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6484144980 con 0x7f64840b4450 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f64837fe640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484152320 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f647800c480 tx=0x7f647800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:43.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478018070 con 0x7f64840b4450 2024-04-22T01:16:43.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6478007500 con 0x7f64840b4450 2024-04-22T01:16:43.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64841438e0 con 0x7f64840b4450 2024-04-22T01:16:43.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478004a30 con 0x7f64840b4450 2024-04-22T01:16:43.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.114+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6484143ea0 con 0x7f64840b4450 2024-04-22T01:16:43.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.118+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6484068210 con 0x7f64840b4450 2024-04-22T01:16:43.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.118+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f647801b070 con 0x7f64840b4450 2024-04-22T01:16:43.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.118+0000 7f6480ff9640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64700678c0 0x7f6470069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:43.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.118+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f6478085070 con 0x7f64840b4450 2024-04-22T01:16:43.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.118+0000 7f6483fff640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64700678c0 0x7f6470069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:43.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.118+0000 7f6483fff640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64700678c0 0x7f6470069d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f64740ad000 tx=0x7f64740a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:43.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.122+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6478055ab0 con 0x7f64840b4450 2024-04-22T01:16:43.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:43.322+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f64840d01e0 con 0x7f64840b4450 2024-04-22T01:16:44.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.326+0000 7f6480ff9640 1 -- 172.21.15.70:0/3987252126 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f64780553b0 con 0x7f64840b4450 2024-04-22T01:16:44.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.326+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64700678c0 msgr2=0x7f6470069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.326+0000 7f648a555640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64700678c0 0x7f6470069d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f64740ad000 tx=0x7f64740a6000 comp rx=0 tx=0).stop 2024-04-22T01:16:44.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.326+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 msgr2=0x7f6484152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.326+0000 7f648a555640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484152320 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f647800c480 tx=0x7f647800c940 comp rx=0 tx=0).stop 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.326+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 shutdown_connections 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f64700678c0 0x7f6470069d80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64840cd220 0x7f648414ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64840b4450 0x7f6484152320 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 --2- 172.21.15.70:0/3987252126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64840720b0 0x7f648414ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 >> 172.21.15.70:0/3987252126 conn(0x7f64840b1d40 msgr2=0x7f64840b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 shutdown_connections 2024-04-22T01:16:44.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.330+0000 7f648a555640 1 -- 172.21.15.70:0/3987252126 wait complete. 2024-04-22T01:16:44.350 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-22T01:16:44.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:16:44.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.474+0000 7fe769ca4640 1 Processor -- start 2024-04-22T01:16:44.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.474+0000 7fe769ca4640 1 -- start start 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe769ca4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 0x7fe7640ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe769ca4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe76415c290 con 0x7fe7640b4450 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe769ca4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe76415c570 con 0x7fe7640cd050 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe769ca4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe76415c850 con 0x7fe764072280 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7637fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe7640b4450 0x7fe764071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36038/0 (socket says 172.21.15.70:36038) 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7637fe640 1 -- 172.21.15.70:0/1754494175 learned_addr learned my addr 172.21.15.70:0/1754494175 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 0x7fe7640ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788025938 0 0) 0x7fe76415c290 con 0x7fe7640b4450 2024-04-22T01:16:44.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe74c003530 con 0x7fe7640b4450 2024-04-22T01:16:44.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024493816 0 0) 0x7fe74c003530 con 0x7fe7640b4450 2024-04-22T01:16:44.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 msgr2=0x7fe7640ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:44.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 0x7fe7640ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 >> v1:172.21.15.98:6789/0 conn(0x7fe7640cd050 legacy=0x7fe7640b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76415cb30 con 0x7fe7640b4450 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fe754001680 con 0x7fe7640b4450 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 >> v1:172.21.15.70:6789/0 conn(0x7fe7640b4450 legacy=0x7fe764071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0039f0 0x7fe74c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe74c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 0x7fe74c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe76415c850 con 0x7fe74c004810 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c00ba80 con 0x7fe74c0039f0 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe74c00bc80 con 0x7fe764072280 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe74c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe7637fe640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0039f0 0x7fe74c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe763fff640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 0x7fe74c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 -- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 msgr2=0x7fe74c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:44.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764072280 0x7fe74c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 -- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0039f0 msgr2=0x7fe74c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:44.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0039f0 0x7fe74c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.478+0000 7fe762ffd640 1 -- 172.21.15.70:0/1754494175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c003530 con 0x7fe74c004810 2024-04-22T01:16:44.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe762ffd640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe74c004c90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe758009ab0 tx=0x7fe75801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b0b71cf0cc23bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:44.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7580b2070 con 0x7fe74c004810 2024-04-22T01:16:44.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7580a5510 con 0x7fe74c004810 2024-04-22T01:16:44.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7627fc640 1 -- 172.21.15.70:0/1754494175 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7580096b0 con 0x7fe74c004810 2024-04-22T01:16:44.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/1754494175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 msgr2=0x7fe74c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 --2- 172.21.15.70:0/1754494175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe74c004c90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe758009ab0 tx=0x7fe75801ca30 comp rx=0 tx=0).stop 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe763fff640 1 -- 172.21.15.70:0/1754494175 reap_dead start 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/1754494175 shutdown_connections 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/1754494175 >> 172.21.15.70:0/1754494175 conn(0x7fe7640b1d40 msgr2=0x7fe7640cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/1754494175 shutdown_connections 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/1754494175 wait complete. 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 Processor -- start 2024-04-22T01:16:44.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- start start 2024-04-22T01:16:44.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe7641432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764151e10 0x7fe764152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7640b04a0 0x7fe7640b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe7640b4830 con 0x7fe74c004810 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7640cd4a0 con 0x7fe7640b04a0 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe7640b49b0 con 0x7fe764151e10 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe7641432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe7641432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49112/0 (socket says 172.21.15.70:49112) 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 -- 172.21.15.70:0/2267966462 learned_addr learned my addr 172.21.15.70:0/2267966462 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe762ffd640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764151e10 0x7fe764152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe763fff640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7640b04a0 0x7fe7640b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 -- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764151e10 msgr2=0x7fe764152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764151e10 0x7fe764152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 -- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7640b04a0 msgr2=0x7fe7640b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7640b04a0 0x7fe7640b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:44.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 -- 172.21.15.70:0/2267966462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe764156b00 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe7637fe640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe7641432f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe754011850 tx=0x7fe754011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754010040 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe75401a070 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe764152980 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75400fdc0 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.482+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe764152f40 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.486+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe764068210 con 0x7fe74c004810 2024-04-22T01:16:44.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.486+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe75401f460 con 0x7fe74c004810 2024-04-22T01:16:44.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.486+0000 7fe74bfff640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe7500678c0 0x7fe750069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:44.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.486+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fe754088eb0 con 0x7fe74c004810 2024-04-22T01:16:44.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.486+0000 7fe762ffd640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe7500678c0 0x7fe750069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:44.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.486+0000 7fe762ffd640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe7500678c0 0x7fe750069d80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe7580af000 tx=0x7fe7580046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:44.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.490+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe7540598f0 con 0x7fe74c004810 2024-04-22T01:16:44.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:44.686+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe7640d01e0 con 0x7fe74c004810 2024-04-22T01:16:45.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.334+0000 7fe74bfff640 1 -- 172.21.15.70:0/2267966462 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7fe7540591f0 con 0x7fe74c004810 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.334+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe7500678c0 msgr2=0x7fe750069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.334+0000 7fe769ca4640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe7500678c0 0x7fe750069d80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe7580af000 tx=0x7fe7580046d0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 msgr2=0x7fe7641432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe7641432f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe754011850 tx=0x7fe754011d10 comp rx=0 tx=0).stop 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 shutdown_connections 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe7500678c0 0x7fe750069d80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7640b04a0 0x7fe7640b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe764151e10 0x7fe764152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 --2- 172.21.15.70:0/2267966462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe74c004810 0x7fe7641432f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 >> 172.21.15.70:0/2267966462 conn(0x7fe7640b1d40 msgr2=0x7fe764071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:45.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 shutdown_connections 2024-04-22T01:16:45.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.338+0000 7fe769ca4640 1 -- 172.21.15.70:0/2267966462 wait complete. 2024-04-22T01:16:45.359 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:16:45.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:45.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.482+0000 7f716ac65640 1 Processor -- start 2024-04-22T01:16:45.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.482+0000 7f716ac65640 1 -- start start 2024-04-22T01:16:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f716ac65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f716415aa30 0x7f71640b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f716ac65640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716406dbd0 con 0x7f71640aee40 2024-04-22T01:16:45.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f716ac65640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716406de70 con 0x7f716415bb50 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f716ac65640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f716406e150 con 0x7f716415aa30 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7169202640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f71640aee40 0x7f71640b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36050/0 (socket says 172.21.15.70:36050) 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7169202640 1 -- 172.21.15.70:0/2792693 learned_addr learned my addr 172.21.15.70:0/2792693 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7168a01640 1 --2- 172.21.15.70:0/2792693 >> v2:172.21.15.98:3300/0 conn(0x7f716415aa30 0x7f71640b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2177086877 0 0) 0x7f716406dbd0 con 0x7f71640aee40 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7154003530 con 0x7f71640aee40 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 300150648 0 0) 0x7f7154003530 con 0x7f71640aee40 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 >> v2:172.21.15.98:3300/0 conn(0x7f716415aa30 msgr2=0x7f71640b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 --2- 172.21.15.70:0/2792693 >> v2:172.21.15.98:3300/0 conn(0x7f716415aa30 0x7f71640b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 >> v1:172.21.15.98:6789/0 conn(0x7f716415bb50 legacy=0x7f71640ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716406e430 con 0x7f71640aee40 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f715c001a20 con 0x7f71640aee40 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 >> v1:172.21.15.70:6789/0 conn(0x7f71640aee40 legacy=0x7f71640b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71540039c0 0x7f7154003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f7154004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71540051d0 0x7f715400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f716406e150 con 0x7f7154004810 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f715400c070 con 0x7f71540051d0 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f715400c270 con 0x7f71540039c0 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f7154004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7169202640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71540051d0 0x7f715400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7168a01640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71540039c0 0x7f7154003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 -- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71540039c0 msgr2=0x7f7154003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71540039c0 0x7f7154003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 -- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71540051d0 msgr2=0x7f715400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71540051d0 0x7f715400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 -- 172.21.15.70:0/2792693 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7154003530 con 0x7f7154004810 2024-04-22T01:16:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7169202640 1 -- 172.21.15.70:0/2792693 reap_dead start 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f7163fff640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f7154004c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7150001780 tx=0x7f715001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890630744697b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500013a0 con 0x7f7154004810 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71500052f0 con 0x7f7154004810 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.486+0000 7f71637fe640 1 -- 172.21.15.70:0/2792693 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150003840 con 0x7f7154004810 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/2792693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 msgr2=0x7f7154004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f7154004c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7150001780 tx=0x7f715001c400 comp rx=0 tx=0).stop 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/2792693 shutdown_connections 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 --2- 172.21.15.70:0/2792693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f7154004c90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/2792693 >> 172.21.15.70:0/2792693 conn(0x7f716406b1f0 msgr2=0x7f716406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:45.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/2792693 shutdown_connections 2024-04-22T01:16:45.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/2792693 wait complete. 2024-04-22T01:16:45.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 Processor -- start 2024-04-22T01:16:45.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- start start 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f71641e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640a84c0 0x7f7164071d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71640a9d20 0x7f71640aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f716415ae10 con 0x7f7154004810 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71640b4d00 con 0x7f71640a84c0 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f716415af90 con 0x7f71640a9d20 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f71641e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f71641e70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49134/0 (socket says 172.21.15.70:49134) 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 -- 172.21.15.70:0/4114643539 learned_addr learned my addr 172.21.15.70:0/4114643539 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7163fff640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640a84c0 0x7f7164071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7169202640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71640a9d20 0x7f71640aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 -- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71640a9d20 msgr2=0x7f71640aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71640a9d20 0x7f71640aba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 -- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640a84c0 msgr2=0x7f7164071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640a84c0 0x7f7164071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 -- 172.21.15.70:0/4114643539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71640a8ca0 con 0x7f7154004810 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f7168a01640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f71641e70c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7158011500 tx=0x7f71580119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158015870 con 0x7f7154004810 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7158015a10 con 0x7f7154004810 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71640a8f80 con 0x7f7154004810 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158015dc0 con 0x7f7154004810 2024-04-22T01:16:45.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.490+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71640a9820 con 0x7f7154004810 2024-04-22T01:16:45.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.494+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71640a9150 con 0x7f7154004810 2024-04-22T01:16:45.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.494+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7158013400 con 0x7f7154004810 2024-04-22T01:16:45.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.494+0000 7f71617fa640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71480678c0 0x7f7148069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.494+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f71580868a0 con 0x7f7154004810 2024-04-22T01:16:45.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.494+0000 7f7163fff640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71480678c0 0x7f7148069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.494+0000 7f7163fff640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71480678c0 0x7f7148069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7150001750 tx=0x7f71500011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:45.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.498+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7158019030 con 0x7f7154004810 2024-04-22T01:16:45.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7164068210 con 0x7f7154004810 2024-04-22T01:16:45.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f71617fa640 1 -- 172.21.15.70:0/4114643539 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5178 (secure 0 0 0) 0x7f7158057130 con 0x7f7154004810 2024-04-22T01:16:45.695 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:45.695 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:41.643333+0000","modified":"2024-04-22T01:16:44.689368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5922},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5922":{"gid":5922,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2188446179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2188446179},{"type":"v1","addr":"172.21.15.70:6837","nonce":2188446179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5922,"qdb_cluster":[5922]},"id":3}]} 2024-04-22T01:16:45.695 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 25 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71480678c0 msgr2=0x7f7148069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71480678c0 0x7f7148069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7150001750 tx=0x7f71500011a0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 msgr2=0x7f71641e70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f71641e70c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7158011500 tx=0x7f71580119c0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 shutdown_connections 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71480678c0 0x7f7148069d80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71640a9d20 0x7f71640aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640a84c0 0x7f7164071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 --2- 172.21.15.70:0/4114643539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7154004810 0x7f71641e70c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 >> 172.21.15.70:0/4114643539 conn(0x7f716406b1f0 msgr2=0x7f71640ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:45.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 shutdown_connections 2024-04-22T01:16:45.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.694+0000 7f716ac65640 1 -- 172.21.15.70:0/4114643539 wait complete. 2024-04-22T01:16:45.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:45.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.826+0000 7f05575a0640 1 Processor -- start 2024-04-22T01:16:45.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.826+0000 7f05575a0640 1 -- start start 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05501455b0 0x7f05500cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05500d0250 con 0x7f0550072230 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05500cd7b0 con 0x7f05500cc150 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f05500cc710 con 0x7f05501455b0 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f055533c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05501455b0 0x7f05500cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f0555b3d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0550072230 0x7f055006e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36056/0 (socket says 172.21.15.70:36056) 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f0555b3d640 1 -- 172.21.15.70:0/3960280 learned_addr learned my addr 172.21.15.70:0/3960280 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f055533c640 1 -- 172.21.15.70:0/3960280 >> v1:172.21.15.98:6789/0 conn(0x7f05500cc150 legacy=0x7f0550071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f055533c640 1 -- 172.21.15.70:0/3960280 >> v1:172.21.15.70:6789/0 conn(0x7f0550072230 legacy=0x7f055006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f055533c640 1 -- 172.21.15.70:0/3960280 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05500cc9f0 con 0x7f05501455b0 2024-04-22T01:16:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f0547fff640 1 -- 172.21.15.70:0/3960280 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 913390505 0 0) 0x7f05500d0250 con 0x7f0550072230 2024-04-22T01:16:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f055533c640 1 --2- 172.21.15.70:0/3960280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05501455b0 0x7f05500cfd10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f054c0097c0 tx=0x7f054c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf49ac627e850d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f0547fff640 1 -- 172.21.15.70:0/3960280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c0b0070 con 0x7f05501455b0 2024-04-22T01:16:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f0547fff640 1 -- 172.21.15.70:0/3960280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f054c0ab600 con 0x7f05501455b0 2024-04-22T01:16:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f0547fff640 1 -- 172.21.15.70:0/3960280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c0ab9d0 con 0x7f05501455b0 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- 172.21.15.70:0/3960280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05501455b0 msgr2=0x7f05500cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 --2- 172.21.15.70:0/3960280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05501455b0 0x7f05500cfd10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f054c0097c0 tx=0x7f054c01c340 comp rx=0 tx=0).stop 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- 172.21.15.70:0/3960280 shutdown_connections 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 --2- 172.21.15.70:0/3960280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05501455b0 0x7f05500cfd10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- 172.21.15.70:0/3960280 >> 172.21.15.70:0/3960280 conn(0x7f055006b1f0 msgr2=0x7f055006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- 172.21.15.70:0/3960280 shutdown_connections 2024-04-22T01:16:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.830+0000 7f05575a0640 1 -- 172.21.15.70:0/3960280 wait complete. 2024-04-22T01:16:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 Processor -- start 2024-04-22T01:16:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- start start 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0550072230 0x7f0550142680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 0x7f05500a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501455b0 0x7f055013c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0550169ee0 con 0x7f05500cc150 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0550169d60 con 0x7f05501455b0 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f055016a060 con 0x7f0550072230 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 0x7f05500a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 0x7f05500a8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49138/0 (socket says 172.21.15.70:49138) 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 -- 172.21.15.70:0/1093079024 learned_addr learned my addr 172.21.15.70:0/1093079024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f055533c640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0550072230 0x7f0550142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0555b3d640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501455b0 0x7f055013c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 -- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0550072230 msgr2=0x7f0550142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0550072230 0x7f0550142680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 -- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501455b0 msgr2=0x7f055013c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501455b0 0x7f055013c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 -- 172.21.15.70:0/1093079024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055015b640 con 0x7f05500cc150 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0554b3b640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 0x7f05500a8ed0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f054000b410 tx=0x7f054000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540015030 con 0x7f05500cc150 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0540004020 con 0x7f05500cc150 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540004450 con 0x7f05500cc150 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0550142cf0 con 0x7f05500cc150 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05501432b0 con 0x7f05500cc150 2024-04-22T01:16:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.834+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0550068210 con 0x7f05500cc150 2024-04-22T01:16:45.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.838+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0540018070 con 0x7f05500cc150 2024-04-22T01:16:45.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.838+0000 7f0545ffb640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05340678c0 0x7f0534069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:45.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.838+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f0540085940 con 0x7f05500cc150 2024-04-22T01:16:45.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.838+0000 7f055533c640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05340678c0 0x7f0534069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:45.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.838+0000 7f055533c640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05340678c0 0x7f0534069d80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f054c0ad000 tx=0x7f054c0a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:45.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.842+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f055015b640 con 0x7f05500cc150 2024-04-22T01:16:45.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.994+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f055015f510 con 0x7f05500cc150 2024-04-22T01:16:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.994+0000 7f0545ffb640 1 -- 172.21.15.70:0/1093079024 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f0540056380 con 0x7f05500cc150 2024-04-22T01:16:45.997 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:45.997 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:43.266695+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05340678c0 msgr2=0x7f0534069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05340678c0 0x7f0534069d80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f054c0ad000 tx=0x7f054c0a6000 comp rx=0 tx=0).stop 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 msgr2=0x7f05500a8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 0x7f05500a8ed0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f054000b410 tx=0x7f054000b8d0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 shutdown_connections 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05340678c0 0x7f0534069d80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05501455b0 0x7f055013c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05500cc150 0x7f05500a8ed0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 --2- 172.21.15.70:0/1093079024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0550072230 0x7f0550142680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 >> 172.21.15.70:0/1093079024 conn(0x7f055006b1f0 msgr2=0x7f055006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:46.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 shutdown_connections 2024-04-22T01:16:46.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:45.998+0000 7f05575a0640 1 -- 172.21.15.70:0/1093079024 wait complete. 2024-04-22T01:16:46.014 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:16:46.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:16:46.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.118+0000 7f452f421640 1 Processor -- start 2024-04-22T01:16:46.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.118+0000 7f452f421640 1 -- start start 2024-04-22T01:16:46.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452f421640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f452815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452f421640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45280cd740 0x7f452806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452f421640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f452815fd10 con 0x7f452806b730 2024-04-22T01:16:46.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452f421640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f452806bb80 con 0x7f45280cd740 2024-04-22T01:16:46.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452f421640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45280cfd10 con 0x7f452815bc70 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f452815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f452815f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:60146/0 (socket says 172.21.15.70:60146) 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452c9bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45280cd740 0x7f452806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 -- 172.21.15.70:0/3528315495 learned_addr learned my addr 172.21.15.70:0/3528315495 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 -- 172.21.15.70:0/3528315495 >> v2:172.21.15.98:3300/0 conn(0x7f45280cd740 msgr2=0x7f452806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 --2- 172.21.15.70:0/3528315495 >> v2:172.21.15.98:3300/0 conn(0x7f45280cd740 0x7f452806b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 -- 172.21.15.70:0/3528315495 >> v1:172.21.15.98:6789/0 conn(0x7f452806b730 legacy=0x7f45280ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 -- 172.21.15.70:0/3528315495 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452815c050 con 0x7f452815bc70 2024-04-22T01:16:46.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452c9bc640 1 --2- 172.21.15.70:0/3528315495 >> v2:172.21.15.98:3300/0 conn(0x7f45280cd740 0x7f452806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:16:46.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.122+0000 7f452d1bd640 1 --2- 172.21.15.70:0/3528315495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f452815f7d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4524009870 tx=0x7f452401c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdfc4947846f953c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f451ffff640 1 -- 172.21.15.70:0/3528315495 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45240ab640 con 0x7f452815bc70 2024-04-22T01:16:46.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f451ffff640 1 -- 172.21.15.70:0/3528315495 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45240b0070 con 0x7f452815bc70 2024-04-22T01:16:46.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f451ffff640 1 -- 172.21.15.70:0/3528315495 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45240aba00 con 0x7f452815bc70 2024-04-22T01:16:46.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/3528315495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 msgr2=0x7f452815f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 --2- 172.21.15.70:0/3528315495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f452815f7d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4524009870 tx=0x7f452401c530 comp rx=0 tx=0).stop 2024-04-22T01:16:46.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/3528315495 shutdown_connections 2024-04-22T01:16:46.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 --2- 172.21.15.70:0/3528315495 >> v2:172.21.15.98:3300/0 conn(0x7f45280cd740 0x7f452806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 --2- 172.21.15.70:0/3528315495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f452815f7d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/3528315495 >> 172.21.15.70:0/3528315495 conn(0x7f4528099a60 msgr2=0x7f4528099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:46.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/3528315495 shutdown_connections 2024-04-22T01:16:46.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/3528315495 wait complete. 2024-04-22T01:16:46.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 Processor -- start 2024-04-22T01:16:46.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- start start 2024-04-22T01:16:46.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f452806b730 0x7f45280b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 0x7f45280b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f45280b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4528168b70 con 0x7f45280cd740 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45280cff60 con 0x7f452806b730 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4528168cf0 con 0x7f452815bc70 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 0x7f45280b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 0x7f45280b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60566/0 (socket says 172.21.15.70:60566) 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 -- 172.21.15.70:0/2444667858 learned_addr learned my addr 172.21.15.70:0/2444667858 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452d9be640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f45280b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452d1bd640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f452806b730 0x7f45280b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 -- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 msgr2=0x7f45280b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f45280b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 -- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f452806b730 msgr2=0x7f45280b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f452806b730 0x7f45280b2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 -- 172.21.15.70:0/2444667858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45280b0f40 con 0x7f45280cd740 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452c9bc640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 0x7f45280b29a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f451800ebe0 tx=0x7f451800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518017070 con 0x7f45280cd740 2024-04-22T01:16:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f451800cc30 con 0x7f45280cd740 2024-04-22T01:16:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45280b1220 con 0x7f45280cd740 2024-04-22T01:16:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518013470 con 0x7f45280cd740 2024-04-22T01:16:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.126+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45280b17e0 con 0x7f45280cd740 2024-04-22T01:16:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.130+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4528153670 con 0x7f45280cd740 2024-04-22T01:16:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.130+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f451801a070 con 0x7f45280cd740 2024-04-22T01:16:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.130+0000 7f451dffb640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f450c0678c0 0x7f450c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.130+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4518086520 con 0x7f45280cd740 2024-04-22T01:16:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.130+0000 7f452d1bd640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f450c0678c0 0x7f450c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.130+0000 7f452d1bd640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f450c0678c0 0x7f450c069d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f45240af000 tx=0x7f45240a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.134+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4518056fb0 con 0x7f45280cd740 2024-04-22T01:16:46.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.278+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4528068920 con 0x7f450c0678c0 2024-04-22T01:16:46.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.294+0000 7f451dffb640 1 -- 172.21.15.70:0/2444667858 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251953 (secure 0 0 0) 0x7f4528068920 con 0x7f450c0678c0 2024-04-22T01:16:46.299 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:46.301 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f450c0678c0 msgr2=0x7f450c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f450c0678c0 0x7f450c069d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f45240af000 tx=0x7f45240a7000 comp rx=0 tx=0).stop 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 msgr2=0x7f45280b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 0x7f45280b29a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f451800ebe0 tx=0x7f451800c520 comp rx=0 tx=0).stop 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 shutdown_connections 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f450c0678c0 0x7f450c069d80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f452815bc70 0x7f45280b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45280cd740 0x7f45280b29a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 --2- 172.21.15.70:0/2444667858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f452806b730 0x7f45280b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 >> 172.21.15.70:0/2444667858 conn(0x7f4528099a60 msgr2=0x7f45280ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 shutdown_connections 2024-04-22T01:16:46.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.298+0000 7f452f421640 1 -- 172.21.15.70:0/2444667858 wait complete. 2024-04-22T01:16:46.316 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-22T01:16:44.623127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251272,"kb_used_data":5972,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754723448,"statfs":{"total":773094113280,"available":772836810752,"internally_reserved":0,"allocated":6115328,"data_stored":3346864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.368931"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273487+0000","last_change":"2024-04-22T01:16:42.297132+0000","last_active":"2024-04-22T01:16:43.273487+0000","last_peered":"2024-04-22T01:16:43.273487+0000","last_clean":"2024-04-22T01:16:43.273487+0000","last_became_active":"2024-04-22T01:16:42.296705+0000","last_became_peered":"2024-04-22T01:16:42.296705+0000","last_unstale":"2024-04-22T01:16:43.273487+0000","last_undegraded":"2024-04-22T01:16:43.273487+0000","last_fullsized":"2024-04-22T01:16:43.273487+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:40.262829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587931+0000","last_change":"2024-04-22T01:16:42.328580+0000","last_active":"2024-04-22T01:16:43.587931+0000","last_peered":"2024-04-22T01:16:43.587931+0000","last_clean":"2024-04-22T01:16:43.587931+0000","last_became_active":"2024-04-22T01:16:42.322602+0000","last_became_peered":"2024-04-22T01:16:42.322602+0000","last_unstale":"2024-04-22T01:16:43.587931+0000","last_undegraded":"2024-04-22T01:16:43.587931+0000","last_fullsized":"2024-04-22T01:16:43.587931+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:58:39.203936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588404+0000","last_change":"2024-04-22T01:16:42.283515+0000","last_active":"2024-04-22T01:16:43.588404+0000","last_peered":"2024-04-22T01:16:43.588404+0000","last_clean":"2024-04-22T01:16:43.588404+0000","last_became_active":"2024-04-22T01:16:42.283099+0000","last_became_peered":"2024-04-22T01:16:42.283099+0000","last_unstale":"2024-04-22T01:16:43.588404+0000","last_undegraded":"2024-04-22T01:16:43.588404+0000","last_fullsized":"2024-04-22T01:16:43.588404+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:31.183960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587803+0000","last_change":"2024-04-22T01:16:42.322015+0000","last_active":"2024-04-22T01:16:43.587803+0000","last_peered":"2024-04-22T01:16:43.587803+0000","last_clean":"2024-04-22T01:16:43.587803+0000","last_became_active":"2024-04-22T01:16:42.320226+0000","last_became_peered":"2024-04-22T01:16:42.320226+0000","last_unstale":"2024-04-22T01:16:43.587803+0000","last_undegraded":"2024-04-22T01:16:43.587803+0000","last_fullsized":"2024-04-22T01:16:43.587803+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:14.845222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586483+0000","last_change":"2024-04-22T01:16:42.298812+0000","last_active":"2024-04-22T01:16:43.586483+0000","last_peered":"2024-04-22T01:16:43.586483+0000","last_clean":"2024-04-22T01:16:43.586483+0000","last_became_active":"2024-04-22T01:16:42.298003+0000","last_became_peered":"2024-04-22T01:16:42.298003+0000","last_unstale":"2024-04-22T01:16:43.586483+0000","last_undegraded":"2024-04-22T01:16:43.586483+0000","last_fullsized":"2024-04-22T01:16:43.586483+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:06.384082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272771+0000","last_change":"2024-04-22T01:16:42.291471+0000","last_active":"2024-04-22T01:16:43.272771+0000","last_peered":"2024-04-22T01:16:43.272771+0000","last_clean":"2024-04-22T01:16:43.272771+0000","last_became_active":"2024-04-22T01:16:42.290951+0000","last_became_peered":"2024-04-22T01:16:42.290951+0000","last_unstale":"2024-04-22T01:16:43.272771+0000","last_undegraded":"2024-04-22T01:16:43.272771+0000","last_fullsized":"2024-04-22T01:16:43.272771+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:02:25.597982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586706+0000","last_change":"2024-04-22T01:16:42.291719+0000","last_active":"2024-04-22T01:16:43.586706+0000","last_peered":"2024-04-22T01:16:43.586706+0000","last_clean":"2024-04-22T01:16:43.586706+0000","last_became_active":"2024-04-22T01:16:42.285461+0000","last_became_peered":"2024-04-22T01:16:42.285461+0000","last_unstale":"2024-04-22T01:16:43.586706+0000","last_undegraded":"2024-04-22T01:16:43.586706+0000","last_fullsized":"2024-04-22T01:16:43.586706+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.020114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588996+0000","last_change":"2024-04-22T01:16:42.316255+0000","last_active":"2024-04-22T01:16:43.588996+0000","last_peered":"2024-04-22T01:16:43.588996+0000","last_clean":"2024-04-22T01:16:43.588996+0000","last_became_active":"2024-04-22T01:16:42.315687+0000","last_became_peered":"2024-04-22T01:16:42.315687+0000","last_unstale":"2024-04-22T01:16:43.588996+0000","last_undegraded":"2024-04-22T01:16:43.588996+0000","last_fullsized":"2024-04-22T01:16:43.588996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:26.800461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591096+0000","last_change":"2024-04-22T01:16:42.316227+0000","last_active":"2024-04-22T01:16:43.591096+0000","last_peered":"2024-04-22T01:16:43.591096+0000","last_clean":"2024-04-22T01:16:43.591096+0000","last_became_active":"2024-04-22T01:16:42.315857+0000","last_became_peered":"2024-04-22T01:16:42.315857+0000","last_unstale":"2024-04-22T01:16:43.591096+0000","last_undegraded":"2024-04-22T01:16:43.591096+0000","last_fullsized":"2024-04-22T01:16:43.591096+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:25:30.883785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273149+0000","last_change":"2024-04-22T01:16:42.307041+0000","last_active":"2024-04-22T01:16:43.273149+0000","last_peered":"2024-04-22T01:16:43.273149+0000","last_clean":"2024-04-22T01:16:43.273149+0000","last_became_active":"2024-04-22T01:16:42.306524+0000","last_became_peered":"2024-04-22T01:16:42.306524+0000","last_unstale":"2024-04-22T01:16:43.273149+0000","last_undegraded":"2024-04-22T01:16:43.273149+0000","last_fullsized":"2024-04-22T01:16:43.273149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:10.171708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588199+0000","last_change":"2024-04-22T01:16:42.313655+0000","last_active":"2024-04-22T01:16:43.588199+0000","last_peered":"2024-04-22T01:16:43.588199+0000","last_clean":"2024-04-22T01:16:43.588199+0000","last_became_active":"2024-04-22T01:16:42.313160+0000","last_became_peered":"2024-04-22T01:16:42.313160+0000","last_unstale":"2024-04-22T01:16:43.588199+0000","last_undegraded":"2024-04-22T01:16:43.588199+0000","last_fullsized":"2024-04-22T01:16:43.588199+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:22.348897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590803+0000","last_change":"2024-04-22T01:16:42.308708+0000","last_active":"2024-04-22T01:16:43.590803+0000","last_peered":"2024-04-22T01:16:43.590803+0000","last_clean":"2024-04-22T01:16:43.590803+0000","last_became_active":"2024-04-22T01:16:42.308430+0000","last_became_peered":"2024-04-22T01:16:42.308430+0000","last_unstale":"2024-04-22T01:16:43.590803+0000","last_undegraded":"2024-04-22T01:16:43.590803+0000","last_fullsized":"2024-04-22T01:16:43.590803+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:17.745700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272472+0000","last_change":"2024-04-22T01:16:42.307164+0000","last_active":"2024-04-22T01:16:43.272472+0000","last_peered":"2024-04-22T01:16:43.272472+0000","last_clean":"2024-04-22T01:16:43.272472+0000","last_became_active":"2024-04-22T01:16:42.306668+0000","last_became_peered":"2024-04-22T01:16:42.306668+0000","last_unstale":"2024-04-22T01:16:43.272472+0000","last_undegraded":"2024-04-22T01:16:43.272472+0000","last_fullsized":"2024-04-22T01:16:43.272472+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:33.380606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587053+0000","last_change":"2024-04-22T01:16:42.314168+0000","last_active":"2024-04-22T01:16:43.587053+0000","last_peered":"2024-04-22T01:16:43.587053+0000","last_clean":"2024-04-22T01:16:43.587053+0000","last_became_active":"2024-04-22T01:16:42.313798+0000","last_became_peered":"2024-04-22T01:16:42.313798+0000","last_unstale":"2024-04-22T01:16:43.587053+0000","last_undegraded":"2024-04-22T01:16:43.587053+0000","last_fullsized":"2024-04-22T01:16:43.587053+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:26.800461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586764+0000","last_change":"2024-04-22T01:16:42.322366+0000","last_active":"2024-04-22T01:16:43.586764+0000","last_peered":"2024-04-22T01:16:43.586764+0000","last_clean":"2024-04-22T01:16:43.586764+0000","last_became_active":"2024-04-22T01:16:42.321520+0000","last_became_peered":"2024-04-22T01:16:42.321520+0000","last_unstale":"2024-04-22T01:16:43.586764+0000","last_undegraded":"2024-04-22T01:16:43.586764+0000","last_fullsized":"2024-04-22T01:16:43.586764+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:21.076434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272592+0000","last_change":"2024-04-22T01:16:42.308132+0000","last_active":"2024-04-22T01:16:43.272592+0000","last_peered":"2024-04-22T01:16:43.272592+0000","last_clean":"2024-04-22T01:16:43.272592+0000","last_became_active":"2024-04-22T01:16:42.307603+0000","last_became_peered":"2024-04-22T01:16:42.307603+0000","last_unstale":"2024-04-22T01:16:43.272592+0000","last_undegraded":"2024-04-22T01:16:43.272592+0000","last_fullsized":"2024-04-22T01:16:43.272592+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:15.786674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589077+0000","last_change":"2024-04-22T01:16:42.314883+0000","last_active":"2024-04-22T01:16:43.589077+0000","last_peered":"2024-04-22T01:16:43.589077+0000","last_clean":"2024-04-22T01:16:43.589077+0000","last_became_active":"2024-04-22T01:16:42.314344+0000","last_became_peered":"2024-04-22T01:16:42.314344+0000","last_unstale":"2024-04-22T01:16:43.589077+0000","last_undegraded":"2024-04-22T01:16:43.589077+0000","last_fullsized":"2024-04-22T01:16:43.589077+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:07.621817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588076+0000","last_change":"2024-04-22T01:16:42.328618+0000","last_active":"2024-04-22T01:16:43.588076+0000","last_peered":"2024-04-22T01:16:43.588076+0000","last_clean":"2024-04-22T01:16:43.588076+0000","last_became_active":"2024-04-22T01:16:42.323472+0000","last_became_peered":"2024-04-22T01:16:42.323472+0000","last_unstale":"2024-04-22T01:16:43.588076+0000","last_undegraded":"2024-04-22T01:16:43.588076+0000","last_fullsized":"2024-04-22T01:16:43.588076+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:19.311224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588535+0000","last_change":"2024-04-22T01:16:42.313966+0000","last_active":"2024-04-22T01:16:43.588535+0000","last_peered":"2024-04-22T01:16:43.588535+0000","last_clean":"2024-04-22T01:16:43.588535+0000","last_became_active":"2024-04-22T01:16:42.299795+0000","last_became_peered":"2024-04-22T01:16:42.299795+0000","last_unstale":"2024-04-22T01:16:43.588535+0000","last_undegraded":"2024-04-22T01:16:43.588535+0000","last_fullsized":"2024-04-22T01:16:43.588535+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:17.745700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276254+0000","last_change":"2024-04-22T01:16:42.309101+0000","last_active":"2024-04-22T01:16:43.276254+0000","last_peered":"2024-04-22T01:16:43.276254+0000","last_clean":"2024-04-22T01:16:43.276254+0000","last_became_active":"2024-04-22T01:16:42.307750+0000","last_became_peered":"2024-04-22T01:16:42.307750+0000","last_unstale":"2024-04-22T01:16:43.276254+0000","last_undegraded":"2024-04-22T01:16:43.276254+0000","last_fullsized":"2024-04-22T01:16:43.276254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:03.530479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276084+0000","last_change":"2024-04-22T01:16:42.306145+0000","last_active":"2024-04-22T01:16:43.276084+0000","last_peered":"2024-04-22T01:16:43.276084+0000","last_clean":"2024-04-22T01:16:43.276084+0000","last_became_active":"2024-04-22T01:16:42.304669+0000","last_became_peered":"2024-04-22T01:16:42.304669+0000","last_unstale":"2024-04-22T01:16:43.276084+0000","last_undegraded":"2024-04-22T01:16:43.276084+0000","last_fullsized":"2024-04-22T01:16:43.276084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:15.786674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588531+0000","last_change":"2024-04-22T01:16:42.299940+0000","last_active":"2024-04-22T01:16:43.588531+0000","last_peered":"2024-04-22T01:16:43.588531+0000","last_clean":"2024-04-22T01:16:43.588531+0000","last_became_active":"2024-04-22T01:16:42.298977+0000","last_became_peered":"2024-04-22T01:16:42.298977+0000","last_unstale":"2024-04-22T01:16:43.588531+0000","last_undegraded":"2024-04-22T01:16:43.588531+0000","last_fullsized":"2024-04-22T01:16:43.588531+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.020114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587443+0000","last_change":"2024-04-22T01:16:42.314427+0000","last_active":"2024-04-22T01:16:43.587443+0000","last_peered":"2024-04-22T01:16:43.587443+0000","last_clean":"2024-04-22T01:16:43.587443+0000","last_became_active":"2024-04-22T01:16:42.313443+0000","last_became_peered":"2024-04-22T01:16:42.313443+0000","last_unstale":"2024-04-22T01:16:43.587443+0000","last_undegraded":"2024-04-22T01:16:43.587443+0000","last_fullsized":"2024-04-22T01:16:43.587443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:45.511733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586628+0000","last_change":"2024-04-22T01:16:42.284569+0000","last_active":"2024-04-22T01:16:43.586628+0000","last_peered":"2024-04-22T01:16:43.586628+0000","last_clean":"2024-04-22T01:16:43.586628+0000","last_became_active":"2024-04-22T01:16:42.284055+0000","last_became_peered":"2024-04-22T01:16:42.284055+0000","last_unstale":"2024-04-22T01:16:43.586628+0000","last_undegraded":"2024-04-22T01:16:43.586628+0000","last_fullsized":"2024-04-22T01:16:43.586628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:34.517785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589497+0000","last_change":"2024-04-22T01:16:42.313172+0000","last_active":"2024-04-22T01:16:43.589497+0000","last_peered":"2024-04-22T01:16:43.589497+0000","last_clean":"2024-04-22T01:16:43.589497+0000","last_became_active":"2024-04-22T01:16:42.299260+0000","last_became_peered":"2024-04-22T01:16:42.299260+0000","last_unstale":"2024-04-22T01:16:43.589497+0000","last_undegraded":"2024-04-22T01:16:43.589497+0000","last_fullsized":"2024-04-22T01:16:43.589497+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:32.752089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590743+0000","last_change":"2024-04-22T01:16:42.295341+0000","last_active":"2024-04-22T01:16:43.590743+0000","last_peered":"2024-04-22T01:16:43.590743+0000","last_clean":"2024-04-22T01:16:43.590743+0000","last_became_active":"2024-04-22T01:16:42.294922+0000","last_became_peered":"2024-04-22T01:16:42.294922+0000","last_unstale":"2024-04-22T01:16:43.590743+0000","last_undegraded":"2024-04-22T01:16:43.590743+0000","last_fullsized":"2024-04-22T01:16:43.590743+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:31.183960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273528+0000","last_change":"2024-04-22T01:16:42.319657+0000","last_active":"2024-04-22T01:16:43.273528+0000","last_peered":"2024-04-22T01:16:43.273528+0000","last_clean":"2024-04-22T01:16:43.273528+0000","last_became_active":"2024-04-22T01:16:42.319353+0000","last_became_peered":"2024-04-22T01:16:42.319353+0000","last_unstale":"2024-04-22T01:16:43.273528+0000","last_undegraded":"2024-04-22T01:16:43.273528+0000","last_fullsized":"2024-04-22T01:16:43.273528+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:33.750136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276370+0000","last_change":"2024-04-22T01:16:42.287849+0000","last_active":"2024-04-22T01:16:43.276370+0000","last_peered":"2024-04-22T01:16:43.276370+0000","last_clean":"2024-04-22T01:16:43.276370+0000","last_became_active":"2024-04-22T01:16:42.287102+0000","last_became_peered":"2024-04-22T01:16:42.287102+0000","last_unstale":"2024-04-22T01:16:43.276370+0000","last_undegraded":"2024-04-22T01:16:43.276370+0000","last_fullsized":"2024-04-22T01:16:43.276370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:40.004739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587731+0000","last_change":"2024-04-22T01:16:42.318878+0000","last_active":"2024-04-22T01:16:43.587731+0000","last_peered":"2024-04-22T01:16:43.587731+0000","last_clean":"2024-04-22T01:16:43.587731+0000","last_became_active":"2024-04-22T01:16:42.318484+0000","last_became_peered":"2024-04-22T01:16:42.318484+0000","last_unstale":"2024-04-22T01:16:43.587731+0000","last_undegraded":"2024-04-22T01:16:43.587731+0000","last_fullsized":"2024-04-22T01:16:43.587731+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:18.176769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586921+0000","last_change":"2024-04-22T01:16:42.321572+0000","last_active":"2024-04-22T01:16:43.586921+0000","last_peered":"2024-04-22T01:16:43.586921+0000","last_clean":"2024-04-22T01:16:43.586921+0000","last_became_active":"2024-04-22T01:16:42.321097+0000","last_became_peered":"2024-04-22T01:16:42.321097+0000","last_unstale":"2024-04-22T01:16:43.586921+0000","last_undegraded":"2024-04-22T01:16:43.586921+0000","last_fullsized":"2024-04-22T01:16:43.586921+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:52:44.520751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587074+0000","last_change":"2024-04-22T01:16:42.298826+0000","last_active":"2024-04-22T01:16:43.587074+0000","last_peered":"2024-04-22T01:16:43.587074+0000","last_clean":"2024-04-22T01:16:43.587074+0000","last_became_active":"2024-04-22T01:16:42.298488+0000","last_became_peered":"2024-04-22T01:16:42.298488+0000","last_unstale":"2024-04-22T01:16:43.587074+0000","last_undegraded":"2024-04-22T01:16:43.587074+0000","last_fullsized":"2024-04-22T01:16:43.587074+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:19.712873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273837+0000","last_change":"2024-04-22T01:16:42.297976+0000","last_active":"2024-04-22T01:16:43.273837+0000","last_peered":"2024-04-22T01:16:43.273837+0000","last_clean":"2024-04-22T01:16:43.273837+0000","last_became_active":"2024-04-22T01:16:42.294104+0000","last_became_peered":"2024-04-22T01:16:42.294104+0000","last_unstale":"2024-04-22T01:16:43.273837+0000","last_undegraded":"2024-04-22T01:16:43.273837+0000","last_fullsized":"2024-04-22T01:16:43.273837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:33.750136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586547+0000","last_change":"2024-04-22T01:16:42.329218+0000","last_active":"2024-04-22T01:16:43.586547+0000","last_peered":"2024-04-22T01:16:43.586547+0000","last_clean":"2024-04-22T01:16:43.586547+0000","last_became_active":"2024-04-22T01:16:42.323604+0000","last_became_peered":"2024-04-22T01:16:42.323604+0000","last_unstale":"2024-04-22T01:16:43.586547+0000","last_undegraded":"2024-04-22T01:16:43.586547+0000","last_fullsized":"2024-04-22T01:16:43.586547+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:33.750136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276266+0000","last_change":"2024-04-22T01:16:41.299226+0000","last_active":"2024-04-22T01:16:43.276266+0000","last_peered":"2024-04-22T01:16:43.276266+0000","last_clean":"2024-04-22T01:16:43.276266+0000","last_became_active":"2024-04-22T01:16:41.298647+0000","last_became_peered":"2024-04-22T01:16:41.298647+0000","last_unstale":"2024-04-22T01:16:43.276266+0000","last_undegraded":"2024-04-22T01:16:43.276266+0000","last_fullsized":"2024-04-22T01:16:43.276266+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:45:25.792994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275952+0000","last_change":"2024-04-22T01:16:42.318792+0000","last_active":"2024-04-22T01:16:43.275952+0000","last_peered":"2024-04-22T01:16:43.275952+0000","last_clean":"2024-04-22T01:16:43.275952+0000","last_became_active":"2024-04-22T01:16:42.317988+0000","last_became_peered":"2024-04-22T01:16:42.317988+0000","last_unstale":"2024-04-22T01:16:43.275952+0000","last_undegraded":"2024-04-22T01:16:43.275952+0000","last_fullsized":"2024-04-22T01:16:43.275952+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:14.845222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273282+0000","last_change":"2024-04-22T01:16:41.286305+0000","last_active":"2024-04-22T01:16:43.273282+0000","last_peered":"2024-04-22T01:16:43.273282+0000","last_clean":"2024-04-22T01:16:43.273282+0000","last_became_active":"2024-04-22T01:16:41.285839+0000","last_became_peered":"2024-04-22T01:16:41.285839+0000","last_unstale":"2024-04-22T01:16:43.273282+0000","last_undegraded":"2024-04-22T01:16:43.273282+0000","last_fullsized":"2024-04-22T01:16:43.273282+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:13.230682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275553+0000","last_change":"2024-04-22T01:16:42.308126+0000","last_active":"2024-04-22T01:16:43.275553+0000","last_peered":"2024-04-22T01:16:43.275553+0000","last_clean":"2024-04-22T01:16:43.275553+0000","last_became_active":"2024-04-22T01:16:42.307387+0000","last_became_peered":"2024-04-22T01:16:42.307387+0000","last_unstale":"2024-04-22T01:16:43.275553+0000","last_undegraded":"2024-04-22T01:16:43.275553+0000","last_fullsized":"2024-04-22T01:16:43.275553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:33.380606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273451+0000","last_change":"2024-04-22T01:16:41.299592+0000","last_active":"2024-04-22T01:16:43.273451+0000","last_peered":"2024-04-22T01:16:43.273451+0000","last_clean":"2024-04-22T01:16:43.273451+0000","last_became_active":"2024-04-22T01:16:41.299241+0000","last_became_peered":"2024-04-22T01:16:41.299241+0000","last_unstale":"2024-04-22T01:16:43.273451+0000","last_undegraded":"2024-04-22T01:16:43.273451+0000","last_fullsized":"2024-04-22T01:16:43.273451+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:14:25.445161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589758+0000","last_change":"2024-04-22T01:16:42.293507+0000","last_active":"2024-04-22T01:16:43.589758+0000","last_peered":"2024-04-22T01:16:43.589758+0000","last_clean":"2024-04-22T01:16:43.589758+0000","last_became_active":"2024-04-22T01:16:42.292862+0000","last_became_peered":"2024-04-22T01:16:42.292862+0000","last_unstale":"2024-04-22T01:16:43.589758+0000","last_undegraded":"2024-04-22T01:16:43.589758+0000","last_fullsized":"2024-04-22T01:16:43.589758+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:22.563055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275271+0000","last_change":"2024-04-22T01:16:41.294821+0000","last_active":"2024-04-22T01:16:43.275271+0000","last_peered":"2024-04-22T01:16:43.275271+0000","last_clean":"2024-04-22T01:16:43.275271+0000","last_became_active":"2024-04-22T01:16:41.294128+0000","last_became_peered":"2024-04-22T01:16:41.294128+0000","last_unstale":"2024-04-22T01:16:43.275271+0000","last_undegraded":"2024-04-22T01:16:43.275271+0000","last_fullsized":"2024-04-22T01:16:43.275271+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:44.052756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587481+0000","last_change":"2024-04-22T01:16:42.280148+0000","last_active":"2024-04-22T01:16:43.587481+0000","last_peered":"2024-04-22T01:16:43.587481+0000","last_clean":"2024-04-22T01:16:43.587481+0000","last_became_active":"2024-04-22T01:16:42.279293+0000","last_became_peered":"2024-04-22T01:16:42.279293+0000","last_unstale":"2024-04-22T01:16:43.587481+0000","last_undegraded":"2024-04-22T01:16:43.587481+0000","last_fullsized":"2024-04-22T01:16:43.587481+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:33.380606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586729+0000","last_change":"2024-04-22T01:16:41.289933+0000","last_active":"2024-04-22T01:16:43.586729+0000","last_peered":"2024-04-22T01:16:43.586729+0000","last_clean":"2024-04-22T01:16:43.586729+0000","last_became_active":"2024-04-22T01:16:41.289583+0000","last_became_peered":"2024-04-22T01:16:41.289583+0000","last_unstale":"2024-04-22T01:16:43.586729+0000","last_undegraded":"2024-04-22T01:16:43.586729+0000","last_fullsized":"2024-04-22T01:16:43.586729+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:05.380345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588135+0000","last_change":"2024-04-22T01:16:42.329811+0000","last_active":"2024-04-22T01:16:43.588135+0000","last_peered":"2024-04-22T01:16:43.588135+0000","last_clean":"2024-04-22T01:16:43.588135+0000","last_became_active":"2024-04-22T01:16:42.329087+0000","last_became_peered":"2024-04-22T01:16:42.329087+0000","last_unstale":"2024-04-22T01:16:43.588135+0000","last_undegraded":"2024-04-22T01:16:43.588135+0000","last_fullsized":"2024-04-22T01:16:43.588135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:04.039485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586411+0000","last_change":"2024-04-22T01:16:41.299672+0000","last_active":"2024-04-22T01:16:43.586411+0000","last_peered":"2024-04-22T01:16:43.586411+0000","last_clean":"2024-04-22T01:16:43.586411+0000","last_became_active":"2024-04-22T01:16:41.298905+0000","last_became_peered":"2024-04-22T01:16:41.298905+0000","last_unstale":"2024-04-22T01:16:43.586411+0000","last_undegraded":"2024-04-22T01:16:43.586411+0000","last_fullsized":"2024-04-22T01:16:43.586411+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:14:25.445161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591215+0000","last_change":"2024-04-22T01:16:42.295190+0000","last_active":"2024-04-22T01:16:43.591215+0000","last_peered":"2024-04-22T01:16:43.591215+0000","last_clean":"2024-04-22T01:16:43.591215+0000","last_became_active":"2024-04-22T01:16:42.294626+0000","last_became_peered":"2024-04-22T01:16:42.294626+0000","last_unstale":"2024-04-22T01:16:43.591215+0000","last_undegraded":"2024-04-22T01:16:43.591215+0000","last_fullsized":"2024-04-22T01:16:43.591215+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:33.573703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275345+0000","last_change":"2024-04-22T01:16:41.302031+0000","last_active":"2024-04-22T01:16:43.275345+0000","last_peered":"2024-04-22T01:16:43.275345+0000","last_clean":"2024-04-22T01:16:43.275345+0000","last_became_active":"2024-04-22T01:16:41.301420+0000","last_became_peered":"2024-04-22T01:16:41.301420+0000","last_unstale":"2024-04-22T01:16:43.275345+0000","last_undegraded":"2024-04-22T01:16:43.275345+0000","last_fullsized":"2024-04-22T01:16:43.275345+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:12.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587845+0000","last_change":"2024-04-22T01:16:42.328477+0000","last_active":"2024-04-22T01:16:43.587845+0000","last_peered":"2024-04-22T01:16:43.587845+0000","last_clean":"2024-04-22T01:16:43.587845+0000","last_became_active":"2024-04-22T01:16:42.323323+0000","last_became_peered":"2024-04-22T01:16:42.323323+0000","last_unstale":"2024-04-22T01:16:43.587845+0000","last_undegraded":"2024-04-22T01:16:43.587845+0000","last_fullsized":"2024-04-22T01:16:43.587845+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:21:23.616725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586864+0000","last_change":"2024-04-22T01:16:41.301533+0000","last_active":"2024-04-22T01:16:43.586864+0000","last_peered":"2024-04-22T01:16:43.586864+0000","last_clean":"2024-04-22T01:16:43.586864+0000","last_became_active":"2024-04-22T01:16:41.301042+0000","last_became_peered":"2024-04-22T01:16:41.301042+0000","last_unstale":"2024-04-22T01:16:43.586864+0000","last_undegraded":"2024-04-22T01:16:43.586864+0000","last_fullsized":"2024-04-22T01:16:43.586864+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:55:52.957089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586097+0000","last_change":"2024-04-22T01:16:42.295270+0000","last_active":"2024-04-22T01:16:43.586097+0000","last_peered":"2024-04-22T01:16:43.586097+0000","last_clean":"2024-04-22T01:16:43.586097+0000","last_became_active":"2024-04-22T01:16:42.294393+0000","last_became_peered":"2024-04-22T01:16:42.294393+0000","last_unstale":"2024-04-22T01:16:43.586097+0000","last_undegraded":"2024-04-22T01:16:43.586097+0000","last_fullsized":"2024-04-22T01:16:43.586097+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:32.555943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588245+0000","last_change":"2024-04-22T01:16:41.305789+0000","last_active":"2024-04-22T01:16:43.588245+0000","last_peered":"2024-04-22T01:16:43.588245+0000","last_clean":"2024-04-22T01:16:43.588245+0000","last_became_active":"2024-04-22T01:16:41.305231+0000","last_became_peered":"2024-04-22T01:16:41.305231+0000","last_unstale":"2024-04-22T01:16:43.588245+0000","last_undegraded":"2024-04-22T01:16:43.588245+0000","last_fullsized":"2024-04-22T01:16:43.588245+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:31.552205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274210+0000","last_change":"2024-04-22T01:16:42.292497+0000","last_active":"2024-04-22T01:16:43.274210+0000","last_peered":"2024-04-22T01:16:43.274210+0000","last_clean":"2024-04-22T01:16:43.274210+0000","last_became_active":"2024-04-22T01:16:42.291743+0000","last_became_peered":"2024-04-22T01:16:42.291743+0000","last_unstale":"2024-04-22T01:16:43.274210+0000","last_undegraded":"2024-04-22T01:16:43.274210+0000","last_fullsized":"2024-04-22T01:16:43.274210+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:21.076434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587958+0000","last_change":"2024-04-22T01:16:41.279655+0000","last_active":"2024-04-22T01:16:43.587958+0000","last_peered":"2024-04-22T01:16:43.587958+0000","last_clean":"2024-04-22T01:16:43.587958+0000","last_became_active":"2024-04-22T01:16:41.278708+0000","last_became_peered":"2024-04-22T01:16:41.278708+0000","last_unstale":"2024-04-22T01:16:43.587958+0000","last_undegraded":"2024-04-22T01:16:43.587958+0000","last_fullsized":"2024-04-22T01:16:43.587958+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:12.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587457+0000","last_change":"2024-04-22T01:16:42.318567+0000","last_active":"2024-04-22T01:16:43.587457+0000","last_peered":"2024-04-22T01:16:43.587457+0000","last_clean":"2024-04-22T01:16:43.587457+0000","last_became_active":"2024-04-22T01:16:42.318289+0000","last_became_peered":"2024-04-22T01:16:42.318289+0000","last_unstale":"2024-04-22T01:16:43.587457+0000","last_undegraded":"2024-04-22T01:16:43.587457+0000","last_fullsized":"2024-04-22T01:16:43.587457+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:22.563055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588014+0000","last_change":"2024-04-22T01:16:41.300777+0000","last_active":"2024-04-22T01:16:43.588014+0000","last_peered":"2024-04-22T01:16:43.588014+0000","last_clean":"2024-04-22T01:16:43.588014+0000","last_became_active":"2024-04-22T01:16:41.299861+0000","last_became_peered":"2024-04-22T01:16:41.299861+0000","last_unstale":"2024-04-22T01:16:43.588014+0000","last_undegraded":"2024-04-22T01:16:43.588014+0000","last_fullsized":"2024-04-22T01:16:43.588014+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:05.380345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587198+0000","last_change":"2024-04-22T01:16:42.280582+0000","last_active":"2024-04-22T01:16:43.587198+0000","last_peered":"2024-04-22T01:16:43.587198+0000","last_clean":"2024-04-22T01:16:43.587198+0000","last_became_active":"2024-04-22T01:16:42.279459+0000","last_became_peered":"2024-04-22T01:16:42.279459+0000","last_unstale":"2024-04-22T01:16:43.587198+0000","last_undegraded":"2024-04-22T01:16:43.587198+0000","last_fullsized":"2024-04-22T01:16:43.587198+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:45.056493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590798+0000","last_change":"2024-04-22T01:16:41.298783+0000","last_active":"2024-04-22T01:16:43.590798+0000","last_peered":"2024-04-22T01:16:43.590798+0000","last_clean":"2024-04-22T01:16:43.590798+0000","last_became_active":"2024-04-22T01:16:41.294126+0000","last_became_peered":"2024-04-22T01:16:41.294126+0000","last_unstale":"2024-04-22T01:16:43.590798+0000","last_undegraded":"2024-04-22T01:16:43.590798+0000","last_fullsized":"2024-04-22T01:16:43.590798+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:02.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274791+0000","last_change":"2024-04-22T01:16:42.316992+0000","last_active":"2024-04-22T01:16:43.274791+0000","last_peered":"2024-04-22T01:16:43.274791+0000","last_clean":"2024-04-22T01:16:43.274791+0000","last_became_active":"2024-04-22T01:16:42.316319+0000","last_became_peered":"2024-04-22T01:16:42.316319+0000","last_unstale":"2024-04-22T01:16:43.274791+0000","last_undegraded":"2024-04-22T01:16:43.274791+0000","last_fullsized":"2024-04-22T01:16:43.274791+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:21:23.616725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589944+0000","last_change":"2024-04-22T01:16:41.296260+0000","last_active":"2024-04-22T01:16:43.589944+0000","last_peered":"2024-04-22T01:16:43.589944+0000","last_clean":"2024-04-22T01:16:43.589944+0000","last_became_active":"2024-04-22T01:16:41.295404+0000","last_became_peered":"2024-04-22T01:16:41.295404+0000","last_unstale":"2024-04-22T01:16:43.589944+0000","last_undegraded":"2024-04-22T01:16:43.589944+0000","last_fullsized":"2024-04-22T01:16:43.589944+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:44.052756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588767+0000","last_change":"2024-04-22T01:16:42.314778+0000","last_active":"2024-04-22T01:16:43.588767+0000","last_peered":"2024-04-22T01:16:43.588767+0000","last_clean":"2024-04-22T01:16:43.588767+0000","last_became_active":"2024-04-22T01:16:42.314327+0000","last_became_peered":"2024-04-22T01:16:42.314327+0000","last_unstale":"2024-04-22T01:16:43.588767+0000","last_undegraded":"2024-04-22T01:16:43.588767+0000","last_fullsized":"2024-04-22T01:16:43.588767+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:25:30.883785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275861+0000","last_change":"2024-04-22T01:16:41.303302+0000","last_active":"2024-04-22T01:16:43.275861+0000","last_peered":"2024-04-22T01:16:43.275861+0000","last_clean":"2024-04-22T01:16:43.275861+0000","last_became_active":"2024-04-22T01:16:41.302610+0000","last_became_peered":"2024-04-22T01:16:41.302610+0000","last_unstale":"2024-04-22T01:16:43.275861+0000","last_undegraded":"2024-04-22T01:16:43.275861+0000","last_fullsized":"2024-04-22T01:16:43.275861+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:14:25.445161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591189+0000","last_change":"2024-04-22T01:16:42.313396+0000","last_active":"2024-04-22T01:16:43.591189+0000","last_peered":"2024-04-22T01:16:43.591189+0000","last_clean":"2024-04-22T01:16:43.591189+0000","last_became_active":"2024-04-22T01:16:42.313066+0000","last_became_peered":"2024-04-22T01:16:42.313066+0000","last_unstale":"2024-04-22T01:16:43.591189+0000","last_undegraded":"2024-04-22T01:16:43.591189+0000","last_fullsized":"2024-04-22T01:16:43.591189+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:09.047831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586754+0000","last_change":"2024-04-22T01:16:41.304915+0000","last_active":"2024-04-22T01:16:43.586754+0000","last_peered":"2024-04-22T01:16:43.586754+0000","last_clean":"2024-04-22T01:16:43.586754+0000","last_became_active":"2024-04-22T01:16:41.304190+0000","last_became_peered":"2024-04-22T01:16:41.304190+0000","last_unstale":"2024-04-22T01:16:43.586754+0000","last_undegraded":"2024-04-22T01:16:43.586754+0000","last_fullsized":"2024-04-22T01:16:43.586754+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:42.939970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589381+0000","last_change":"2024-04-22T01:16:42.314668+0000","last_active":"2024-04-22T01:16:43.589381+0000","last_peered":"2024-04-22T01:16:43.589381+0000","last_clean":"2024-04-22T01:16:43.589381+0000","last_became_active":"2024-04-22T01:16:42.314048+0000","last_became_peered":"2024-04-22T01:16:42.314048+0000","last_unstale":"2024-04-22T01:16:43.589381+0000","last_undegraded":"2024-04-22T01:16:43.589381+0000","last_fullsized":"2024-04-22T01:16:43.589381+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:33.573703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589473+0000","last_change":"2024-04-22T01:16:41.294520+0000","last_active":"2024-04-22T01:16:43.589473+0000","last_peered":"2024-04-22T01:16:43.589473+0000","last_clean":"2024-04-22T01:16:43.589473+0000","last_became_active":"2024-04-22T01:16:41.294221+0000","last_became_peered":"2024-04-22T01:16:41.294221+0000","last_unstale":"2024-04-22T01:16:43.589473+0000","last_undegraded":"2024-04-22T01:16:43.589473+0000","last_fullsized":"2024-04-22T01:16:43.589473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:09.167971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589003+0000","last_change":"2024-04-22T01:16:42.296959+0000","last_active":"2024-04-22T01:16:43.589003+0000","last_peered":"2024-04-22T01:16:43.589003+0000","last_clean":"2024-04-22T01:16:43.589003+0000","last_became_active":"2024-04-22T01:16:42.296698+0000","last_became_peered":"2024-04-22T01:16:42.296698+0000","last_unstale":"2024-04-22T01:16:43.589003+0000","last_undegraded":"2024-04-22T01:16:43.589003+0000","last_fullsized":"2024-04-22T01:16:43.589003+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:19.712873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589547+0000","last_change":"2024-04-22T01:16:41.300743+0000","last_active":"2024-04-22T01:16:43.589547+0000","last_peered":"2024-04-22T01:16:43.589547+0000","last_clean":"2024-04-22T01:16:43.589547+0000","last_became_active":"2024-04-22T01:16:41.300222+0000","last_became_peered":"2024-04-22T01:16:41.300222+0000","last_unstale":"2024-04-22T01:16:43.589547+0000","last_undegraded":"2024-04-22T01:16:43.589547+0000","last_fullsized":"2024-04-22T01:16:43.589547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:14.782936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272650+0000","last_change":"2024-04-22T01:16:42.300582+0000","last_active":"2024-04-22T01:16:43.272650+0000","last_peered":"2024-04-22T01:16:43.272650+0000","last_clean":"2024-04-22T01:16:43.272650+0000","last_became_active":"2024-04-22T01:16:42.300286+0000","last_became_peered":"2024-04-22T01:16:42.300286+0000","last_unstale":"2024-04-22T01:16:43.272650+0000","last_undegraded":"2024-04-22T01:16:43.272650+0000","last_fullsized":"2024-04-22T01:16:43.272650+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:03.530479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590547+0000","last_change":"2024-04-22T01:16:41.293170+0000","last_active":"2024-04-22T01:16:43.590547+0000","last_peered":"2024-04-22T01:16:43.590547+0000","last_clean":"2024-04-22T01:16:43.590547+0000","last_became_active":"2024-04-22T01:16:41.292688+0000","last_became_peered":"2024-04-22T01:16:41.292688+0000","last_unstale":"2024-04-22T01:16:43.590547+0000","last_undegraded":"2024-04-22T01:16:43.590547+0000","last_fullsized":"2024-04-22T01:16:43.590547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:54.914899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586279+0000","last_change":"2024-04-22T01:16:42.297878+0000","last_active":"2024-04-22T01:16:43.586279+0000","last_peered":"2024-04-22T01:16:43.586279+0000","last_clean":"2024-04-22T01:16:43.586279+0000","last_became_active":"2024-04-22T01:16:42.297155+0000","last_became_peered":"2024-04-22T01:16:42.297155+0000","last_unstale":"2024-04-22T01:16:43.586279+0000","last_undegraded":"2024-04-22T01:16:43.586279+0000","last_fullsized":"2024-04-22T01:16:43.586279+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:43.943708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276775+0000","last_change":"2024-04-22T01:16:41.293911+0000","last_active":"2024-04-22T01:16:43.276775+0000","last_peered":"2024-04-22T01:16:43.276775+0000","last_clean":"2024-04-22T01:16:43.276775+0000","last_became_active":"2024-04-22T01:16:41.293166+0000","last_became_peered":"2024-04-22T01:16:41.293166+0000","last_unstale":"2024-04-22T01:16:43.276775+0000","last_undegraded":"2024-04-22T01:16:43.276775+0000","last_fullsized":"2024-04-22T01:16:43.276775+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:31.626341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272678+0000","last_change":"2024-04-22T01:16:42.292637+0000","last_active":"2024-04-22T01:16:43.272678+0000","last_peered":"2024-04-22T01:16:43.272678+0000","last_clean":"2024-04-22T01:16:43.272678+0000","last_became_active":"2024-04-22T01:16:42.292153+0000","last_became_peered":"2024-04-22T01:16:42.292153+0000","last_unstale":"2024-04-22T01:16:43.272678+0000","last_undegraded":"2024-04-22T01:16:43.272678+0000","last_fullsized":"2024-04-22T01:16:43.272678+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:40.004739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276503+0000","last_change":"2024-04-22T01:16:41.306902+0000","last_active":"2024-04-22T01:16:43.276503+0000","last_peered":"2024-04-22T01:16:43.276503+0000","last_clean":"2024-04-22T01:16:43.276503+0000","last_became_active":"2024-04-22T01:16:41.306227+0000","last_became_peered":"2024-04-22T01:16:41.306227+0000","last_unstale":"2024-04-22T01:16:43.276503+0000","last_undegraded":"2024-04-22T01:16:43.276503+0000","last_fullsized":"2024-04-22T01:16:43.276503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.432697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275046+0000","last_change":"2024-04-22T01:16:42.284747+0000","last_active":"2024-04-22T01:16:43.275046+0000","last_peered":"2024-04-22T01:16:43.275046+0000","last_clean":"2024-04-22T01:16:43.275046+0000","last_became_active":"2024-04-22T01:16:42.284004+0000","last_became_peered":"2024-04-22T01:16:42.284004+0000","last_unstale":"2024-04-22T01:16:43.275046+0000","last_undegraded":"2024-04-22T01:16:43.275046+0000","last_fullsized":"2024-04-22T01:16:43.275046+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:58:39.203936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272791+0000","last_change":"2024-04-22T01:16:41.288670+0000","last_active":"2024-04-22T01:16:43.272791+0000","last_peered":"2024-04-22T01:16:43.272791+0000","last_clean":"2024-04-22T01:16:43.272791+0000","last_became_active":"2024-04-22T01:16:41.288375+0000","last_became_peered":"2024-04-22T01:16:41.288375+0000","last_unstale":"2024-04-22T01:16:43.272791+0000","last_undegraded":"2024-04-22T01:16:43.272791+0000","last_fullsized":"2024-04-22T01:16:43.272791+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:12.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276513+0000","last_change":"2024-04-22T01:16:42.311799+0000","last_active":"2024-04-22T01:16:43.276513+0000","last_peered":"2024-04-22T01:16:43.276513+0000","last_clean":"2024-04-22T01:16:43.276513+0000","last_became_active":"2024-04-22T01:16:42.311064+0000","last_became_peered":"2024-04-22T01:16:42.311064+0000","last_unstale":"2024-04-22T01:16:43.276513+0000","last_undegraded":"2024-04-22T01:16:43.276513+0000","last_fullsized":"2024-04-22T01:16:43.276513+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:02:25.597982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588794+0000","last_change":"2024-04-22T01:16:41.300766+0000","last_active":"2024-04-22T01:16:43.588794+0000","last_peered":"2024-04-22T01:16:43.588794+0000","last_clean":"2024-04-22T01:16:43.588794+0000","last_became_active":"2024-04-22T01:16:41.300346+0000","last_became_peered":"2024-04-22T01:16:41.300346+0000","last_unstale":"2024-04-22T01:16:43.588794+0000","last_undegraded":"2024-04-22T01:16:43.588794+0000","last_fullsized":"2024-04-22T01:16:43.588794+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:39.001001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589508+0000","last_change":"2024-04-22T01:16:42.319426+0000","last_active":"2024-04-22T01:16:43.589508+0000","last_peered":"2024-04-22T01:16:43.589508+0000","last_clean":"2024-04-22T01:16:43.589508+0000","last_became_active":"2024-04-22T01:16:42.315069+0000","last_became_peered":"2024-04-22T01:16:42.315069+0000","last_unstale":"2024-04-22T01:16:43.589508+0000","last_undegraded":"2024-04-22T01:16:43.589508+0000","last_fullsized":"2024-04-22T01:16:43.589508+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:12.850534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587985+0000","last_change":"2024-04-22T01:16:41.279372+0000","last_active":"2024-04-22T01:16:43.587985+0000","last_peered":"2024-04-22T01:16:43.587985+0000","last_clean":"2024-04-22T01:16:43.587985+0000","last_became_active":"2024-04-22T01:16:41.277927+0000","last_became_peered":"2024-04-22T01:16:41.277927+0000","last_unstale":"2024-04-22T01:16:43.587985+0000","last_undegraded":"2024-04-22T01:16:43.587985+0000","last_fullsized":"2024-04-22T01:16:43.587985+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:42.939970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273665+0000","last_change":"2024-04-22T01:16:42.293982+0000","last_active":"2024-04-22T01:16:43.273665+0000","last_peered":"2024-04-22T01:16:43.273665+0000","last_clean":"2024-04-22T01:16:43.273665+0000","last_became_active":"2024-04-22T01:16:42.293593+0000","last_became_peered":"2024-04-22T01:16:42.293593+0000","last_unstale":"2024-04-22T01:16:43.273665+0000","last_undegraded":"2024-04-22T01:16:43.273665+0000","last_fullsized":"2024-04-22T01:16:43.273665+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:21.076434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587501+0000","last_change":"2024-04-22T01:16:41.307711+0000","last_active":"2024-04-22T01:16:43.587501+0000","last_peered":"2024-04-22T01:16:43.587501+0000","last_clean":"2024-04-22T01:16:43.587501+0000","last_became_active":"2024-04-22T01:16:41.307267+0000","last_became_peered":"2024-04-22T01:16:41.307267+0000","last_unstale":"2024-04-22T01:16:43.587501+0000","last_undegraded":"2024-04-22T01:16:43.587501+0000","last_fullsized":"2024-04-22T01:16:43.587501+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:32:10.417022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274228+0000","last_change":"2024-04-22T01:16:42.294626+0000","last_active":"2024-04-22T01:16:43.274228+0000","last_peered":"2024-04-22T01:16:43.274228+0000","last_clean":"2024-04-22T01:16:43.274228+0000","last_became_active":"2024-04-22T01:16:42.293865+0000","last_became_peered":"2024-04-22T01:16:42.293865+0000","last_unstale":"2024-04-22T01:16:43.274228+0000","last_undegraded":"2024-04-22T01:16:43.274228+0000","last_fullsized":"2024-04-22T01:16:43.274228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:52:44.520751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276559+0000","last_change":"2024-04-22T01:16:41.303104+0000","last_active":"2024-04-22T01:16:43.276559+0000","last_peered":"2024-04-22T01:16:43.276559+0000","last_clean":"2024-04-22T01:16:43.276559+0000","last_became_active":"2024-04-22T01:16:41.302511+0000","last_became_peered":"2024-04-22T01:16:41.302511+0000","last_unstale":"2024-04-22T01:16:43.276559+0000","last_undegraded":"2024-04-22T01:16:43.276559+0000","last_fullsized":"2024-04-22T01:16:43.276559+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:00:51.721140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273143+0000","last_change":"2024-04-22T01:16:42.287083+0000","last_active":"2024-04-22T01:16:43.273143+0000","last_peered":"2024-04-22T01:16:43.273143+0000","last_clean":"2024-04-22T01:16:43.273143+0000","last_became_active":"2024-04-22T01:16:42.286782+0000","last_became_peered":"2024-04-22T01:16:42.286782+0000","last_unstale":"2024-04-22T01:16:43.273143+0000","last_undegraded":"2024-04-22T01:16:43.273143+0000","last_fullsized":"2024-04-22T01:16:43.273143+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:55.123394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588267+0000","last_change":"2024-04-22T01:16:41.305635+0000","last_active":"2024-04-22T01:16:43.588267+0000","last_peered":"2024-04-22T01:16:43.588267+0000","last_clean":"2024-04-22T01:16:43.588267+0000","last_became_active":"2024-04-22T01:16:41.305094+0000","last_became_peered":"2024-04-22T01:16:41.305094+0000","last_unstale":"2024-04-22T01:16:43.588267+0000","last_undegraded":"2024-04-22T01:16:43.588267+0000","last_fullsized":"2024-04-22T01:16:43.588267+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:44.052756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590288+0000","last_change":"2024-04-22T01:16:42.307611+0000","last_active":"2024-04-22T01:16:43.590288+0000","last_peered":"2024-04-22T01:16:43.590288+0000","last_clean":"2024-04-22T01:16:43.590288+0000","last_became_active":"2024-04-22T01:16:42.307270+0000","last_became_peered":"2024-04-22T01:16:42.307270+0000","last_unstale":"2024-04-22T01:16:43.590288+0000","last_undegraded":"2024-04-22T01:16:43.590288+0000","last_fullsized":"2024-04-22T01:16:43.590288+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:18.176769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589307+0000","last_change":"2024-04-22T01:16:41.290391+0000","last_active":"2024-04-22T01:16:43.589307+0000","last_peered":"2024-04-22T01:16:43.589307+0000","last_clean":"2024-04-22T01:16:43.589307+0000","last_became_active":"2024-04-22T01:16:41.289895+0000","last_became_peered":"2024-04-22T01:16:41.289895+0000","last_unstale":"2024-04-22T01:16:43.589307+0000","last_undegraded":"2024-04-22T01:16:43.589307+0000","last_fullsized":"2024-04-22T01:16:43.589307+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:32.376868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589836+0000","last_change":"2024-04-22T01:16:42.313744+0000","last_active":"2024-04-22T01:16:43.589836+0000","last_peered":"2024-04-22T01:16:43.589836+0000","last_clean":"2024-04-22T01:16:43.589836+0000","last_became_active":"2024-04-22T01:16:42.313438+0000","last_became_peered":"2024-04-22T01:16:42.313438+0000","last_unstale":"2024-04-22T01:16:43.589836+0000","last_undegraded":"2024-04-22T01:16:43.589836+0000","last_fullsized":"2024-04-22T01:16:43.589836+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:25.700498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590170+0000","last_change":"2024-04-22T01:16:41.294486+0000","last_active":"2024-04-22T01:16:43.590170+0000","last_peered":"2024-04-22T01:16:43.590170+0000","last_clean":"2024-04-22T01:16:43.590170+0000","last_became_active":"2024-04-22T01:16:41.294147+0000","last_became_peered":"2024-04-22T01:16:41.294147+0000","last_unstale":"2024-04-22T01:16:43.590170+0000","last_undegraded":"2024-04-22T01:16:43.590170+0000","last_fullsized":"2024-04-22T01:16:43.590170+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:54.119657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275660+0000","last_change":"2024-04-22T01:16:42.318672+0000","last_active":"2024-04-22T01:16:43.275660+0000","last_peered":"2024-04-22T01:16:43.275660+0000","last_clean":"2024-04-22T01:16:43.275660+0000","last_became_active":"2024-04-22T01:16:42.318021+0000","last_became_peered":"2024-04-22T01:16:42.318021+0000","last_unstale":"2024-04-22T01:16:43.275660+0000","last_undegraded":"2024-04-22T01:16:43.275660+0000","last_fullsized":"2024-04-22T01:16:43.275660+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:19.311224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589034+0000","last_change":"2024-04-22T01:16:41.278493+0000","last_active":"2024-04-22T01:16:43.589034+0000","last_peered":"2024-04-22T01:16:43.589034+0000","last_clean":"2024-04-22T01:16:43.589034+0000","last_became_active":"2024-04-22T01:16:41.278001+0000","last_became_peered":"2024-04-22T01:16:41.278001+0000","last_unstale":"2024-04-22T01:16:43.589034+0000","last_undegraded":"2024-04-22T01:16:43.589034+0000","last_fullsized":"2024-04-22T01:16:43.589034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:02:24.594245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589145+0000","last_change":"2024-04-22T01:16:42.299128+0000","last_active":"2024-04-22T01:16:43.589145+0000","last_peered":"2024-04-22T01:16:43.589145+0000","last_clean":"2024-04-22T01:16:43.589145+0000","last_became_active":"2024-04-22T01:16:42.298790+0000","last_became_peered":"2024-04-22T01:16:42.298790+0000","last_unstale":"2024-04-22T01:16:43.589145+0000","last_undegraded":"2024-04-22T01:16:43.589145+0000","last_fullsized":"2024-04-22T01:16:43.589145+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:09.047831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586040+0000","last_change":"2024-04-22T01:16:41.276295+0000","last_active":"2024-04-22T01:16:43.586040+0000","last_peered":"2024-04-22T01:16:43.586040+0000","last_clean":"2024-04-22T01:16:43.586040+0000","last_became_active":"2024-04-22T01:16:41.273094+0000","last_became_peered":"2024-04-22T01:16:41.273094+0000","last_unstale":"2024-04-22T01:16:43.586040+0000","last_undegraded":"2024-04-22T01:16:43.586040+0000","last_fullsized":"2024-04-22T01:16:43.586040+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:13.230682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591157+0000","last_change":"2024-04-22T01:16:42.293420+0000","last_active":"2024-04-22T01:16:43.591157+0000","last_peered":"2024-04-22T01:16:43.591157+0000","last_clean":"2024-04-22T01:16:43.591157+0000","last_became_active":"2024-04-22T01:16:42.293156+0000","last_became_peered":"2024-04-22T01:16:42.293156+0000","last_unstale":"2024-04-22T01:16:43.591157+0000","last_undegraded":"2024-04-22T01:16:43.591157+0000","last_fullsized":"2024-04-22T01:16:43.591157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:07.621817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274462+0000","last_change":"2024-04-22T01:16:41.289193+0000","last_active":"2024-04-22T01:16:43.274462+0000","last_peered":"2024-04-22T01:16:43.274462+0000","last_clean":"2024-04-22T01:16:43.274462+0000","last_became_active":"2024-04-22T01:16:41.286911+0000","last_became_peered":"2024-04-22T01:16:41.286911+0000","last_unstale":"2024-04-22T01:16:43.274462+0000","last_undegraded":"2024-04-22T01:16:43.274462+0000","last_fullsized":"2024-04-22T01:16:43.274462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:32.376868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273996+0000","last_change":"2024-04-22T01:16:42.300814+0000","last_active":"2024-04-22T01:16:43.273996+0000","last_peered":"2024-04-22T01:16:43.273996+0000","last_clean":"2024-04-22T01:16:43.273996+0000","last_became_active":"2024-04-22T01:16:42.299882+0000","last_became_peered":"2024-04-22T01:16:42.299882+0000","last_unstale":"2024-04-22T01:16:43.273996+0000","last_undegraded":"2024-04-22T01:16:43.273996+0000","last_fullsized":"2024-04-22T01:16:43.273996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:45.056493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589585+0000","last_change":"2024-04-22T01:16:41.295927+0000","last_active":"2024-04-22T01:16:43.589585+0000","last_peered":"2024-04-22T01:16:43.589585+0000","last_clean":"2024-04-22T01:16:43.589585+0000","last_became_active":"2024-04-22T01:16:41.295504+0000","last_became_peered":"2024-04-22T01:16:41.295504+0000","last_unstale":"2024-04-22T01:16:43.589585+0000","last_undegraded":"2024-04-22T01:16:43.589585+0000","last_fullsized":"2024-04-22T01:16:43.589585+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:39.001001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586608+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:16:43.586608+0000","last_peered":"2024-04-22T01:16:43.586608+0000","last_clean":"2024-04-22T01:16:43.586608+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:16:43.586608+0000","last_undegraded":"2024-04-22T01:16:43.586608+0000","last_fullsized":"2024-04-22T01:16:43.586608+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":859819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":269979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":269327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":269909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35956,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335884,"statfs":{"total":96636764160,"available":96599945216,"internally_reserved":0,"allocated":622592,"data_stored":270961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":270939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":276179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":859751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:16:46.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:16:46.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 Processor -- start 2024-04-22T01:16:46.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 -- start start 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9a5015dfb0 0x7f9a500b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9a500d3120 0x7f9a500ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a5015e390 con 0x7f9a500cea80 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9a500cf420 con 0x7f9a500d3120 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a5707c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9a50161e00 con 0x7f9a5015dfb0 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a55619640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9a500d3120 0x7f9a500ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a4ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9a500cea80 0x7f9a500b5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56978/0 (socket says 172.21.15.70:56978) 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a55619640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9a500d3120 0x7f9a500ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60582/0 (socket says 172.21.15.70:60582) 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a4ffff640 1 -- 172.21.15.70:0/3246725025 learned_addr learned my addr 172.21.15.70:0/3246725025 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a54e18640 1 --2- 172.21.15.70:0/3246725025 >> v2:172.21.15.98:3300/0 conn(0x7f9a5015dfb0 0x7f9a500b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a55619640 1 -- 172.21.15.70:0/3246725025 >> v2:172.21.15.98:3300/0 conn(0x7f9a5015dfb0 msgr2=0x7f9a500b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a55619640 1 --2- 172.21.15.70:0/3246725025 >> v2:172.21.15.98:3300/0 conn(0x7f9a5015dfb0 0x7f9a500b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a55619640 1 -- 172.21.15.70:0/3246725025 >> v1:172.21.15.70:6789/0 conn(0x7f9a500cea80 legacy=0x7f9a500b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.474+0000 7f9a55619640 1 -- 172.21.15.70:0/3246725025 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a501620e0 con 0x7f9a500d3120 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a55619640 1 --2- 172.21.15.70:0/3246725025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a500d3120 0x7f9a500ceee0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9a500d4d80 tx=0x7f9a4801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735bbd4eca65e469 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a4f7fe640 1 -- 172.21.15.70:0/3246725025 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480b2070 con 0x7f9a500d3120 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a4f7fe640 1 -- 172.21.15.70:0/3246725025 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a480a5820 con 0x7f9a500d3120 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a4f7fe640 1 -- 172.21.15.70:0/3246725025 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480a5bf0 con 0x7f9a500d3120 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- 172.21.15.70:0/3246725025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a500d3120 msgr2=0x7f9a500ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3246725025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a500d3120 0x7f9a500ceee0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9a500d4d80 tx=0x7f9a4801c930 comp rx=0 tx=0).stop 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- 172.21.15.70:0/3246725025 shutdown_connections 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3246725025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a500d3120 0x7f9a500ceee0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3246725025 >> v2:172.21.15.98:3300/0 conn(0x7f9a5015dfb0 0x7f9a500b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- 172.21.15.70:0/3246725025 >> 172.21.15.70:0/3246725025 conn(0x7f9a5006c9b0 msgr2=0x7f9a5006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:46.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- 172.21.15.70:0/3246725025 shutdown_connections 2024-04-22T01:16:46.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- 172.21.15.70:0/3246725025 wait complete. 2024-04-22T01:16:46.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 Processor -- start 2024-04-22T01:16:46.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- start start 2024-04-22T01:16:46.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a500cea80 0x7f9a50151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a500d3120 0x7f9a500b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 0x7f9a500b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5016add0 con 0x7f9a5015dfb0 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a50161a90 con 0x7f9a500d3120 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a5707c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5016af50 con 0x7f9a500cea80 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a55619640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 0x7f9a500b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a55619640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 0x7f9a500b71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60594/0 (socket says 172.21.15.70:60594) 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.478+0000 7f9a55619640 1 -- 172.21.15.70:0/3374515052 learned_addr learned my addr 172.21.15.70:0/3374515052 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4ffff640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a500d3120 0x7f9a500b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a54e18640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a500cea80 0x7f9a50151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a55619640 1 -- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a500cea80 msgr2=0x7f9a50151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a55619640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a500cea80 0x7f9a50151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a55619640 1 -- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a500d3120 msgr2=0x7f9a500b6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a55619640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a500d3120 0x7f9a500b6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a55619640 1 -- 172.21.15.70:0/3374515052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5015a860 con 0x7f9a5015dfb0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a54e18640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a500cea80 0x7f9a50151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a55619640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 0x7f9a500b71e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9a480b0040 tx=0x7f9a480196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480b2050 con 0x7f9a5015dfb0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a480be070 con 0x7f9a5015dfb0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a501597c0 con 0x7f9a5015dfb0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480ada70 con 0x7f9a5015dfb0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a50159d80 con 0x7f9a5015dfb0 2024-04-22T01:16:46.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a50068210 con 0x7f9a5015dfb0 2024-04-22T01:16:46.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9a480adc10 con 0x7f9a5015dfb0 2024-04-22T01:16:46.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4d7fa640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a3c0678c0 0x7f9a3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a54e18640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a3c0678c0 0x7f9a3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f9a48120f10 con 0x7f9a5015dfb0 2024-04-22T01:16:46.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.482+0000 7f9a54e18640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a3c0678c0 0x7f9a3c069d80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9a40006fd0 tx=0x7f9a40008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.486+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9a48016520 con 0x7f9a5015dfb0 2024-04-22T01:16:46.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.638+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9a500b9890 con 0x7f9a3c0678c0 2024-04-22T01:16:46.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.650+0000 7f9a4d7fa640 1 -- 172.21.15.70:0/3374515052 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251953 (secure 0 0 0) 0x7f9a500b9890 con 0x7f9a3c0678c0 2024-04-22T01:16:46.654 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:46.655 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a3c0678c0 msgr2=0x7f9a3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a3c0678c0 0x7f9a3c069d80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9a40006fd0 tx=0x7f9a40008040 comp rx=0 tx=0).stop 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 msgr2=0x7f9a500b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 0x7f9a500b71e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9a480b0040 tx=0x7f9a480196d0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 shutdown_connections 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a3c0678c0 0x7f9a3c069d80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5015dfb0 0x7f9a500b71e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a500d3120 0x7f9a500b6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 --2- 172.21.15.70:0/3374515052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a500cea80 0x7f9a50151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 >> 172.21.15.70:0/3374515052 conn(0x7f9a5006c9b0 msgr2=0x7f9a500ca040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 shutdown_connections 2024-04-22T01:16:46.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.654+0000 7f9a5707c640 1 -- 172.21.15.70:0/3374515052 wait complete. 2024-04-22T01:16:46.670 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-22T01:16:46.623989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251272,"kb_used_data":5972,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754723448,"statfs":{"total":773094113280,"available":772836810752,"internally_reserved":0,"allocated":6115328,"data_stored":3346864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001619"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273487+0000","last_change":"2024-04-22T01:16:42.297132+0000","last_active":"2024-04-22T01:16:43.273487+0000","last_peered":"2024-04-22T01:16:43.273487+0000","last_clean":"2024-04-22T01:16:43.273487+0000","last_became_active":"2024-04-22T01:16:42.296705+0000","last_became_peered":"2024-04-22T01:16:42.296705+0000","last_unstale":"2024-04-22T01:16:43.273487+0000","last_undegraded":"2024-04-22T01:16:43.273487+0000","last_fullsized":"2024-04-22T01:16:43.273487+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:40.262829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587931+0000","last_change":"2024-04-22T01:16:42.328580+0000","last_active":"2024-04-22T01:16:43.587931+0000","last_peered":"2024-04-22T01:16:43.587931+0000","last_clean":"2024-04-22T01:16:43.587931+0000","last_became_active":"2024-04-22T01:16:42.322602+0000","last_became_peered":"2024-04-22T01:16:42.322602+0000","last_unstale":"2024-04-22T01:16:43.587931+0000","last_undegraded":"2024-04-22T01:16:43.587931+0000","last_fullsized":"2024-04-22T01:16:43.587931+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:58:39.203936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588404+0000","last_change":"2024-04-22T01:16:42.283515+0000","last_active":"2024-04-22T01:16:43.588404+0000","last_peered":"2024-04-22T01:16:43.588404+0000","last_clean":"2024-04-22T01:16:43.588404+0000","last_became_active":"2024-04-22T01:16:42.283099+0000","last_became_peered":"2024-04-22T01:16:42.283099+0000","last_unstale":"2024-04-22T01:16:43.588404+0000","last_undegraded":"2024-04-22T01:16:43.588404+0000","last_fullsized":"2024-04-22T01:16:43.588404+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:31.183960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587803+0000","last_change":"2024-04-22T01:16:42.322015+0000","last_active":"2024-04-22T01:16:43.587803+0000","last_peered":"2024-04-22T01:16:43.587803+0000","last_clean":"2024-04-22T01:16:43.587803+0000","last_became_active":"2024-04-22T01:16:42.320226+0000","last_became_peered":"2024-04-22T01:16:42.320226+0000","last_unstale":"2024-04-22T01:16:43.587803+0000","last_undegraded":"2024-04-22T01:16:43.587803+0000","last_fullsized":"2024-04-22T01:16:43.587803+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:14.845222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586483+0000","last_change":"2024-04-22T01:16:42.298812+0000","last_active":"2024-04-22T01:16:43.586483+0000","last_peered":"2024-04-22T01:16:43.586483+0000","last_clean":"2024-04-22T01:16:43.586483+0000","last_became_active":"2024-04-22T01:16:42.298003+0000","last_became_peered":"2024-04-22T01:16:42.298003+0000","last_unstale":"2024-04-22T01:16:43.586483+0000","last_undegraded":"2024-04-22T01:16:43.586483+0000","last_fullsized":"2024-04-22T01:16:43.586483+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:06.384082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272771+0000","last_change":"2024-04-22T01:16:42.291471+0000","last_active":"2024-04-22T01:16:43.272771+0000","last_peered":"2024-04-22T01:16:43.272771+0000","last_clean":"2024-04-22T01:16:43.272771+0000","last_became_active":"2024-04-22T01:16:42.290951+0000","last_became_peered":"2024-04-22T01:16:42.290951+0000","last_unstale":"2024-04-22T01:16:43.272771+0000","last_undegraded":"2024-04-22T01:16:43.272771+0000","last_fullsized":"2024-04-22T01:16:43.272771+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:02:25.597982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586706+0000","last_change":"2024-04-22T01:16:42.291719+0000","last_active":"2024-04-22T01:16:43.586706+0000","last_peered":"2024-04-22T01:16:43.586706+0000","last_clean":"2024-04-22T01:16:43.586706+0000","last_became_active":"2024-04-22T01:16:42.285461+0000","last_became_peered":"2024-04-22T01:16:42.285461+0000","last_unstale":"2024-04-22T01:16:43.586706+0000","last_undegraded":"2024-04-22T01:16:43.586706+0000","last_fullsized":"2024-04-22T01:16:43.586706+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.020114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588996+0000","last_change":"2024-04-22T01:16:42.316255+0000","last_active":"2024-04-22T01:16:43.588996+0000","last_peered":"2024-04-22T01:16:43.588996+0000","last_clean":"2024-04-22T01:16:43.588996+0000","last_became_active":"2024-04-22T01:16:42.315687+0000","last_became_peered":"2024-04-22T01:16:42.315687+0000","last_unstale":"2024-04-22T01:16:43.588996+0000","last_undegraded":"2024-04-22T01:16:43.588996+0000","last_fullsized":"2024-04-22T01:16:43.588996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:26.800461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591096+0000","last_change":"2024-04-22T01:16:42.316227+0000","last_active":"2024-04-22T01:16:43.591096+0000","last_peered":"2024-04-22T01:16:43.591096+0000","last_clean":"2024-04-22T01:16:43.591096+0000","last_became_active":"2024-04-22T01:16:42.315857+0000","last_became_peered":"2024-04-22T01:16:42.315857+0000","last_unstale":"2024-04-22T01:16:43.591096+0000","last_undegraded":"2024-04-22T01:16:43.591096+0000","last_fullsized":"2024-04-22T01:16:43.591096+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:25:30.883785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273149+0000","last_change":"2024-04-22T01:16:42.307041+0000","last_active":"2024-04-22T01:16:43.273149+0000","last_peered":"2024-04-22T01:16:43.273149+0000","last_clean":"2024-04-22T01:16:43.273149+0000","last_became_active":"2024-04-22T01:16:42.306524+0000","last_became_peered":"2024-04-22T01:16:42.306524+0000","last_unstale":"2024-04-22T01:16:43.273149+0000","last_undegraded":"2024-04-22T01:16:43.273149+0000","last_fullsized":"2024-04-22T01:16:43.273149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:10.171708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588199+0000","last_change":"2024-04-22T01:16:42.313655+0000","last_active":"2024-04-22T01:16:43.588199+0000","last_peered":"2024-04-22T01:16:43.588199+0000","last_clean":"2024-04-22T01:16:43.588199+0000","last_became_active":"2024-04-22T01:16:42.313160+0000","last_became_peered":"2024-04-22T01:16:42.313160+0000","last_unstale":"2024-04-22T01:16:43.588199+0000","last_undegraded":"2024-04-22T01:16:43.588199+0000","last_fullsized":"2024-04-22T01:16:43.588199+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:22.348897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590803+0000","last_change":"2024-04-22T01:16:42.308708+0000","last_active":"2024-04-22T01:16:43.590803+0000","last_peered":"2024-04-22T01:16:43.590803+0000","last_clean":"2024-04-22T01:16:43.590803+0000","last_became_active":"2024-04-22T01:16:42.308430+0000","last_became_peered":"2024-04-22T01:16:42.308430+0000","last_unstale":"2024-04-22T01:16:43.590803+0000","last_undegraded":"2024-04-22T01:16:43.590803+0000","last_fullsized":"2024-04-22T01:16:43.590803+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:17.745700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272472+0000","last_change":"2024-04-22T01:16:42.307164+0000","last_active":"2024-04-22T01:16:43.272472+0000","last_peered":"2024-04-22T01:16:43.272472+0000","last_clean":"2024-04-22T01:16:43.272472+0000","last_became_active":"2024-04-22T01:16:42.306668+0000","last_became_peered":"2024-04-22T01:16:42.306668+0000","last_unstale":"2024-04-22T01:16:43.272472+0000","last_undegraded":"2024-04-22T01:16:43.272472+0000","last_fullsized":"2024-04-22T01:16:43.272472+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:33.380606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587053+0000","last_change":"2024-04-22T01:16:42.314168+0000","last_active":"2024-04-22T01:16:43.587053+0000","last_peered":"2024-04-22T01:16:43.587053+0000","last_clean":"2024-04-22T01:16:43.587053+0000","last_became_active":"2024-04-22T01:16:42.313798+0000","last_became_peered":"2024-04-22T01:16:42.313798+0000","last_unstale":"2024-04-22T01:16:43.587053+0000","last_undegraded":"2024-04-22T01:16:43.587053+0000","last_fullsized":"2024-04-22T01:16:43.587053+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:26.800461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586764+0000","last_change":"2024-04-22T01:16:42.322366+0000","last_active":"2024-04-22T01:16:43.586764+0000","last_peered":"2024-04-22T01:16:43.586764+0000","last_clean":"2024-04-22T01:16:43.586764+0000","last_became_active":"2024-04-22T01:16:42.321520+0000","last_became_peered":"2024-04-22T01:16:42.321520+0000","last_unstale":"2024-04-22T01:16:43.586764+0000","last_undegraded":"2024-04-22T01:16:43.586764+0000","last_fullsized":"2024-04-22T01:16:43.586764+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:21.076434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272592+0000","last_change":"2024-04-22T01:16:42.308132+0000","last_active":"2024-04-22T01:16:43.272592+0000","last_peered":"2024-04-22T01:16:43.272592+0000","last_clean":"2024-04-22T01:16:43.272592+0000","last_became_active":"2024-04-22T01:16:42.307603+0000","last_became_peered":"2024-04-22T01:16:42.307603+0000","last_unstale":"2024-04-22T01:16:43.272592+0000","last_undegraded":"2024-04-22T01:16:43.272592+0000","last_fullsized":"2024-04-22T01:16:43.272592+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:15.786674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589077+0000","last_change":"2024-04-22T01:16:42.314883+0000","last_active":"2024-04-22T01:16:43.589077+0000","last_peered":"2024-04-22T01:16:43.589077+0000","last_clean":"2024-04-22T01:16:43.589077+0000","last_became_active":"2024-04-22T01:16:42.314344+0000","last_became_peered":"2024-04-22T01:16:42.314344+0000","last_unstale":"2024-04-22T01:16:43.589077+0000","last_undegraded":"2024-04-22T01:16:43.589077+0000","last_fullsized":"2024-04-22T01:16:43.589077+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:07.621817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588076+0000","last_change":"2024-04-22T01:16:42.328618+0000","last_active":"2024-04-22T01:16:43.588076+0000","last_peered":"2024-04-22T01:16:43.588076+0000","last_clean":"2024-04-22T01:16:43.588076+0000","last_became_active":"2024-04-22T01:16:42.323472+0000","last_became_peered":"2024-04-22T01:16:42.323472+0000","last_unstale":"2024-04-22T01:16:43.588076+0000","last_undegraded":"2024-04-22T01:16:43.588076+0000","last_fullsized":"2024-04-22T01:16:43.588076+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:19.311224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588535+0000","last_change":"2024-04-22T01:16:42.313966+0000","last_active":"2024-04-22T01:16:43.588535+0000","last_peered":"2024-04-22T01:16:43.588535+0000","last_clean":"2024-04-22T01:16:43.588535+0000","last_became_active":"2024-04-22T01:16:42.299795+0000","last_became_peered":"2024-04-22T01:16:42.299795+0000","last_unstale":"2024-04-22T01:16:43.588535+0000","last_undegraded":"2024-04-22T01:16:43.588535+0000","last_fullsized":"2024-04-22T01:16:43.588535+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:17.745700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276254+0000","last_change":"2024-04-22T01:16:42.309101+0000","last_active":"2024-04-22T01:16:43.276254+0000","last_peered":"2024-04-22T01:16:43.276254+0000","last_clean":"2024-04-22T01:16:43.276254+0000","last_became_active":"2024-04-22T01:16:42.307750+0000","last_became_peered":"2024-04-22T01:16:42.307750+0000","last_unstale":"2024-04-22T01:16:43.276254+0000","last_undegraded":"2024-04-22T01:16:43.276254+0000","last_fullsized":"2024-04-22T01:16:43.276254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:03.530479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276084+0000","last_change":"2024-04-22T01:16:42.306145+0000","last_active":"2024-04-22T01:16:43.276084+0000","last_peered":"2024-04-22T01:16:43.276084+0000","last_clean":"2024-04-22T01:16:43.276084+0000","last_became_active":"2024-04-22T01:16:42.304669+0000","last_became_peered":"2024-04-22T01:16:42.304669+0000","last_unstale":"2024-04-22T01:16:43.276084+0000","last_undegraded":"2024-04-22T01:16:43.276084+0000","last_fullsized":"2024-04-22T01:16:43.276084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:15.786674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588531+0000","last_change":"2024-04-22T01:16:42.299940+0000","last_active":"2024-04-22T01:16:43.588531+0000","last_peered":"2024-04-22T01:16:43.588531+0000","last_clean":"2024-04-22T01:16:43.588531+0000","last_became_active":"2024-04-22T01:16:42.298977+0000","last_became_peered":"2024-04-22T01:16:42.298977+0000","last_unstale":"2024-04-22T01:16:43.588531+0000","last_undegraded":"2024-04-22T01:16:43.588531+0000","last_fullsized":"2024-04-22T01:16:43.588531+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.020114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587443+0000","last_change":"2024-04-22T01:16:42.314427+0000","last_active":"2024-04-22T01:16:43.587443+0000","last_peered":"2024-04-22T01:16:43.587443+0000","last_clean":"2024-04-22T01:16:43.587443+0000","last_became_active":"2024-04-22T01:16:42.313443+0000","last_became_peered":"2024-04-22T01:16:42.313443+0000","last_unstale":"2024-04-22T01:16:43.587443+0000","last_undegraded":"2024-04-22T01:16:43.587443+0000","last_fullsized":"2024-04-22T01:16:43.587443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:45.511733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586628+0000","last_change":"2024-04-22T01:16:42.284569+0000","last_active":"2024-04-22T01:16:43.586628+0000","last_peered":"2024-04-22T01:16:43.586628+0000","last_clean":"2024-04-22T01:16:43.586628+0000","last_became_active":"2024-04-22T01:16:42.284055+0000","last_became_peered":"2024-04-22T01:16:42.284055+0000","last_unstale":"2024-04-22T01:16:43.586628+0000","last_undegraded":"2024-04-22T01:16:43.586628+0000","last_fullsized":"2024-04-22T01:16:43.586628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:34.517785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589497+0000","last_change":"2024-04-22T01:16:42.313172+0000","last_active":"2024-04-22T01:16:43.589497+0000","last_peered":"2024-04-22T01:16:43.589497+0000","last_clean":"2024-04-22T01:16:43.589497+0000","last_became_active":"2024-04-22T01:16:42.299260+0000","last_became_peered":"2024-04-22T01:16:42.299260+0000","last_unstale":"2024-04-22T01:16:43.589497+0000","last_undegraded":"2024-04-22T01:16:43.589497+0000","last_fullsized":"2024-04-22T01:16:43.589497+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:32.752089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590743+0000","last_change":"2024-04-22T01:16:42.295341+0000","last_active":"2024-04-22T01:16:43.590743+0000","last_peered":"2024-04-22T01:16:43.590743+0000","last_clean":"2024-04-22T01:16:43.590743+0000","last_became_active":"2024-04-22T01:16:42.294922+0000","last_became_peered":"2024-04-22T01:16:42.294922+0000","last_unstale":"2024-04-22T01:16:43.590743+0000","last_undegraded":"2024-04-22T01:16:43.590743+0000","last_fullsized":"2024-04-22T01:16:43.590743+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:31.183960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273528+0000","last_change":"2024-04-22T01:16:42.319657+0000","last_active":"2024-04-22T01:16:43.273528+0000","last_peered":"2024-04-22T01:16:43.273528+0000","last_clean":"2024-04-22T01:16:43.273528+0000","last_became_active":"2024-04-22T01:16:42.319353+0000","last_became_peered":"2024-04-22T01:16:42.319353+0000","last_unstale":"2024-04-22T01:16:43.273528+0000","last_undegraded":"2024-04-22T01:16:43.273528+0000","last_fullsized":"2024-04-22T01:16:43.273528+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:33.750136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276370+0000","last_change":"2024-04-22T01:16:42.287849+0000","last_active":"2024-04-22T01:16:43.276370+0000","last_peered":"2024-04-22T01:16:43.276370+0000","last_clean":"2024-04-22T01:16:43.276370+0000","last_became_active":"2024-04-22T01:16:42.287102+0000","last_became_peered":"2024-04-22T01:16:42.287102+0000","last_unstale":"2024-04-22T01:16:43.276370+0000","last_undegraded":"2024-04-22T01:16:43.276370+0000","last_fullsized":"2024-04-22T01:16:43.276370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:40.004739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587731+0000","last_change":"2024-04-22T01:16:42.318878+0000","last_active":"2024-04-22T01:16:43.587731+0000","last_peered":"2024-04-22T01:16:43.587731+0000","last_clean":"2024-04-22T01:16:43.587731+0000","last_became_active":"2024-04-22T01:16:42.318484+0000","last_became_peered":"2024-04-22T01:16:42.318484+0000","last_unstale":"2024-04-22T01:16:43.587731+0000","last_undegraded":"2024-04-22T01:16:43.587731+0000","last_fullsized":"2024-04-22T01:16:43.587731+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:18.176769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586921+0000","last_change":"2024-04-22T01:16:42.321572+0000","last_active":"2024-04-22T01:16:43.586921+0000","last_peered":"2024-04-22T01:16:43.586921+0000","last_clean":"2024-04-22T01:16:43.586921+0000","last_became_active":"2024-04-22T01:16:42.321097+0000","last_became_peered":"2024-04-22T01:16:42.321097+0000","last_unstale":"2024-04-22T01:16:43.586921+0000","last_undegraded":"2024-04-22T01:16:43.586921+0000","last_fullsized":"2024-04-22T01:16:43.586921+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:52:44.520751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587074+0000","last_change":"2024-04-22T01:16:42.298826+0000","last_active":"2024-04-22T01:16:43.587074+0000","last_peered":"2024-04-22T01:16:43.587074+0000","last_clean":"2024-04-22T01:16:43.587074+0000","last_became_active":"2024-04-22T01:16:42.298488+0000","last_became_peered":"2024-04-22T01:16:42.298488+0000","last_unstale":"2024-04-22T01:16:43.587074+0000","last_undegraded":"2024-04-22T01:16:43.587074+0000","last_fullsized":"2024-04-22T01:16:43.587074+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:19.712873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273837+0000","last_change":"2024-04-22T01:16:42.297976+0000","last_active":"2024-04-22T01:16:43.273837+0000","last_peered":"2024-04-22T01:16:43.273837+0000","last_clean":"2024-04-22T01:16:43.273837+0000","last_became_active":"2024-04-22T01:16:42.294104+0000","last_became_peered":"2024-04-22T01:16:42.294104+0000","last_unstale":"2024-04-22T01:16:43.273837+0000","last_undegraded":"2024-04-22T01:16:43.273837+0000","last_fullsized":"2024-04-22T01:16:43.273837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:33.750136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586547+0000","last_change":"2024-04-22T01:16:42.329218+0000","last_active":"2024-04-22T01:16:43.586547+0000","last_peered":"2024-04-22T01:16:43.586547+0000","last_clean":"2024-04-22T01:16:43.586547+0000","last_became_active":"2024-04-22T01:16:42.323604+0000","last_became_peered":"2024-04-22T01:16:42.323604+0000","last_unstale":"2024-04-22T01:16:43.586547+0000","last_undegraded":"2024-04-22T01:16:43.586547+0000","last_fullsized":"2024-04-22T01:16:43.586547+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:33.750136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276266+0000","last_change":"2024-04-22T01:16:41.299226+0000","last_active":"2024-04-22T01:16:43.276266+0000","last_peered":"2024-04-22T01:16:43.276266+0000","last_clean":"2024-04-22T01:16:43.276266+0000","last_became_active":"2024-04-22T01:16:41.298647+0000","last_became_peered":"2024-04-22T01:16:41.298647+0000","last_unstale":"2024-04-22T01:16:43.276266+0000","last_undegraded":"2024-04-22T01:16:43.276266+0000","last_fullsized":"2024-04-22T01:16:43.276266+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:45:25.792994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275952+0000","last_change":"2024-04-22T01:16:42.318792+0000","last_active":"2024-04-22T01:16:43.275952+0000","last_peered":"2024-04-22T01:16:43.275952+0000","last_clean":"2024-04-22T01:16:43.275952+0000","last_became_active":"2024-04-22T01:16:42.317988+0000","last_became_peered":"2024-04-22T01:16:42.317988+0000","last_unstale":"2024-04-22T01:16:43.275952+0000","last_undegraded":"2024-04-22T01:16:43.275952+0000","last_fullsized":"2024-04-22T01:16:43.275952+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:14.845222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273282+0000","last_change":"2024-04-22T01:16:41.286305+0000","last_active":"2024-04-22T01:16:43.273282+0000","last_peered":"2024-04-22T01:16:43.273282+0000","last_clean":"2024-04-22T01:16:43.273282+0000","last_became_active":"2024-04-22T01:16:41.285839+0000","last_became_peered":"2024-04-22T01:16:41.285839+0000","last_unstale":"2024-04-22T01:16:43.273282+0000","last_undegraded":"2024-04-22T01:16:43.273282+0000","last_fullsized":"2024-04-22T01:16:43.273282+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:13.230682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275553+0000","last_change":"2024-04-22T01:16:42.308126+0000","last_active":"2024-04-22T01:16:43.275553+0000","last_peered":"2024-04-22T01:16:43.275553+0000","last_clean":"2024-04-22T01:16:43.275553+0000","last_became_active":"2024-04-22T01:16:42.307387+0000","last_became_peered":"2024-04-22T01:16:42.307387+0000","last_unstale":"2024-04-22T01:16:43.275553+0000","last_undegraded":"2024-04-22T01:16:43.275553+0000","last_fullsized":"2024-04-22T01:16:43.275553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:33.380606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273451+0000","last_change":"2024-04-22T01:16:41.299592+0000","last_active":"2024-04-22T01:16:43.273451+0000","last_peered":"2024-04-22T01:16:43.273451+0000","last_clean":"2024-04-22T01:16:43.273451+0000","last_became_active":"2024-04-22T01:16:41.299241+0000","last_became_peered":"2024-04-22T01:16:41.299241+0000","last_unstale":"2024-04-22T01:16:43.273451+0000","last_undegraded":"2024-04-22T01:16:43.273451+0000","last_fullsized":"2024-04-22T01:16:43.273451+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:14:25.445161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589758+0000","last_change":"2024-04-22T01:16:42.293507+0000","last_active":"2024-04-22T01:16:43.589758+0000","last_peered":"2024-04-22T01:16:43.589758+0000","last_clean":"2024-04-22T01:16:43.589758+0000","last_became_active":"2024-04-22T01:16:42.292862+0000","last_became_peered":"2024-04-22T01:16:42.292862+0000","last_unstale":"2024-04-22T01:16:43.589758+0000","last_undegraded":"2024-04-22T01:16:43.589758+0000","last_fullsized":"2024-04-22T01:16:43.589758+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:22.563055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275271+0000","last_change":"2024-04-22T01:16:41.294821+0000","last_active":"2024-04-22T01:16:43.275271+0000","last_peered":"2024-04-22T01:16:43.275271+0000","last_clean":"2024-04-22T01:16:43.275271+0000","last_became_active":"2024-04-22T01:16:41.294128+0000","last_became_peered":"2024-04-22T01:16:41.294128+0000","last_unstale":"2024-04-22T01:16:43.275271+0000","last_undegraded":"2024-04-22T01:16:43.275271+0000","last_fullsized":"2024-04-22T01:16:43.275271+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:44.052756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587481+0000","last_change":"2024-04-22T01:16:42.280148+0000","last_active":"2024-04-22T01:16:43.587481+0000","last_peered":"2024-04-22T01:16:43.587481+0000","last_clean":"2024-04-22T01:16:43.587481+0000","last_became_active":"2024-04-22T01:16:42.279293+0000","last_became_peered":"2024-04-22T01:16:42.279293+0000","last_unstale":"2024-04-22T01:16:43.587481+0000","last_undegraded":"2024-04-22T01:16:43.587481+0000","last_fullsized":"2024-04-22T01:16:43.587481+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:33.380606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586729+0000","last_change":"2024-04-22T01:16:41.289933+0000","last_active":"2024-04-22T01:16:43.586729+0000","last_peered":"2024-04-22T01:16:43.586729+0000","last_clean":"2024-04-22T01:16:43.586729+0000","last_became_active":"2024-04-22T01:16:41.289583+0000","last_became_peered":"2024-04-22T01:16:41.289583+0000","last_unstale":"2024-04-22T01:16:43.586729+0000","last_undegraded":"2024-04-22T01:16:43.586729+0000","last_fullsized":"2024-04-22T01:16:43.586729+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:05.380345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588135+0000","last_change":"2024-04-22T01:16:42.329811+0000","last_active":"2024-04-22T01:16:43.588135+0000","last_peered":"2024-04-22T01:16:43.588135+0000","last_clean":"2024-04-22T01:16:43.588135+0000","last_became_active":"2024-04-22T01:16:42.329087+0000","last_became_peered":"2024-04-22T01:16:42.329087+0000","last_unstale":"2024-04-22T01:16:43.588135+0000","last_undegraded":"2024-04-22T01:16:43.588135+0000","last_fullsized":"2024-04-22T01:16:43.588135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:04.039485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586411+0000","last_change":"2024-04-22T01:16:41.299672+0000","last_active":"2024-04-22T01:16:43.586411+0000","last_peered":"2024-04-22T01:16:43.586411+0000","last_clean":"2024-04-22T01:16:43.586411+0000","last_became_active":"2024-04-22T01:16:41.298905+0000","last_became_peered":"2024-04-22T01:16:41.298905+0000","last_unstale":"2024-04-22T01:16:43.586411+0000","last_undegraded":"2024-04-22T01:16:43.586411+0000","last_fullsized":"2024-04-22T01:16:43.586411+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:14:25.445161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591215+0000","last_change":"2024-04-22T01:16:42.295190+0000","last_active":"2024-04-22T01:16:43.591215+0000","last_peered":"2024-04-22T01:16:43.591215+0000","last_clean":"2024-04-22T01:16:43.591215+0000","last_became_active":"2024-04-22T01:16:42.294626+0000","last_became_peered":"2024-04-22T01:16:42.294626+0000","last_unstale":"2024-04-22T01:16:43.591215+0000","last_undegraded":"2024-04-22T01:16:43.591215+0000","last_fullsized":"2024-04-22T01:16:43.591215+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:33.573703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275345+0000","last_change":"2024-04-22T01:16:41.302031+0000","last_active":"2024-04-22T01:16:43.275345+0000","last_peered":"2024-04-22T01:16:43.275345+0000","last_clean":"2024-04-22T01:16:43.275345+0000","last_became_active":"2024-04-22T01:16:41.301420+0000","last_became_peered":"2024-04-22T01:16:41.301420+0000","last_unstale":"2024-04-22T01:16:43.275345+0000","last_undegraded":"2024-04-22T01:16:43.275345+0000","last_fullsized":"2024-04-22T01:16:43.275345+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:12.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587845+0000","last_change":"2024-04-22T01:16:42.328477+0000","last_active":"2024-04-22T01:16:43.587845+0000","last_peered":"2024-04-22T01:16:43.587845+0000","last_clean":"2024-04-22T01:16:43.587845+0000","last_became_active":"2024-04-22T01:16:42.323323+0000","last_became_peered":"2024-04-22T01:16:42.323323+0000","last_unstale":"2024-04-22T01:16:43.587845+0000","last_undegraded":"2024-04-22T01:16:43.587845+0000","last_fullsized":"2024-04-22T01:16:43.587845+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:21:23.616725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586864+0000","last_change":"2024-04-22T01:16:41.301533+0000","last_active":"2024-04-22T01:16:43.586864+0000","last_peered":"2024-04-22T01:16:43.586864+0000","last_clean":"2024-04-22T01:16:43.586864+0000","last_became_active":"2024-04-22T01:16:41.301042+0000","last_became_peered":"2024-04-22T01:16:41.301042+0000","last_unstale":"2024-04-22T01:16:43.586864+0000","last_undegraded":"2024-04-22T01:16:43.586864+0000","last_fullsized":"2024-04-22T01:16:43.586864+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:55:52.957089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586097+0000","last_change":"2024-04-22T01:16:42.295270+0000","last_active":"2024-04-22T01:16:43.586097+0000","last_peered":"2024-04-22T01:16:43.586097+0000","last_clean":"2024-04-22T01:16:43.586097+0000","last_became_active":"2024-04-22T01:16:42.294393+0000","last_became_peered":"2024-04-22T01:16:42.294393+0000","last_unstale":"2024-04-22T01:16:43.586097+0000","last_undegraded":"2024-04-22T01:16:43.586097+0000","last_fullsized":"2024-04-22T01:16:43.586097+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:32.555943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588245+0000","last_change":"2024-04-22T01:16:41.305789+0000","last_active":"2024-04-22T01:16:43.588245+0000","last_peered":"2024-04-22T01:16:43.588245+0000","last_clean":"2024-04-22T01:16:43.588245+0000","last_became_active":"2024-04-22T01:16:41.305231+0000","last_became_peered":"2024-04-22T01:16:41.305231+0000","last_unstale":"2024-04-22T01:16:43.588245+0000","last_undegraded":"2024-04-22T01:16:43.588245+0000","last_fullsized":"2024-04-22T01:16:43.588245+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:31.552205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274210+0000","last_change":"2024-04-22T01:16:42.292497+0000","last_active":"2024-04-22T01:16:43.274210+0000","last_peered":"2024-04-22T01:16:43.274210+0000","last_clean":"2024-04-22T01:16:43.274210+0000","last_became_active":"2024-04-22T01:16:42.291743+0000","last_became_peered":"2024-04-22T01:16:42.291743+0000","last_unstale":"2024-04-22T01:16:43.274210+0000","last_undegraded":"2024-04-22T01:16:43.274210+0000","last_fullsized":"2024-04-22T01:16:43.274210+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:21.076434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587958+0000","last_change":"2024-04-22T01:16:41.279655+0000","last_active":"2024-04-22T01:16:43.587958+0000","last_peered":"2024-04-22T01:16:43.587958+0000","last_clean":"2024-04-22T01:16:43.587958+0000","last_became_active":"2024-04-22T01:16:41.278708+0000","last_became_peered":"2024-04-22T01:16:41.278708+0000","last_unstale":"2024-04-22T01:16:43.587958+0000","last_undegraded":"2024-04-22T01:16:43.587958+0000","last_fullsized":"2024-04-22T01:16:43.587958+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:12.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587457+0000","last_change":"2024-04-22T01:16:42.318567+0000","last_active":"2024-04-22T01:16:43.587457+0000","last_peered":"2024-04-22T01:16:43.587457+0000","last_clean":"2024-04-22T01:16:43.587457+0000","last_became_active":"2024-04-22T01:16:42.318289+0000","last_became_peered":"2024-04-22T01:16:42.318289+0000","last_unstale":"2024-04-22T01:16:43.587457+0000","last_undegraded":"2024-04-22T01:16:43.587457+0000","last_fullsized":"2024-04-22T01:16:43.587457+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:22.563055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588014+0000","last_change":"2024-04-22T01:16:41.300777+0000","last_active":"2024-04-22T01:16:43.588014+0000","last_peered":"2024-04-22T01:16:43.588014+0000","last_clean":"2024-04-22T01:16:43.588014+0000","last_became_active":"2024-04-22T01:16:41.299861+0000","last_became_peered":"2024-04-22T01:16:41.299861+0000","last_unstale":"2024-04-22T01:16:43.588014+0000","last_undegraded":"2024-04-22T01:16:43.588014+0000","last_fullsized":"2024-04-22T01:16:43.588014+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:05.380345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587198+0000","last_change":"2024-04-22T01:16:42.280582+0000","last_active":"2024-04-22T01:16:43.587198+0000","last_peered":"2024-04-22T01:16:43.587198+0000","last_clean":"2024-04-22T01:16:43.587198+0000","last_became_active":"2024-04-22T01:16:42.279459+0000","last_became_peered":"2024-04-22T01:16:42.279459+0000","last_unstale":"2024-04-22T01:16:43.587198+0000","last_undegraded":"2024-04-22T01:16:43.587198+0000","last_fullsized":"2024-04-22T01:16:43.587198+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:45.056493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590798+0000","last_change":"2024-04-22T01:16:41.298783+0000","last_active":"2024-04-22T01:16:43.590798+0000","last_peered":"2024-04-22T01:16:43.590798+0000","last_clean":"2024-04-22T01:16:43.590798+0000","last_became_active":"2024-04-22T01:16:41.294126+0000","last_became_peered":"2024-04-22T01:16:41.294126+0000","last_unstale":"2024-04-22T01:16:43.590798+0000","last_undegraded":"2024-04-22T01:16:43.590798+0000","last_fullsized":"2024-04-22T01:16:43.590798+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:02.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274791+0000","last_change":"2024-04-22T01:16:42.316992+0000","last_active":"2024-04-22T01:16:43.274791+0000","last_peered":"2024-04-22T01:16:43.274791+0000","last_clean":"2024-04-22T01:16:43.274791+0000","last_became_active":"2024-04-22T01:16:42.316319+0000","last_became_peered":"2024-04-22T01:16:42.316319+0000","last_unstale":"2024-04-22T01:16:43.274791+0000","last_undegraded":"2024-04-22T01:16:43.274791+0000","last_fullsized":"2024-04-22T01:16:43.274791+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:21:23.616725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589944+0000","last_change":"2024-04-22T01:16:41.296260+0000","last_active":"2024-04-22T01:16:43.589944+0000","last_peered":"2024-04-22T01:16:43.589944+0000","last_clean":"2024-04-22T01:16:43.589944+0000","last_became_active":"2024-04-22T01:16:41.295404+0000","last_became_peered":"2024-04-22T01:16:41.295404+0000","last_unstale":"2024-04-22T01:16:43.589944+0000","last_undegraded":"2024-04-22T01:16:43.589944+0000","last_fullsized":"2024-04-22T01:16:43.589944+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:44.052756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588767+0000","last_change":"2024-04-22T01:16:42.314778+0000","last_active":"2024-04-22T01:16:43.588767+0000","last_peered":"2024-04-22T01:16:43.588767+0000","last_clean":"2024-04-22T01:16:43.588767+0000","last_became_active":"2024-04-22T01:16:42.314327+0000","last_became_peered":"2024-04-22T01:16:42.314327+0000","last_unstale":"2024-04-22T01:16:43.588767+0000","last_undegraded":"2024-04-22T01:16:43.588767+0000","last_fullsized":"2024-04-22T01:16:43.588767+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:25:30.883785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275861+0000","last_change":"2024-04-22T01:16:41.303302+0000","last_active":"2024-04-22T01:16:43.275861+0000","last_peered":"2024-04-22T01:16:43.275861+0000","last_clean":"2024-04-22T01:16:43.275861+0000","last_became_active":"2024-04-22T01:16:41.302610+0000","last_became_peered":"2024-04-22T01:16:41.302610+0000","last_unstale":"2024-04-22T01:16:43.275861+0000","last_undegraded":"2024-04-22T01:16:43.275861+0000","last_fullsized":"2024-04-22T01:16:43.275861+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:14:25.445161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591189+0000","last_change":"2024-04-22T01:16:42.313396+0000","last_active":"2024-04-22T01:16:43.591189+0000","last_peered":"2024-04-22T01:16:43.591189+0000","last_clean":"2024-04-22T01:16:43.591189+0000","last_became_active":"2024-04-22T01:16:42.313066+0000","last_became_peered":"2024-04-22T01:16:42.313066+0000","last_unstale":"2024-04-22T01:16:43.591189+0000","last_undegraded":"2024-04-22T01:16:43.591189+0000","last_fullsized":"2024-04-22T01:16:43.591189+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:09.047831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586754+0000","last_change":"2024-04-22T01:16:41.304915+0000","last_active":"2024-04-22T01:16:43.586754+0000","last_peered":"2024-04-22T01:16:43.586754+0000","last_clean":"2024-04-22T01:16:43.586754+0000","last_became_active":"2024-04-22T01:16:41.304190+0000","last_became_peered":"2024-04-22T01:16:41.304190+0000","last_unstale":"2024-04-22T01:16:43.586754+0000","last_undegraded":"2024-04-22T01:16:43.586754+0000","last_fullsized":"2024-04-22T01:16:43.586754+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:42.939970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589381+0000","last_change":"2024-04-22T01:16:42.314668+0000","last_active":"2024-04-22T01:16:43.589381+0000","last_peered":"2024-04-22T01:16:43.589381+0000","last_clean":"2024-04-22T01:16:43.589381+0000","last_became_active":"2024-04-22T01:16:42.314048+0000","last_became_peered":"2024-04-22T01:16:42.314048+0000","last_unstale":"2024-04-22T01:16:43.589381+0000","last_undegraded":"2024-04-22T01:16:43.589381+0000","last_fullsized":"2024-04-22T01:16:43.589381+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:33.573703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589473+0000","last_change":"2024-04-22T01:16:41.294520+0000","last_active":"2024-04-22T01:16:43.589473+0000","last_peered":"2024-04-22T01:16:43.589473+0000","last_clean":"2024-04-22T01:16:43.589473+0000","last_became_active":"2024-04-22T01:16:41.294221+0000","last_became_peered":"2024-04-22T01:16:41.294221+0000","last_unstale":"2024-04-22T01:16:43.589473+0000","last_undegraded":"2024-04-22T01:16:43.589473+0000","last_fullsized":"2024-04-22T01:16:43.589473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:09.167971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589003+0000","last_change":"2024-04-22T01:16:42.296959+0000","last_active":"2024-04-22T01:16:43.589003+0000","last_peered":"2024-04-22T01:16:43.589003+0000","last_clean":"2024-04-22T01:16:43.589003+0000","last_became_active":"2024-04-22T01:16:42.296698+0000","last_became_peered":"2024-04-22T01:16:42.296698+0000","last_unstale":"2024-04-22T01:16:43.589003+0000","last_undegraded":"2024-04-22T01:16:43.589003+0000","last_fullsized":"2024-04-22T01:16:43.589003+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:19.712873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589547+0000","last_change":"2024-04-22T01:16:41.300743+0000","last_active":"2024-04-22T01:16:43.589547+0000","last_peered":"2024-04-22T01:16:43.589547+0000","last_clean":"2024-04-22T01:16:43.589547+0000","last_became_active":"2024-04-22T01:16:41.300222+0000","last_became_peered":"2024-04-22T01:16:41.300222+0000","last_unstale":"2024-04-22T01:16:43.589547+0000","last_undegraded":"2024-04-22T01:16:43.589547+0000","last_fullsized":"2024-04-22T01:16:43.589547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:14.782936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272650+0000","last_change":"2024-04-22T01:16:42.300582+0000","last_active":"2024-04-22T01:16:43.272650+0000","last_peered":"2024-04-22T01:16:43.272650+0000","last_clean":"2024-04-22T01:16:43.272650+0000","last_became_active":"2024-04-22T01:16:42.300286+0000","last_became_peered":"2024-04-22T01:16:42.300286+0000","last_unstale":"2024-04-22T01:16:43.272650+0000","last_undegraded":"2024-04-22T01:16:43.272650+0000","last_fullsized":"2024-04-22T01:16:43.272650+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:03.530479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590547+0000","last_change":"2024-04-22T01:16:41.293170+0000","last_active":"2024-04-22T01:16:43.590547+0000","last_peered":"2024-04-22T01:16:43.590547+0000","last_clean":"2024-04-22T01:16:43.590547+0000","last_became_active":"2024-04-22T01:16:41.292688+0000","last_became_peered":"2024-04-22T01:16:41.292688+0000","last_unstale":"2024-04-22T01:16:43.590547+0000","last_undegraded":"2024-04-22T01:16:43.590547+0000","last_fullsized":"2024-04-22T01:16:43.590547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:54.914899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586279+0000","last_change":"2024-04-22T01:16:42.297878+0000","last_active":"2024-04-22T01:16:43.586279+0000","last_peered":"2024-04-22T01:16:43.586279+0000","last_clean":"2024-04-22T01:16:43.586279+0000","last_became_active":"2024-04-22T01:16:42.297155+0000","last_became_peered":"2024-04-22T01:16:42.297155+0000","last_unstale":"2024-04-22T01:16:43.586279+0000","last_undegraded":"2024-04-22T01:16:43.586279+0000","last_fullsized":"2024-04-22T01:16:43.586279+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:43.943708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276775+0000","last_change":"2024-04-22T01:16:41.293911+0000","last_active":"2024-04-22T01:16:43.276775+0000","last_peered":"2024-04-22T01:16:43.276775+0000","last_clean":"2024-04-22T01:16:43.276775+0000","last_became_active":"2024-04-22T01:16:41.293166+0000","last_became_peered":"2024-04-22T01:16:41.293166+0000","last_unstale":"2024-04-22T01:16:43.276775+0000","last_undegraded":"2024-04-22T01:16:43.276775+0000","last_fullsized":"2024-04-22T01:16:43.276775+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:31.626341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272678+0000","last_change":"2024-04-22T01:16:42.292637+0000","last_active":"2024-04-22T01:16:43.272678+0000","last_peered":"2024-04-22T01:16:43.272678+0000","last_clean":"2024-04-22T01:16:43.272678+0000","last_became_active":"2024-04-22T01:16:42.292153+0000","last_became_peered":"2024-04-22T01:16:42.292153+0000","last_unstale":"2024-04-22T01:16:43.272678+0000","last_undegraded":"2024-04-22T01:16:43.272678+0000","last_fullsized":"2024-04-22T01:16:43.272678+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:40.004739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276503+0000","last_change":"2024-04-22T01:16:41.306902+0000","last_active":"2024-04-22T01:16:43.276503+0000","last_peered":"2024-04-22T01:16:43.276503+0000","last_clean":"2024-04-22T01:16:43.276503+0000","last_became_active":"2024-04-22T01:16:41.306227+0000","last_became_peered":"2024-04-22T01:16:41.306227+0000","last_unstale":"2024-04-22T01:16:43.276503+0000","last_undegraded":"2024-04-22T01:16:43.276503+0000","last_fullsized":"2024-04-22T01:16:43.276503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.432697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275046+0000","last_change":"2024-04-22T01:16:42.284747+0000","last_active":"2024-04-22T01:16:43.275046+0000","last_peered":"2024-04-22T01:16:43.275046+0000","last_clean":"2024-04-22T01:16:43.275046+0000","last_became_active":"2024-04-22T01:16:42.284004+0000","last_became_peered":"2024-04-22T01:16:42.284004+0000","last_unstale":"2024-04-22T01:16:43.275046+0000","last_undegraded":"2024-04-22T01:16:43.275046+0000","last_fullsized":"2024-04-22T01:16:43.275046+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:58:39.203936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.272791+0000","last_change":"2024-04-22T01:16:41.288670+0000","last_active":"2024-04-22T01:16:43.272791+0000","last_peered":"2024-04-22T01:16:43.272791+0000","last_clean":"2024-04-22T01:16:43.272791+0000","last_became_active":"2024-04-22T01:16:41.288375+0000","last_became_peered":"2024-04-22T01:16:41.288375+0000","last_unstale":"2024-04-22T01:16:43.272791+0000","last_undegraded":"2024-04-22T01:16:43.272791+0000","last_fullsized":"2024-04-22T01:16:43.272791+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:12.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276513+0000","last_change":"2024-04-22T01:16:42.311799+0000","last_active":"2024-04-22T01:16:43.276513+0000","last_peered":"2024-04-22T01:16:43.276513+0000","last_clean":"2024-04-22T01:16:43.276513+0000","last_became_active":"2024-04-22T01:16:42.311064+0000","last_became_peered":"2024-04-22T01:16:42.311064+0000","last_unstale":"2024-04-22T01:16:43.276513+0000","last_undegraded":"2024-04-22T01:16:43.276513+0000","last_fullsized":"2024-04-22T01:16:43.276513+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:02:25.597982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588794+0000","last_change":"2024-04-22T01:16:41.300766+0000","last_active":"2024-04-22T01:16:43.588794+0000","last_peered":"2024-04-22T01:16:43.588794+0000","last_clean":"2024-04-22T01:16:43.588794+0000","last_became_active":"2024-04-22T01:16:41.300346+0000","last_became_peered":"2024-04-22T01:16:41.300346+0000","last_unstale":"2024-04-22T01:16:43.588794+0000","last_undegraded":"2024-04-22T01:16:43.588794+0000","last_fullsized":"2024-04-22T01:16:43.588794+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:39.001001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589508+0000","last_change":"2024-04-22T01:16:42.319426+0000","last_active":"2024-04-22T01:16:43.589508+0000","last_peered":"2024-04-22T01:16:43.589508+0000","last_clean":"2024-04-22T01:16:43.589508+0000","last_became_active":"2024-04-22T01:16:42.315069+0000","last_became_peered":"2024-04-22T01:16:42.315069+0000","last_unstale":"2024-04-22T01:16:43.589508+0000","last_undegraded":"2024-04-22T01:16:43.589508+0000","last_fullsized":"2024-04-22T01:16:43.589508+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:12.850534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587985+0000","last_change":"2024-04-22T01:16:41.279372+0000","last_active":"2024-04-22T01:16:43.587985+0000","last_peered":"2024-04-22T01:16:43.587985+0000","last_clean":"2024-04-22T01:16:43.587985+0000","last_became_active":"2024-04-22T01:16:41.277927+0000","last_became_peered":"2024-04-22T01:16:41.277927+0000","last_unstale":"2024-04-22T01:16:43.587985+0000","last_undegraded":"2024-04-22T01:16:43.587985+0000","last_fullsized":"2024-04-22T01:16:43.587985+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:42.939970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273665+0000","last_change":"2024-04-22T01:16:42.293982+0000","last_active":"2024-04-22T01:16:43.273665+0000","last_peered":"2024-04-22T01:16:43.273665+0000","last_clean":"2024-04-22T01:16:43.273665+0000","last_became_active":"2024-04-22T01:16:42.293593+0000","last_became_peered":"2024-04-22T01:16:42.293593+0000","last_unstale":"2024-04-22T01:16:43.273665+0000","last_undegraded":"2024-04-22T01:16:43.273665+0000","last_fullsized":"2024-04-22T01:16:43.273665+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:21.076434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.587501+0000","last_change":"2024-04-22T01:16:41.307711+0000","last_active":"2024-04-22T01:16:43.587501+0000","last_peered":"2024-04-22T01:16:43.587501+0000","last_clean":"2024-04-22T01:16:43.587501+0000","last_became_active":"2024-04-22T01:16:41.307267+0000","last_became_peered":"2024-04-22T01:16:41.307267+0000","last_unstale":"2024-04-22T01:16:43.587501+0000","last_undegraded":"2024-04-22T01:16:43.587501+0000","last_fullsized":"2024-04-22T01:16:43.587501+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:32:10.417022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274228+0000","last_change":"2024-04-22T01:16:42.294626+0000","last_active":"2024-04-22T01:16:43.274228+0000","last_peered":"2024-04-22T01:16:43.274228+0000","last_clean":"2024-04-22T01:16:43.274228+0000","last_became_active":"2024-04-22T01:16:42.293865+0000","last_became_peered":"2024-04-22T01:16:42.293865+0000","last_unstale":"2024-04-22T01:16:43.274228+0000","last_undegraded":"2024-04-22T01:16:43.274228+0000","last_fullsized":"2024-04-22T01:16:43.274228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:52:44.520751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.276559+0000","last_change":"2024-04-22T01:16:41.303104+0000","last_active":"2024-04-22T01:16:43.276559+0000","last_peered":"2024-04-22T01:16:43.276559+0000","last_clean":"2024-04-22T01:16:43.276559+0000","last_became_active":"2024-04-22T01:16:41.302511+0000","last_became_peered":"2024-04-22T01:16:41.302511+0000","last_unstale":"2024-04-22T01:16:43.276559+0000","last_undegraded":"2024-04-22T01:16:43.276559+0000","last_fullsized":"2024-04-22T01:16:43.276559+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:00:51.721140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273143+0000","last_change":"2024-04-22T01:16:42.287083+0000","last_active":"2024-04-22T01:16:43.273143+0000","last_peered":"2024-04-22T01:16:43.273143+0000","last_clean":"2024-04-22T01:16:43.273143+0000","last_became_active":"2024-04-22T01:16:42.286782+0000","last_became_peered":"2024-04-22T01:16:42.286782+0000","last_unstale":"2024-04-22T01:16:43.273143+0000","last_undegraded":"2024-04-22T01:16:43.273143+0000","last_fullsized":"2024-04-22T01:16:43.273143+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:55.123394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.588267+0000","last_change":"2024-04-22T01:16:41.305635+0000","last_active":"2024-04-22T01:16:43.588267+0000","last_peered":"2024-04-22T01:16:43.588267+0000","last_clean":"2024-04-22T01:16:43.588267+0000","last_became_active":"2024-04-22T01:16:41.305094+0000","last_became_peered":"2024-04-22T01:16:41.305094+0000","last_unstale":"2024-04-22T01:16:43.588267+0000","last_undegraded":"2024-04-22T01:16:43.588267+0000","last_fullsized":"2024-04-22T01:16:43.588267+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:44.052756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590288+0000","last_change":"2024-04-22T01:16:42.307611+0000","last_active":"2024-04-22T01:16:43.590288+0000","last_peered":"2024-04-22T01:16:43.590288+0000","last_clean":"2024-04-22T01:16:43.590288+0000","last_became_active":"2024-04-22T01:16:42.307270+0000","last_became_peered":"2024-04-22T01:16:42.307270+0000","last_unstale":"2024-04-22T01:16:43.590288+0000","last_undegraded":"2024-04-22T01:16:43.590288+0000","last_fullsized":"2024-04-22T01:16:43.590288+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:18.176769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589307+0000","last_change":"2024-04-22T01:16:41.290391+0000","last_active":"2024-04-22T01:16:43.589307+0000","last_peered":"2024-04-22T01:16:43.589307+0000","last_clean":"2024-04-22T01:16:43.589307+0000","last_became_active":"2024-04-22T01:16:41.289895+0000","last_became_peered":"2024-04-22T01:16:41.289895+0000","last_unstale":"2024-04-22T01:16:43.589307+0000","last_undegraded":"2024-04-22T01:16:43.589307+0000","last_fullsized":"2024-04-22T01:16:43.589307+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:32.376868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589836+0000","last_change":"2024-04-22T01:16:42.313744+0000","last_active":"2024-04-22T01:16:43.589836+0000","last_peered":"2024-04-22T01:16:43.589836+0000","last_clean":"2024-04-22T01:16:43.589836+0000","last_became_active":"2024-04-22T01:16:42.313438+0000","last_became_peered":"2024-04-22T01:16:42.313438+0000","last_unstale":"2024-04-22T01:16:43.589836+0000","last_undegraded":"2024-04-22T01:16:43.589836+0000","last_fullsized":"2024-04-22T01:16:43.589836+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:25.700498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.590170+0000","last_change":"2024-04-22T01:16:41.294486+0000","last_active":"2024-04-22T01:16:43.590170+0000","last_peered":"2024-04-22T01:16:43.590170+0000","last_clean":"2024-04-22T01:16:43.590170+0000","last_became_active":"2024-04-22T01:16:41.294147+0000","last_became_peered":"2024-04-22T01:16:41.294147+0000","last_unstale":"2024-04-22T01:16:43.590170+0000","last_undegraded":"2024-04-22T01:16:43.590170+0000","last_fullsized":"2024-04-22T01:16:43.590170+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:54.119657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.275660+0000","last_change":"2024-04-22T01:16:42.318672+0000","last_active":"2024-04-22T01:16:43.275660+0000","last_peered":"2024-04-22T01:16:43.275660+0000","last_clean":"2024-04-22T01:16:43.275660+0000","last_became_active":"2024-04-22T01:16:42.318021+0000","last_became_peered":"2024-04-22T01:16:42.318021+0000","last_unstale":"2024-04-22T01:16:43.275660+0000","last_undegraded":"2024-04-22T01:16:43.275660+0000","last_fullsized":"2024-04-22T01:16:43.275660+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:19.311224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589034+0000","last_change":"2024-04-22T01:16:41.278493+0000","last_active":"2024-04-22T01:16:43.589034+0000","last_peered":"2024-04-22T01:16:43.589034+0000","last_clean":"2024-04-22T01:16:43.589034+0000","last_became_active":"2024-04-22T01:16:41.278001+0000","last_became_peered":"2024-04-22T01:16:41.278001+0000","last_unstale":"2024-04-22T01:16:43.589034+0000","last_undegraded":"2024-04-22T01:16:43.589034+0000","last_fullsized":"2024-04-22T01:16:43.589034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:02:24.594245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589145+0000","last_change":"2024-04-22T01:16:42.299128+0000","last_active":"2024-04-22T01:16:43.589145+0000","last_peered":"2024-04-22T01:16:43.589145+0000","last_clean":"2024-04-22T01:16:43.589145+0000","last_became_active":"2024-04-22T01:16:42.298790+0000","last_became_peered":"2024-04-22T01:16:42.298790+0000","last_unstale":"2024-04-22T01:16:43.589145+0000","last_undegraded":"2024-04-22T01:16:43.589145+0000","last_fullsized":"2024-04-22T01:16:43.589145+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:09.047831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586040+0000","last_change":"2024-04-22T01:16:41.276295+0000","last_active":"2024-04-22T01:16:43.586040+0000","last_peered":"2024-04-22T01:16:43.586040+0000","last_clean":"2024-04-22T01:16:43.586040+0000","last_became_active":"2024-04-22T01:16:41.273094+0000","last_became_peered":"2024-04-22T01:16:41.273094+0000","last_unstale":"2024-04-22T01:16:43.586040+0000","last_undegraded":"2024-04-22T01:16:43.586040+0000","last_fullsized":"2024-04-22T01:16:43.586040+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:13.230682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.591157+0000","last_change":"2024-04-22T01:16:42.293420+0000","last_active":"2024-04-22T01:16:43.591157+0000","last_peered":"2024-04-22T01:16:43.591157+0000","last_clean":"2024-04-22T01:16:43.591157+0000","last_became_active":"2024-04-22T01:16:42.293156+0000","last_became_peered":"2024-04-22T01:16:42.293156+0000","last_unstale":"2024-04-22T01:16:43.591157+0000","last_undegraded":"2024-04-22T01:16:43.591157+0000","last_fullsized":"2024-04-22T01:16:43.591157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:07.621817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.274462+0000","last_change":"2024-04-22T01:16:41.289193+0000","last_active":"2024-04-22T01:16:43.274462+0000","last_peered":"2024-04-22T01:16:43.274462+0000","last_clean":"2024-04-22T01:16:43.274462+0000","last_became_active":"2024-04-22T01:16:41.286911+0000","last_became_peered":"2024-04-22T01:16:41.286911+0000","last_unstale":"2024-04-22T01:16:43.274462+0000","last_undegraded":"2024-04-22T01:16:43.274462+0000","last_fullsized":"2024-04-22T01:16:43.274462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:54:32.376868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.273996+0000","last_change":"2024-04-22T01:16:42.300814+0000","last_active":"2024-04-22T01:16:43.273996+0000","last_peered":"2024-04-22T01:16:43.273996+0000","last_clean":"2024-04-22T01:16:43.273996+0000","last_became_active":"2024-04-22T01:16:42.299882+0000","last_became_peered":"2024-04-22T01:16:42.299882+0000","last_unstale":"2024-04-22T01:16:43.273996+0000","last_undegraded":"2024-04-22T01:16:43.273996+0000","last_fullsized":"2024-04-22T01:16:43.273996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:41.253791+0000","last_clean_scrub_stamp":"2024-04-22T01:16:41.253791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:45.056493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.589585+0000","last_change":"2024-04-22T01:16:41.295927+0000","last_active":"2024-04-22T01:16:43.589585+0000","last_peered":"2024-04-22T01:16:43.589585+0000","last_clean":"2024-04-22T01:16:43.589585+0000","last_became_active":"2024-04-22T01:16:41.295504+0000","last_became_peered":"2024-04-22T01:16:41.295504+0000","last_unstale":"2024-04-22T01:16:43.589585+0000","last_undegraded":"2024-04-22T01:16:43.589585+0000","last_fullsized":"2024-04-22T01:16:43.589585+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:16:40.250053+0000","last_clean_scrub_stamp":"2024-04-22T01:16:40.250053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:39.001001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T01:16:43.586608+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:16:43.586608+0000","last_peered":"2024-04-22T01:16:43.586608+0000","last_clean":"2024-04-22T01:16:43.586608+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:16:43.586608+0000","last_undegraded":"2024-04-22T01:16:43.586608+0000","last_fullsized":"2024-04-22T01:16:43.586608+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":859819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":269979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":269327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":269909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35956,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335884,"statfs":{"total":96636764160,"available":96599945216,"internally_reserved":0,"allocated":622592,"data_stored":270961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":270939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":276179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":859751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:16:46.674 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:16:46.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-22T01:16:46.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 Processor -- start 2024-04-22T01:16:46.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 -- start start 2024-04-22T01:16:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b80ce920 0x7fb8b8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8b80cedd0 0x7fb8b81551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b80b3800 con 0x7fb8b815e6e0 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b80b3ae0 con 0x7fb8b80ce920 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8befed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b80cde40 con 0x7fb8b80cedd0 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b80ce920 0x7fb8b8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b80ce920 0x7fb8b8154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60602/0 (socket says 172.21.15.70:60602) 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 -- 172.21.15.70:0/776208956 learned_addr learned my addr 172.21.15.70:0/776208956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8bd58a640 1 --2- 172.21.15.70:0/776208956 >> v2:172.21.15.98:3300/0 conn(0x7fb8b80cedd0 0x7fb8b81551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 -- 172.21.15.70:0/776208956 >> v2:172.21.15.98:3300/0 conn(0x7fb8b80cedd0 msgr2=0x7fb8b81551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 --2- 172.21.15.70:0/776208956 >> v2:172.21.15.98:3300/0 conn(0x7fb8b80cedd0 0x7fb8b81551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 -- 172.21.15.70:0/776208956 >> v1:172.21.15.98:6789/0 conn(0x7fb8b815e6e0 legacy=0x7fb8b815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.842+0000 7fb8b7fff640 1 -- 172.21.15.70:0/776208956 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b80ce120 con 0x7fb8b80ce920 2024-04-22T01:16:46.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8b7fff640 1 --2- 172.21.15.70:0/776208956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b8154c60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8158cb0 tx=0x7fb8ac01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78628b53b9ca0603 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8b77fe640 1 -- 172.21.15.70:0/776208956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0aa670 con 0x7fb8b80ce920 2024-04-22T01:16:46.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8b77fe640 1 -- 172.21.15.70:0/776208956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ac0af070 con 0x7fb8b80ce920 2024-04-22T01:16:46.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8b77fe640 1 -- 172.21.15.70:0/776208956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0aaa30 con 0x7fb8b80ce920 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- 172.21.15.70:0/776208956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 msgr2=0x7fb8b8154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 --2- 172.21.15.70:0/776208956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b8154c60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8158cb0 tx=0x7fb8ac01c4f0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- 172.21.15.70:0/776208956 shutdown_connections 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 --2- 172.21.15.70:0/776208956 >> v2:172.21.15.98:3300/0 conn(0x7fb8b80cedd0 0x7fb8b81551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 --2- 172.21.15.70:0/776208956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b8154c60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- 172.21.15.70:0/776208956 >> 172.21.15.70:0/776208956 conn(0x7fb8b80b1d40 msgr2=0x7fb8b80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- 172.21.15.70:0/776208956 shutdown_connections 2024-04-22T01:16:46.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- 172.21.15.70:0/776208956 wait complete. 2024-04-22T01:16:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 Processor -- start 2024-04-22T01:16:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- start start 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b813f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80cedd0 0x7fb8b80a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e6e0 0x7fb8b814d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b80b2e80 con 0x7fb8b80ce920 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b80b2d00 con 0x7fb8b80cedd0 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8befed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b80b3000 con 0x7fb8b815e6e0 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b813f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60616/0 (socket says 172.21.15.70:60616) 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 -- 172.21.15.70:0/3175844473 learned_addr learned my addr 172.21.15.70:0/3175844473 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bd58a640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e6e0 0x7fb8b814d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8b7fff640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80cedd0 0x7fb8b80a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 -- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e6e0 msgr2=0x7fb8b814d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e6e0 0x7fb8b814d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 -- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80cedd0 msgr2=0x7fb8b80a86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80cedd0 0x7fb8b80a86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 -- 172.21.15.70:0/3175844473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b80af910 con 0x7fb8b80ce920 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bd58a640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e6e0 0x7fb8b814d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8bcd89640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b813f290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c370 tx=0x7fb8a800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.846+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80052d0 con 0x7fb8b80ce920 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a8005470 con 0x7fb8b80ce920 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8007260 con 0x7fb8b80ce920 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b80afbb0 con 0x7fb8b80ce920 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b80b0170 con 0x7fb8b80ce920 2024-04-22T01:16:46.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b8068210 con 0x7fb8b80ce920 2024-04-22T01:16:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb8a8007400 con 0x7fb8b80ce920 2024-04-22T01:16:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b57fa640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fb8a8085330 con 0x7fb8b80ce920 2024-04-22T01:16:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b7fff640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:46.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.850+0000 7fb8b7fff640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0ae000 tx=0x7fb8ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:46.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:46.854+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb8a8055c50 con 0x7fb8b80ce920 2024-04-22T01:16:47.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.078+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb8b80d08e0 con 0x7fb8b80ce920 2024-04-22T01:16:47.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.078+0000 7fb8b57fa640 1 -- 172.21.15.70:0/3175844473 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7fb8a8055550 con 0x7fb8b80ce920 2024-04-22T01:16:47.081 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:16:47.081 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:16:47.081 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-22T01:16:47.081 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-22T01:16:47.082 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-22T01:16:47.082 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 msgr2=0x7fb8a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0ae000 tx=0x7fb8ac008040 comp rx=0 tx=0).stop 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 msgr2=0x7fb8b813f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b813f290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c370 tx=0x7fb8a800c830 comp rx=0 tx=0).stop 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 shutdown_connections 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e6e0 0x7fb8b814d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80cedd0 0x7fb8b80a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 --2- 172.21.15.70:0/3175844473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80ce920 0x7fb8b813f290 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 >> 172.21.15.70:0/3175844473 conn(0x7fb8b80b1d40 msgr2=0x7fb8b814f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 shutdown_connections 2024-04-22T01:16:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.082+0000 7fb8befed640 1 -- 172.21.15.70:0/3175844473 wait complete. 2024-04-22T01:16:47.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:47.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 Processor -- start 2024-04-22T01:16:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 -- start start 2024-04-22T01:16:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 0x7fdc7806e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc780d0db0 0x7fdc78066f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7806ed40 con 0x7fdc780cfcc0 2024-04-22T01:16:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdc7806bc00 con 0x7fdc780d0db0 2024-04-22T01:16:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7ce7d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7806bee0 con 0x7fdc7806e420 2024-04-22T01:16:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 0x7fdc7806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 0x7fdc7806e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:59246/0 (socket says 172.21.15.70:59246) 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7659c640 1 -- 172.21.15.70:0/2762558501 learned_addr learned my addr 172.21.15.70:0/2762558501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc75d9b640 1 --2- 172.21.15.70:0/2762558501 >> v2:172.21.15.98:3300/0 conn(0x7fdc780d0db0 0x7fdc78066f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc75d9b640 1 -- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 msgr2=0x7fdc7806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc75d9b640 1 --2- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 0x7fdc7806e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc75d9b640 1 -- 172.21.15.70:0/2762558501 >> v1:172.21.15.98:6789/0 conn(0x7fdc780cfcc0 legacy=0x7fdc780d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc75d9b640 1 -- 172.21.15.70:0/2762558501 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7806c1c0 con 0x7fdc780d0db0 2024-04-22T01:16:47.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.218+0000 7fdc7659c640 1 --2- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 0x7fdc7806e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:16:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc75d9b640 1 --2- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc78066f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0097c0 tx=0x7fdc6c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a45d1b252a761 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7559a640 1 -- 172.21.15.70:0/2762558501 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c0b5070 con 0x7fdc780d0db0 2024-04-22T01:16:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7559a640 1 -- 172.21.15.70:0/2762558501 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6c0a8990 con 0x7fdc780d0db0 2024-04-22T01:16:47.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7559a640 1 -- 172.21.15.70:0/2762558501 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c0a8d60 con 0x7fdc780d0db0 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 msgr2=0x7fdc78066f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc78066f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0097c0 tx=0x7fdc6c01ec90 comp rx=0 tx=0).stop 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/2762558501 shutdown_connections 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc78066f50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/2762558501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc7806e420 0x7fdc7806e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/2762558501 >> 172.21.15.70:0/2762558501 conn(0x7fdc780b3d90 msgr2=0x7fdc780b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/2762558501 shutdown_connections 2024-04-22T01:16:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/2762558501 wait complete. 2024-04-22T01:16:47.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 Processor -- start 2024-04-22T01:16:47.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.222+0000 7fdc7ce7d640 1 -- start start 2024-04-22T01:16:47.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 0x7fdc7815f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780cfcc0 0x7fdc7815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc781f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78067490 con 0x7fdc7806e420 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7806c780 con 0x7fdc780d0db0 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7815e180 con 0x7fdc780cfcc0 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 0x7fdc7815f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 0x7fdc7815f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60628/0 (socket says 172.21.15.70:60628) 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 -- 172.21.15.70:0/4169330417 learned_addr learned my addr 172.21.15.70:0/4169330417 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc75d9b640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780cfcc0 0x7fdc7815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc76d9d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc781f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 -- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780cfcc0 msgr2=0x7fdc7815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780cfcc0 0x7fdc7815f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 -- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 msgr2=0x7fdc781f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc781f7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 -- 172.21.15.70:0/4169330417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc78160110 con 0x7fdc7806e420 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc76d9d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc781f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7659c640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 0x7fdc7815f380 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800e930 tx=0x7fdc6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc68017070 con 0x7fdc7806e420 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6800ca00 con 0x7fdc7806e420 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc78153320 con 0x7fdc7806e420 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6800cdd0 con 0x7fdc7806e420 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc781538e0 con 0x7fdc7806e420 2024-04-22T01:16:47.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc78068920 con 0x7fdc7806e420 2024-04-22T01:16:47.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc6801a070 con 0x7fdc7806e420 2024-04-22T01:16:47.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc66ffd640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc580678c0 0x7fdc58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc75d9b640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc580678c0 0x7fdc58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fdc68085d20 con 0x7fdc7806e420 2024-04-22T01:16:47.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.226+0000 7fdc75d9b640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc580678c0 0x7fdc58069d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0b2000 tx=0x7fdc6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.230+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdc680566d0 con 0x7fdc7806e420 2024-04-22T01:16:47.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.414+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc7815b440 con 0x7fdc7806e420 2024-04-22T01:16:47.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc66ffd640 1 -- 172.21.15.70:0/4169330417 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5178 (secure 0 0 0) 0x7fdc68055fd0 con 0x7fdc7806e420 2024-04-22T01:16:47.418 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:47.418 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:41.643333+0000","modified":"2024-04-22T01:16:44.689368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5922},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5922":{"gid":5922,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2188446179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2188446179},{"type":"v1","addr":"172.21.15.70:6837","nonce":2188446179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5922,"qdb_cluster":[5922]},"id":3}]} 2024-04-22T01:16:47.418 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 25 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc580678c0 msgr2=0x7fdc58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc580678c0 0x7fdc58069d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0b2000 tx=0x7fdc6c008040 comp rx=0 tx=0).stop 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 msgr2=0x7fdc7815f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 0x7fdc7815f380 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800e930 tx=0x7fdc6800edf0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 shutdown_connections 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc580678c0 0x7fdc58069d80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780d0db0 0x7fdc781f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780cfcc0 0x7fdc7815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 --2- 172.21.15.70:0/4169330417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7806e420 0x7fdc7815f380 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 >> 172.21.15.70:0/4169330417 conn(0x7fdc780b3d90 msgr2=0x7fdc780ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:47.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 shutdown_connections 2024-04-22T01:16:47.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.418+0000 7fdc7ce7d640 1 -- 172.21.15.70:0/4169330417 wait complete. 2024-04-22T01:16:47.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:16:41.643333+0000', 'modified': '2024-04-22T01:16:44.689368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5922}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5922': {'gid': 5922, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/2188446179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6835', 'nonce': 2188446179}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 2188446179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5922, 'qdb_cluster': [5922]} 2024-04-22T01:16:47.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:16:47.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-22T01:16:47.912 INFO:tasks.ceph.mds.a.smithi070.stderr:2024-04-22T01:16:47.910+0000 7f831935f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:16:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 Processor -- start 2024-04-22T01:16:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 -- start start 2024-04-22T01:16:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383415bb50 0x7f38340cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f383415bf30 0x7f383406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38341605e0 0x7f383406f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f383415f8b0 con 0x7f383415bb50 2024-04-22T01:16:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f38340cd7a0 con 0x7f383415bf30 2024-04-22T01:16:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f383a0b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f383415fb00 con 0x7f38341605e0 2024-04-22T01:16:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f38337fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383415bb50 0x7f38340cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f3833fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38341605e0 0x7f383406f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f3833fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38341605e0 0x7f383406f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:59274/0 (socket says 172.21.15.70:59274) 2024-04-22T01:16:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f3833fff640 1 -- 172.21.15.70:0/163801914 learned_addr learned my addr 172.21.15.70:0/163801914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f382affd640 1 --2- 172.21.15.70:0/163801914 >> v2:172.21.15.98:3300/0 conn(0x7f383415bf30 0x7f383406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f38337fe640 1 -- 172.21.15.70:0/163801914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38341605e0 msgr2=0x7f383406f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f38337fe640 1 --2- 172.21.15.70:0/163801914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38341605e0 0x7f383406f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f38337fe640 1 -- 172.21.15.70:0/163801914 >> v2:172.21.15.98:3300/0 conn(0x7f383415bf30 msgr2=0x7f383406ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f38337fe640 1 --2- 172.21.15.70:0/163801914 >> v2:172.21.15.98:3300/0 conn(0x7f383415bf30 0x7f383406ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.966+0000 7f38337fe640 1 -- 172.21.15.70:0/163801914 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383415fde0 con 0x7f383415bb50 2024-04-22T01:16:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f38337fe640 1 --2- 172.21.15.70:0/163801914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383415bb50 0x7f38340cce70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f383415fd10 tx=0x7f382401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958c9dd3af6b803a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3832ffd640 1 -- 172.21.15.70:0/163801914 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38240b0070 con 0x7f383415bb50 2024-04-22T01:16:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3832ffd640 1 -- 172.21.15.70:0/163801914 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38240ab600 con 0x7f383415bb50 2024-04-22T01:16:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3832ffd640 1 -- 172.21.15.70:0/163801914 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38240ab9d0 con 0x7f383415bb50 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- 172.21.15.70:0/163801914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383415bb50 msgr2=0x7f38340cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- 172.21.15.70:0/163801914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383415bb50 0x7f38340cce70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f383415fd10 tx=0x7f382401c460 comp rx=0 tx=0).stop 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- 172.21.15.70:0/163801914 shutdown_connections 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- 172.21.15.70:0/163801914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38341605e0 0x7f383406f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- 172.21.15.70:0/163801914 >> v2:172.21.15.98:3300/0 conn(0x7f383415bf30 0x7f383406ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- 172.21.15.70:0/163801914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383415bb50 0x7f38340cce70 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- 172.21.15.70:0/163801914 >> 172.21.15.70:0/163801914 conn(0x7f38340cf9e0 msgr2=0x7f38340cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:47.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- 172.21.15.70:0/163801914 shutdown_connections 2024-04-22T01:16:47.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- 172.21.15.70:0/163801914 wait complete. 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 Processor -- start 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- start start 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f383415bb50 0x7f383414ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f383415bf30 0x7f3834140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 0x7f38340b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f383406e6f0 con 0x7f38341605e0 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f383406e570 con 0x7f383415bf30 2024-04-22T01:16:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f383a0b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f383406e870 con 0x7f383415bb50 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 0x7f38340b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 0x7f38340b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60650/0 (socket says 172.21.15.70:60650) 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 -- 172.21.15.70:0/914094921 learned_addr learned my addr 172.21.15.70:0/914094921 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f382affd640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f383415bf30 0x7f3834140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f38337fe640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f383415bb50 0x7f383414ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 -- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f383415bb50 msgr2=0x7f383414ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f383415bb50 0x7f383414ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 -- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f383415bf30 msgr2=0x7f3834140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f383415bf30 0x7f3834140be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f3833fff640 1 -- 172.21.15.70:0/914094921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3834151980 con 0x7f38341605e0 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.970+0000 7f382affd640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f383415bf30 0x7f3834140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3833fff640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 0x7f38340b34e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f382c00efc0 tx=0x7f382c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3830ff9640 1 -- 172.21.15.70:0/914094921 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c00cd20 con 0x7f38341605e0 2024-04-22T01:16:47.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3830ff9640 1 -- 172.21.15.70:0/914094921 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382c0039a0 con 0x7f38341605e0 2024-04-22T01:16:47.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3834151c60 con 0x7f38341605e0 2024-04-22T01:16:47.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3830ff9640 1 -- 172.21.15.70:0/914094921 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c003d90 con 0x7f38341605e0 2024-04-22T01:16:47.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38340b2fc0 con 0x7f38341605e0 2024-04-22T01:16:47.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3830ff9640 1 -- 172.21.15.70:0/914094921 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f382c012050 con 0x7f38341605e0 2024-04-22T01:16:47.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3830ff9640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f38140678c0 0x7f3814069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f38337fe640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f38140678c0 0x7f3814069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f3830ff9640 1 -- 172.21.15.70:0/914094921 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f382c086360 con 0x7f38341605e0 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f38337fe640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f38140678c0 0x7f3814069d80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f38240ad000 tx=0x7f3824008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f383d22c000 1 Processor -- start 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.974+0000 7f383d22c000 1 -- start start 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d0692d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d01f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558b6ce6d5e0 con 0x558b6d01fbc0 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558b6cdf4c50 con 0x558b6d069810 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x558b6ce695a0 con 0x558b6d068ef0 2024-04-22T01:16:47.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60660/0 (socket says 172.21.15.70:60660) 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 -- 172.21.15.70:0/1862491596 learned_addr learned my addr 172.21.15.70:0/1862491596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d0692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 -- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 msgr2=0x558b6d0692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d0692d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 -- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 msgr2=0x558b6d01f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d01f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 -- 172.21.15.70:0/1862491596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b6cdf2d10 con 0x558b6d01fbc0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d0692d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01ffa0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f382c0093e0 tx=0x7f382c055f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb295236bf3ac4c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38297fa640 1 -- 172.21.15.70:0/1862491596 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c05a920 con 0x558b6d01fbc0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38297fa640 1 -- 172.21.15.70:0/1862491596 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382c05f150 con 0x558b6d01fbc0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38297fa640 1 -- 172.21.15.70:0/1862491596 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c086070 con 0x558b6d01fbc0 2024-04-22T01:16:47.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/1862491596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 msgr2=0x558b6d01ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01ffa0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f382c0093e0 tx=0x7f382c055f90 comp rx=0 tx=0).stop 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/1862491596 shutdown_connections 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01ffa0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d01f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- 172.21.15.70:0/1862491596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d0692d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/1862491596 >> 172.21.15.70:0/1862491596 conn(0x558b6d01bb70 msgr2=0x558b6d01df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/1862491596 shutdown_connections 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/1862491596 wait complete. 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- start start 2024-04-22T01:16:47.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 Processor -- start 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d04df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d04e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d04e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558b6ce27520 con 0x558b6d01fbc0 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558b6ce13810 con 0x558b6d069810 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x558b6d01b9f0 con 0x558b6d068ef0 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 10 client.0 _ll_register_callbacks cb 0x558b6d00a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:16:47.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d04df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d04df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60676/0 (socket says 172.21.15.70:60676) 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 -- 172.21.15.70:0/2607701622 learned_addr learned my addr 172.21.15.70:0/2607701622 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f382affd640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d04e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d04e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 -- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 msgr2=0x558b6d04e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d04e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 -- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 msgr2=0x558b6d04e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d04e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 -- 172.21.15.70:0/2607701622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b6ce057b0 con 0x558b6d01fbc0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f3833fff640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d04e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38337fe640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d04df20 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f38240ae040 tx=0x7f38240046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5cc29ae866e9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38127fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38127fc640 1 -- 172.21.15.70:0/2607701622 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38240c7060 con 0x558b6d01fbc0 2024-04-22T01:16:47.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38127fc640 1 -- 172.21.15.70:0/2607701622 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f38240ab810 con 0x558b6d01fbc0 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382c05f150 con 0x558b6d01fbc0 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f38127fc640 1 -- 172.21.15.70:0/2607701622 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f38240cd050 con 0x558b6d01fbc0 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 10 client.6066 fetch_fsmap learned FSMap version 25 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.978+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x558b6cdf4ea0 con 0x558b6d01fbc0 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f38127fc640 1 -- 172.21.15.70:0/2607701622 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382c05f150 con 0x558b6d01fbc0 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f38127fc640 1 -- 172.21.15.70:0/2607701622 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f38240d0020 con 0x558b6d01fbc0 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 10 client.6066 fetch_fsmap finished waiting for FSMap version 25 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 10 client.6066 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 20 client.6066 populate_metadata read hostname 'smithi070' 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d078f90 0x558b6d07b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 4 client.6066 mds_command: new command op to 5922 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 --> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b6ce08ab0 con 0x558b6d078f90 2024-04-22T01:16:47.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f382affd640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d078f90 0x558b6d07b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f382affd640 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d078f90 0x558b6d07b380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d165390353c061 server_cookie=f8e762655a40334a in_seq=0 out_seq=0 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f38127fc640 10 client.6066 ms_handle_connect on v2:172.21.15.70:6835/2188446179 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f38127fc640 1 -- 172.21.15.70:0/2607701622 <== mds.0 v2:172.21.15.70:6835/2188446179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b6ce08ab0 con 0x558b6d078f90 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f38127fc640 10 client.6066 handle_command_reply: tid=0 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 client.6066 shutdown 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 msgr2=0x558b6d04df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d04df20 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f38240ae040 tx=0x7f38240046d0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 shutdown_connections 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d078f90 0x558b6d07b380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d04e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d068ef0 0x558b6d04e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 --2- 172.21.15.70:0/2607701622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d04df20 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:47.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 >> 172.21.15.70:0/2607701622 conn(0x558b6d01bb70 msgr2=0x558b6d01e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:47.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f38127fc640 0 client.6066 ms_handle_reset on v2:172.21.15.70:6835/2188446179 2024-04-22T01:16:47.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 shutdown_connections 2024-04-22T01:16:47.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 1 -- 172.21.15.70:0/2607701622 wait complete. 2024-04-22T01:16:47.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.982+0000 7f383d22c000 20 client.6066 trim_cache size 0 max 16384 2024-04-22T01:16:48.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.998+0000 7f383d22c000 1 Processor -- start 2024-04-22T01:16:48.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:47.998+0000 7f383d22c000 1 -- start start 2024-04-22T01:16:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d068ef0 0x558b6d01e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d069810 0x558b6d01eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3834203ba0 con 0x558b6d01fbc0 2024-04-22T01:16:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3834202ba0 con 0x558b6d068ef0 2024-04-22T01:16:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3834151740 con 0x558b6d069810 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60690/0 (socket says 172.21.15.70:60690) 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 -- 172.21.15.70:0/3749911926 learned_addr learned my addr 172.21.15.70:0/3749911926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 -- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d069810 msgr2=0x558b6d01eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f3833fff640 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d069810 0x558b6d01eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d069810 0x558b6d01eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d068ef0 0x558b6d01e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 -- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d068ef0 msgr2=0x558b6d01e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d068ef0 0x558b6d01e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 -- 172.21.15.70:0/3749911926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b6cdf5e80 con 0x558b6d01fbc0 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f3833fff640 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d069810 0x558b6d01eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01df60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3824008010 tx=0x7f3824009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb31f1a7cb1bf1d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f3828ff9640 1 -- 172.21.15.70:0/3749911926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38240d5020 con 0x558b6d01fbc0 2024-04-22T01:16:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f3828ff9640 1 -- 172.21.15.70:0/3749911926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38240b1040 con 0x558b6d01fbc0 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f3828ff9640 1 -- 172.21.15.70:0/3749911926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38240c7030 con 0x558b6d01fbc0 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- 172.21.15.70:0/3749911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 msgr2=0x558b6d01df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01df60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3824008010 tx=0x7f3824009190 comp rx=0 tx=0).stop 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- 172.21.15.70:0/3749911926 shutdown_connections 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d069810 0x558b6d01eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d068ef0 0x558b6d01e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- 172.21.15.70:0/3749911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d01fbc0 0x558b6d01df60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- 172.21.15.70:0/3749911926 >> 172.21.15.70:0/3749911926 conn(0x558b6d01bb70 msgr2=0x558b6d0764e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- 172.21.15.70:0/3749911926 shutdown_connections 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- 172.21.15.70:0/3749911926 wait complete. 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- start start 2024-04-22T01:16:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 Processor -- start 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d01fbc0 0x558b6d0e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 0x558b6d0e1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d07b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558b6ce79180 con 0x558b6d068ef0 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f383413dd10 con 0x558b6d069810 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x558b6cf1fe80 con 0x558b6d01fbc0 2024-04-22T01:16:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 10 client.0 _ll_register_callbacks cb 0x7f38240ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 0x558b6d0e1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 0x558b6d0e1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60702/0 (socket says 172.21.15.70:60702) 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 -- 172.21.15.70:0/55418634 learned_addr learned my addr 172.21.15.70:0/55418634 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f3833fff640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d07b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d01fbc0 0x558b6d0e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 -- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d01fbc0 msgr2=0x558b6d0e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d01fbc0 0x558b6d0e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 -- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 msgr2=0x558b6d07b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d07b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 -- 172.21.15.70:0/55418634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b6ce3a760 con 0x558b6d068ef0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38337fe640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d01fbc0 0x558b6d0e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f382affd640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 0x558b6d0e1ca0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f381c000c10 tx=0x7f381c0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379e913fb82a78da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38127fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:16:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38127fc640 1 -- 172.21.15.70:0/55418634 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c00f040 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f38127fc640 1 -- 172.21.15.70:0/55418634 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f381c025420 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.002+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38240b1040 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 1 -- 172.21.15.70:0/55418634 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f381c02c050 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 10 client.6072 fetch_fsmap learned FSMap version 25 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x558b6cec69e0 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 1 -- 172.21.15.70:0/55418634 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38240b1040 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 1 -- 172.21.15.70:0/55418634 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f381c02f020 con 0x558b6d068ef0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 10 client.6072 fetch_fsmap finished waiting for FSMap version 25 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 10 client.6072 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 20 client.6072 populate_metadata read hostname 'smithi070' 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d07b7f0 0x558b6d07bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 4 client.6072 mds_command: new command op to 5922 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 --> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] -- command(tid 0: {"prefix": "status"}) -- 0x558b6ce7ab20 con 0x558b6d07b7f0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38337fe640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d07b7f0 0x558b6d07bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38337fe640 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d07b7f0 0x558b6d07bbd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178e4a459f236267 server_cookie=68d2056d457e0c45 in_seq=0 out_seq=0 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 10 client.6072 ms_handle_connect on v2:172.21.15.70:6835/2188446179 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 1 -- 172.21.15.70:0/55418634 <== mds.0 v2:172.21.15.70:6835/2188446179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x558b6ce7ab20 con 0x558b6d07b7f0 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 10 client.6072 handle_command_reply: tid=0 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 client.6072 shutdown 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 msgr2=0x558b6d0e1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 0x558b6d0e1ca0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f381c000c10 tx=0x7f381c0099b0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 shutdown_connections 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:6835/2188446179,v1:172.21.15.70:6837/2188446179] conn(0x558b6d07b7f0 0x558b6d07bbd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558b6d069810 0x558b6d07b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b6d068ef0 0x558b6d0e1ca0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f38127fc640 0 client.6072 ms_handle_reset on v2:172.21.15.70:6835/2188446179 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 --2- 172.21.15.70:0/55418634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558b6d01fbc0 0x558b6d0e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 >> 172.21.15.70:0/55418634 conn(0x558b6d01bb70 msgr2=0x558b6d067800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 shutdown_connections 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 1 -- 172.21.15.70:0/55418634 wait complete. 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383d22c000 20 client.6072 trim_cache size 0 max 16384 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5922, 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:16:48.013 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 5.7408835380000003, 2024-04-22T01:16:48.014 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 25, 2024-04-22T01:16:48.014 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 52, 2024-04-22T01:16:48.014 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 52, 2024-04-22T01:16:48.014 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 11.102083974999999 2024-04-22T01:16:48.014 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f38140678c0 msgr2=0x7f3814069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f38140678c0 0x7f3814069d80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f38240ad000 tx=0x7f3824008040 comp rx=0 tx=0).stop 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 msgr2=0x7f38340b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 0x7f38340b34e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f382c00efc0 tx=0x7f382c00c490 comp rx=0 tx=0).stop 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 shutdown_connections 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f38140678c0 0x7f3814069d80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38341605e0 0x7f38340b34e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f383415bf30 0x7f3834140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 --2- 172.21.15.70:0/914094921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f383415bb50 0x7f383414ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 >> 172.21.15.70:0/914094921 conn(0x7f38340cf9e0 msgr2=0x7f3834158380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 shutdown_connections 2024-04-22T01:16:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.006+0000 7f383a0b3640 1 -- 172.21.15.70:0/914094921 wait complete. 2024-04-22T01:16:48.023 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:16:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 Processor -- start 2024-04-22T01:16:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 -- start start 2024-04-22T01:16:48.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe9680b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe96815bb50 0x7fe9680ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96806dbd0 con 0x7fe9680aec70 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe9680af0a0 con 0x7fe96815bb50 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96ec4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe96806de70 con 0x7fe96815aa30 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe967fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe96815bb50 0x7fe9680ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96d1e7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe9680aec70 0x7fe96815e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56982/0 (socket says 172.21.15.70:56982) 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96d1e7640 1 -- 172.21.15.70:0/1410887586 learned_addr learned my addr 172.21.15.70:0/1410887586 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe96c9e6640 1 --2- 172.21.15.70:0/1410887586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe9680b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe967fff640 1 -- 172.21.15.70:0/1410887586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 msgr2=0x7fe9680b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe967fff640 1 --2- 172.21.15.70:0/1410887586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe9680b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe967fff640 1 -- 172.21.15.70:0/1410887586 >> v1:172.21.15.70:6789/0 conn(0x7fe9680aec70 legacy=0x7fe96815e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe967fff640 1 -- 172.21.15.70:0/1410887586 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96806e150 con 0x7fe96815bb50 2024-04-22T01:16:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.134+0000 7fe9677fe640 1 -- 172.21.15.70:0/1410887586 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920612882 0 0) 0x7fe96806dbd0 con 0x7fe9680aec70 2024-04-22T01:16:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe967fff640 1 --2- 172.21.15.70:0/1410887586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe96815bb50 0x7fe9680ae730 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe9580097c0 tx=0x7fe95801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61801186b82a28e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:48.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe9677fe640 1 -- 172.21.15.70:0/1410887586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580b0070 con 0x7fe96815bb50 2024-04-22T01:16:48.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe9677fe640 1 -- 172.21.15.70:0/1410887586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9580ab600 con 0x7fe96815bb50 2024-04-22T01:16:48.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe9677fe640 1 -- 172.21.15.70:0/1410887586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580ab9d0 con 0x7fe96815bb50 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- 172.21.15.70:0/1410887586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe96815bb50 msgr2=0x7fe9680ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/1410887586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe96815bb50 0x7fe9680ae730 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe9580097c0 tx=0x7fe95801c4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- 172.21.15.70:0/1410887586 shutdown_connections 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/1410887586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe96815bb50 0x7fe9680ae730 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/1410887586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe9680b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- 172.21.15.70:0/1410887586 >> 172.21.15.70:0/1410887586 conn(0x7fe96806b1f0 msgr2=0x7fe96806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- 172.21.15.70:0/1410887586 shutdown_connections 2024-04-22T01:16:48.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- 172.21.15.70:0/1410887586 wait complete. 2024-04-22T01:16:48.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 Processor -- start 2024-04-22T01:16:48.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- start start 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 0x7fe968145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe968147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe96815bb50 0x7fe968158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe96815af90 con 0x7fe9680aec70 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe96815ae10 con 0x7fe96815bb50 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96ec4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe96806e950 con 0x7fe96815aa30 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 0x7fe968145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 0x7fe968145d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60712/0 (socket says 172.21.15.70:60712) 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 -- 172.21.15.70:0/954447892 learned_addr learned my addr 172.21.15.70:0/954447892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe967fff640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe968147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 -- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 msgr2=0x7fe968147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe968147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 -- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe96815bb50 msgr2=0x7fe968158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe96815bb50 0x7fe968158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 -- 172.21.15.70:0/954447892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe968146260 con 0x7fe9680aec70 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe96c9e6640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 0x7fe968145d20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe95000d5f0 tx=0x7fe95000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.138+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9500125f0 con 0x7fe9680aec70 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe950019070 con 0x7fe9680aec70 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9500129b0 con 0x7fe9680aec70 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9681466b0 con 0x7fe9680aec70 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe968146c70 con 0x7fe9680aec70 2024-04-22T01:16:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe968068210 con 0x7fe9680aec70 2024-04-22T01:16:48.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe950012b50 con 0x7fe9680aec70 2024-04-22T01:16:48.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe9657fa640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9480678c0 0x7fe948069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:48.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fe9500861d0 con 0x7fe9680aec70 2024-04-22T01:16:48.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe967fff640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9480678c0 0x7fe948069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:48.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.142+0000 7fe967fff640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9480678c0 0x7fe948069d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9580ad000 tx=0x7fe958008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:48.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.146+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe968146260 con 0x7fe9680aec70 2024-04-22T01:16:48.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe96806f190 con 0x7fe9680aec70 2024-04-22T01:16:48.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe9657fa640 1 -- 172.21.15.70:0/954447892 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fe950056c10 con 0x7fe9680aec70 2024-04-22T01:16:48.335 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9480678c0 msgr2=0x7fe948069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9480678c0 0x7fe948069d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9580ad000 tx=0x7fe958008040 comp rx=0 tx=0).stop 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 msgr2=0x7fe968145d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 0x7fe968145d20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe95000d5f0 tx=0x7fe95000dab0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 shutdown_connections 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe9480678c0 0x7fe948069d80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe96815bb50 0x7fe968158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe96815aa30 0x7fe968147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 --2- 172.21.15.70:0/954447892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9680aec70 0x7fe968145d20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:48.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 >> 172.21.15.70:0/954447892 conn(0x7fe96806b1f0 msgr2=0x7fe9680ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:48.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 shutdown_connections 2024-04-22T01:16:48.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48.334+0000 7fe96ec4a640 1 -- 172.21.15.70:0/954447892 wait complete. 2024-04-22T01:16:48.352 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:16:48.352 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:16:48.352 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:16:48.353 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:16:48.353 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:48.353 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:16:48.353 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:16:48.353 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:16:48.353 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:16:48.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:48.353 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:16:48.359 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:16:48.359 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:16:48.359 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:16:48.359 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.359 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:16:48.359 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:48.360 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42413sec preferred_lft 42413sec 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:16:48.361 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:16:48.362 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:48.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:48.363 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:16:48.414 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:16:48.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:48.416 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:16:48.466 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:16:48.466 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:16:48.466 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:48.467 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:16:48.467 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:16:48.467 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:16:48.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:16:48.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:16:48.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:16:48.468 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:16:48.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:16:48.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:48.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:48.587 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:16:48.633 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:16:48.633 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:16:48.633 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:16:48.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:48.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:16:48.686 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:16:48.687 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:48.687 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:48.734 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:16:48.734 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:48.734 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:48.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:16:48.839 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:48.886 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:48.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:16:48.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:16:48.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:16:48.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:48.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:48.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:48.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:48.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:48.890 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:16:48.938 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:16:48.938 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:16:48.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:16:48.995 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:49.042 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:16:49.042 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:16:49.043 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:16:49.044 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:16:49.045 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:49.045 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:49.045 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:49.045 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:49.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:49.045 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:16:49.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:16:49.062+0000 7f96e8d22580 -1 init, newargv = 0x560329e098b0 newargc=15 2024-04-22T01:16:49.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[40147]: starting ceph client 2024-04-22T01:16:49.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[40147]: starting fuse 2024-04-22T01:16:50.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:16:50.187 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:50.234 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:50.235 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:16:50.236 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:16:50.237 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:50.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:50.239 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:16:50.285 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:16:50.285 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:16:50.286 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:16:50.287 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:16:50.401 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.40147.asok 2024-04-22T01:16:50.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:50.411 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.40147.asok 2024-04-22T01:16:50.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:50.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.40147.asok status 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:16:50.564 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "40147", 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 6090, 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 6090 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:16:50.565 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2517891049 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2517891049 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.6090 192.168.0.1:0/2517891049", 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/2517891049", 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 25, 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 52, 2024-04-22T01:16:50.566 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:16:50.567 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:16:50.567 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:16:50.567 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:16:50.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:50.578 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:16:50.578 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:50.584 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:16:50.584 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:50.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:50.584 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:50.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:16:50.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.802+0000 7f7a78cbd640 1 Processor -- start 2024-04-22T01:16:50.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.802+0000 7f7a78cbd640 1 -- start start 2024-04-22T01:16:50.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a78cbd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a7415aa30 0x7f7a740b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:50.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a78cbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a740ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:50.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a78cbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a7406dbd0 con 0x7f7a740aec70 2024-04-22T01:16:50.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a78cbd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7a740af0a0 con 0x7f7a7415aa30 2024-04-22T01:16:50.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a78cbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7406de70 con 0x7f7a7415bb50 2024-04-22T01:16:50.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a72d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7a740aec70 0x7f7a7415e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56984/0 (socket says 172.21.15.70:56984) 2024-04-22T01:16:50.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a72d9d640 1 -- 172.21.15.70:0/4031190351 learned_addr learned my addr 172.21.15.70:0/4031190351 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:50.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a71d9b640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a740ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:50.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7259c640 1 --2- 172.21.15.70:0/4031190351 >> v2:172.21.15.98:3300/0 conn(0x7f7a7415aa30 0x7f7a740b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:50.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7159a640 1 -- 172.21.15.70:0/4031190351 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526863981 0 0) 0x7f7a7406dbd0 con 0x7f7a740aec70 2024-04-22T01:16:50.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7159a640 1 -- 172.21.15.70:0/4031190351 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a5c003530 con 0x7f7a740aec70 2024-04-22T01:16:50.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7259c640 1 -- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 msgr2=0x7f7a740ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:50.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7259c640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a740ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:50.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7259c640 1 -- 172.21.15.70:0/4031190351 >> v1:172.21.15.70:6789/0 conn(0x7f7a740aec70 legacy=0x7f7a7415e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:50.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7159a640 1 -- 172.21.15.70:0/4031190351 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1667240878 0 0) 0x7f7a5c003530 con 0x7f7a740aec70 2024-04-22T01:16:50.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7259c640 1 -- 172.21.15.70:0/4031190351 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a7406e150 con 0x7f7a7415aa30 2024-04-22T01:16:50.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a71d9b640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a740ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:16:50.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7259c640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7415aa30 0x7f7a740b4720 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a640097c0 tx=0x7f7a6401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf326dbf12bf892b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:50.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.806+0000 7f7a7159a640 1 -- 172.21.15.70:0/4031190351 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a640b0070 con 0x7f7a7415aa30 2024-04-22T01:16:50.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a7159a640 1 -- 172.21.15.70:0/4031190351 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a640ab400 con 0x7f7a7415aa30 2024-04-22T01:16:50.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a7159a640 1 -- 172.21.15.70:0/4031190351 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c001a00 con 0x7f7a7415aa30 2024-04-22T01:16:50.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7415aa30 msgr2=0x7f7a740b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:50.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7415aa30 0x7f7a740b4720 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a640097c0 tx=0x7f7a6401c3c0 comp rx=0 tx=0).stop 2024-04-22T01:16:50.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/4031190351 shutdown_connections 2024-04-22T01:16:50.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a740ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:50.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/4031190351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a7415aa30 0x7f7a740b4720 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:50.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/4031190351 >> 172.21.15.70:0/4031190351 conn(0x7f7a7406b1f0 msgr2=0x7f7a7406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:50.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/4031190351 shutdown_connections 2024-04-22T01:16:50.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/4031190351 wait complete. 2024-04-22T01:16:50.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 Processor -- start 2024-04-22T01:16:50.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- start start 2024-04-22T01:16:50.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a740aec70 0x7f7a740ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:50.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 0x7f7a7414fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:50.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a74155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:50.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7415af90 con 0x7f7a7415aa30 2024-04-22T01:16:50.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7415ae10 con 0x7f7a740aec70 2024-04-22T01:16:50.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7406e950 con 0x7f7a7415bb50 2024-04-22T01:16:50.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 0x7f7a7414fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:50.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 0x7f7a7414fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60722/0 (socket says 172.21.15.70:60722) 2024-04-22T01:16:50.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 -- 172.21.15.70:0/3667100709 learned_addr learned my addr 172.21.15.70:0/3667100709 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:50.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a7259c640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a740aec70 0x7f7a740ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:50.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a72d9d640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a74155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:50.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 -- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 msgr2=0x7f7a74155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:50.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a74155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:50.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 -- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a740aec70 msgr2=0x7f7a740ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:50.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a740aec70 0x7f7a740ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:50.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 -- 172.21.15.70:0/3667100709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a740ad9c0 con 0x7f7a7415aa30 2024-04-22T01:16:50.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a72d9d640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a74155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:50.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a71d9b640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 0x7f7a7414fd10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800e980 tx=0x7f7a6800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:50.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68017070 con 0x7f7a7415aa30 2024-04-22T01:16:50.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a6800ca00 con 0x7f7a7415aa30 2024-04-22T01:16:50.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a740ad120 con 0x7f7a7415aa30 2024-04-22T01:16:50.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800cdd0 con 0x7f7a7415aa30 2024-04-22T01:16:50.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.810+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a740ad6e0 con 0x7f7a7415aa30 2024-04-22T01:16:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.814+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a74068210 con 0x7f7a7415aa30 2024-04-22T01:16:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.814+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7a6801a070 con 0x7f7a7415aa30 2024-04-22T01:16:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.814+0000 7f7a62ffd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a540678c0 0x7f7a54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.814+0000 7f7a7259c640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a540678c0 0x7f7a54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.814+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f7a68085d40 con 0x7f7a7415aa30 2024-04-22T01:16:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.814+0000 7f7a7259c640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a540678c0 0x7f7a54069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7a640ad000 tx=0x7f7a640a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:50.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:50.818+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7a68056660 con 0x7f7a7415aa30 2024-04-22T01:16:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.014+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7a7406b820 con 0x7f7a7415aa30 2024-04-22T01:16:51.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.014+0000 7f7a62ffd640 1 -- 172.21.15.70:0/3667100709 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f7a68055f60 con 0x7f7a7415aa30 2024-04-22T01:16:51.016 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a540678c0 msgr2=0x7f7a54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a540678c0 0x7f7a54069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7a640ad000 tx=0x7f7a640a6000 comp rx=0 tx=0).stop 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 msgr2=0x7f7a7414fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 0x7f7a7414fd10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800e980 tx=0x7f7a6800ee40 comp rx=0 tx=0).stop 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 shutdown_connections 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7a540678c0 0x7f7a54069d80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a7415bb50 0x7f7a74155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a7415aa30 0x7f7a7414fd10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 --2- 172.21.15.70:0/3667100709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a740aec70 0x7f7a740ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 >> 172.21.15.70:0/3667100709 conn(0x7f7a7406b1f0 msgr2=0x7f7a74158050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 shutdown_connections 2024-04-22T01:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:51.018+0000 7f7a78cbd640 1 -- 172.21.15.70:0/3667100709 wait complete. 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:16:51.035 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:16:51.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.036 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:16:51.043 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42415sec preferred_lft 42415sec 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.044 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:16:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:16:51.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.046 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:16:51.098 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:16:51.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.099 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:16:51.151 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:16:51.152 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:16:51.153 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:16:51.153 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:16:51.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:16:51.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:16:51.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:16:51.153 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:16:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:16:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:16:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:16:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:51.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.277 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:16:51.282 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:16:51.282 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:16:51.282 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:16:51.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:16:51.339 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:16:51.339 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:51.339 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:51.386 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:16:51.387 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:16:51.387 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:51.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:16:51.494 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:16:51.543 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:16:51.544 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:51.545 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:51.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.546 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:16:51.594 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:16:51.594 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:16:51.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:16:51.653 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:16:51.707 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:16:51.708 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:51.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:51.710 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:16:51.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:16:51.716+0000 7f3e6118d580 -1 init, newargv = 0x56525622c7a0 newargc=15 2024-04-22T01:16:51.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34057]: starting ceph client 2024-04-22T01:16:51.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34057]: starting fuse 2024-04-22T01:16:52.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:16:52.842 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:16:52.895 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:16:52.895 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:16:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:16:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:16:52.898 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:52.898 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:16:52.898 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:52.898 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:16:52.898 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:52.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:52.899 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:16:52.946 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:16:52.947 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:16:52.947 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:16:52.948 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:16:52.949 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:16:52.949 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:16:53.073 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.34057.asok 2024-04-22T01:16:53.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:16:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:16:53.084 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34057.asok 2024-04-22T01:16:53.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34057.asok status 2024-04-22T01:16:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:16:53.239 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:16:53.239 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "34057", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4913, 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:16:53.240 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4913 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2227226636 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2227226636 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4913 192.168.0.1:0/2227226636", 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/2227226636", 2024-04-22T01:16:53.241 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:16:53.242 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 25, 2024-04-22T01:16:53.242 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 52, 2024-04-22T01:16:53.242 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:16:53.242 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:16:53.242 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:16:53.242 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:16:53.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.253 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:16:53.253 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:53.259 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:16:53.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:53.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:53.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.314 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-22T01:16:53.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.331 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10A ./subdir) 2024-04-22T01:16:53.381 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.382 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1y00Ki ./subdir) 2024-04-22T01:16:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.432 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v af00 ./subdir) 2024-04-22T01:16:53.479 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.480 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v G ./subdir) 2024-04-22T01:16:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.529 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v '' ./subdir) 2024-04-22T01:16:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.576 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v ' ' ./subdir) 2024-04-22T01:16:53.623 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.624 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v -1t ./subdir) 2024-04-22T01:16:53.671 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:53.673 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v -1 ./subdir) 2024-04-22T01:16:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:53.725 INFO:teuthology.orchestra.run.smithi070.stderr:setfattr: ./subdir: Invalid argument 2024-04-22T01:16:53.725 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:16:53.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:16:53.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:16:53.871 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:16:53.871 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:53.871 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:16:53.871 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:53.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:54.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47c5c5a640 1 Processor -- start 2024-04-22T01:16:54.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47c5c5a640 1 -- start start 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47c5c5a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47c0072230 0x7f47c00aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47c5c5a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c00cf4a0 con 0x7f47c00aeae0 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47c5c5a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c006f260 con 0x7f47c015a550 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47c5c5a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f47c006f010 con 0x7f47c0072230 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47b6ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f47c00aeae0 0x7f47c0071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56994/0 (socket says 172.21.15.70:56994) 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47bffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f47c0072230 0x7f47c00aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47b6ffd640 1 -- 172.21.15.70:0/14566069 learned_addr learned my addr 172.21.15.70:0/14566069 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47bffff640 1 -- 172.21.15.70:0/14566069 >> v1:172.21.15.98:6789/0 conn(0x7f47c015a550 legacy=0x7f47c00cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47bffff640 1 -- 172.21.15.70:0/14566069 >> v1:172.21.15.70:6789/0 conn(0x7f47c00aeae0 legacy=0x7f47c0071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47bffff640 1 -- 172.21.15.70:0/14566069 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c006f540 con 0x7f47c0072230 2024-04-22T01:16:54.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.050+0000 7f47beffd640 1 -- 172.21.15.70:0/14566069 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408239275 0 0) 0x7f47c00cf4a0 con 0x7f47c00aeae0 2024-04-22T01:16:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47bffff640 1 --2- 172.21.15.70:0/14566069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c0072230 0x7f47c00aef50 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009d90 tx=0x7f47b801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d62c65aaa37a0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47beffd640 1 -- 172.21.15.70:0/14566069 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b80b2070 con 0x7f47c0072230 2024-04-22T01:16:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47beffd640 1 -- 172.21.15.70:0/14566069 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b80a5800 con 0x7f47c0072230 2024-04-22T01:16:54.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47beffd640 1 -- 172.21.15.70:0/14566069 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b80a5bd0 con 0x7f47c0072230 2024-04-22T01:16:54.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- 172.21.15.70:0/14566069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c0072230 msgr2=0x7f47c00aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/14566069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c0072230 0x7f47c00aef50 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009d90 tx=0x7f47b801c970 comp rx=0 tx=0).stop 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- 172.21.15.70:0/14566069 shutdown_connections 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/14566069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c0072230 0x7f47c00aef50 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- 172.21.15.70:0/14566069 >> 172.21.15.70:0/14566069 conn(0x7f47c006b1f0 msgr2=0x7f47c006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- 172.21.15.70:0/14566069 shutdown_connections 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- 172.21.15.70:0/14566069 wait complete. 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 Processor -- start 2024-04-22T01:16:54.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- start start 2024-04-22T01:16:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47c0072230 0x7f47c015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 0x7f47c0150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c015a550 0x7f47c00afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00b3150 con 0x7f47c00aeae0 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00b2fd0 con 0x7f47c015a550 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47c5c5a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f47c00b32d0 con 0x7f47c0072230 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 0x7f47c0150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 0x7f47c0150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60730/0 (socket says 172.21.15.70:60730) 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 -- 172.21.15.70:0/3085914456 learned_addr learned my addr 172.21.15.70:0/3085914456 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47bf7fe640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47c0072230 0x7f47c015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 -- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47c0072230 msgr2=0x7f47c015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47c0072230 0x7f47c015d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 -- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c015a550 msgr2=0x7f47c00afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47bffff640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c015a550 0x7f47c00afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c015a550 0x7f47c00afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 -- 172.21.15.70:0/3085914456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c015d6e0 con 0x7f47c00aeae0 2024-04-22T01:16:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.054+0000 7f47b6ffd640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 0x7f47c0150220 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f47a800c480 tx=0x7f47a800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8005cf0 con 0x7f47c00aeae0 2024-04-22T01:16:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a8007070 con 0x7f47c00aeae0 2024-04-22T01:16:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c0158460 con 0x7f47c00aeae0 2024-04-22T01:16:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a80074e0 con 0x7f47c00aeae0 2024-04-22T01:16:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c0158a20 con 0x7f47c00aeae0 2024-04-22T01:16:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c0068210 con 0x7f47c00aeae0 2024-04-22T01:16:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f47a8025430 con 0x7f47c00aeae0 2024-04-22T01:16:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bcff9640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47a00678c0 0x7f47a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f47a8086600 con 0x7f47c00aeae0 2024-04-22T01:16:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bf7fe640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47a00678c0 0x7f47a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.058+0000 7f47bf7fe640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47a00678c0 0x7f47a0069d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f47b0000f70 tx=0x7f47b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.062+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f47a8057020 con 0x7f47c00aeae0 2024-04-22T01:16:54.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.222+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47c0159d80 con 0x7f47c00aeae0 2024-04-22T01:16:54.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47bcff9640 1 -- 172.21.15.70:0/3085914456 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f47a8056920 con 0x7f47c00aeae0 2024-04-22T01:16:54.226 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:54.226 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:43.266695+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:54.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47a00678c0 msgr2=0x7f47a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47a00678c0 0x7f47a0069d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f47b0000f70 tx=0x7f47b0009040 comp rx=0 tx=0).stop 2024-04-22T01:16:54.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 msgr2=0x7f47c0150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 0x7f47c0150220 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f47a800c480 tx=0x7f47a800c940 comp rx=0 tx=0).stop 2024-04-22T01:16:54.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 shutdown_connections 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f47a00678c0 0x7f47a0069d80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47c015a550 0x7f47c00afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c00aeae0 0x7f47c0150220 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 --2- 172.21.15.70:0/3085914456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47c0072230 0x7f47c015d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 >> 172.21.15.70:0/3085914456 conn(0x7f47c006b1f0 msgr2=0x7f47c00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 shutdown_connections 2024-04-22T01:16:54.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.226+0000 7f47c5c5a640 1 -- 172.21.15.70:0/3085914456 wait complete. 2024-04-22T01:16:54.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:16:39.917376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:16:40.609582+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:54.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:54.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.366+0000 7fe39e956640 1 Processor -- start 2024-04-22T01:16:54.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.366+0000 7fe39e956640 1 -- start start 2024-04-22T01:16:54.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.366+0000 7fe39e956640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe39815f3e0 0x7fe3980b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe39e956640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 0x7fe39815fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe39e956640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe398160840 con 0x7fe398160010 2024-04-22T01:16:54.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe39e956640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe3980b2450 con 0x7fe39815f3e0 2024-04-22T01:16:54.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe39e956640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3980cc990 con 0x7fe3980b1ff0 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 0x7fe39815fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe397fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe39815f3e0 0x7fe3980b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 0x7fe39815fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:59358/0 (socket says 172.21.15.70:59358) 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 -- 172.21.15.70:0/4177673441 learned_addr learned my addr 172.21.15.70:0/4177673441 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 -- 172.21.15.70:0/4177673441 >> v2:172.21.15.98:3300/0 conn(0x7fe39815f3e0 msgr2=0x7fe3980b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 --2- 172.21.15.70:0/4177673441 >> v2:172.21.15.98:3300/0 conn(0x7fe39815f3e0 0x7fe3980b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 -- 172.21.15.70:0/4177673441 >> v1:172.21.15.98:6789/0 conn(0x7fe398160010 legacy=0x7fe3980d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 -- 172.21.15.70:0/4177673441 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3980ccc70 con 0x7fe3980b1ff0 2024-04-22T01:16:54.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe38f7fe640 1 --2- 172.21.15.70:0/4177673441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 0x7fe39815fad0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe380009870 tx=0x7fe38001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae07557a0ef5741 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe3977fe640 1 -- 172.21.15.70:0/4177673441 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3800b0070 con 0x7fe3980b1ff0 2024-04-22T01:16:54.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe3977fe640 1 -- 172.21.15.70:0/4177673441 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3800ab600 con 0x7fe3980b1ff0 2024-04-22T01:16:54.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.370+0000 7fe3977fe640 1 -- 172.21.15.70:0/4177673441 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3800ab9d0 con 0x7fe3980b1ff0 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/4177673441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 msgr2=0x7fe39815fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 --2- 172.21.15.70:0/4177673441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 0x7fe39815fad0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe380009870 tx=0x7fe38001c560 comp rx=0 tx=0).stop 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/4177673441 shutdown_connections 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 --2- 172.21.15.70:0/4177673441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3980b1ff0 0x7fe39815fad0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 --2- 172.21.15.70:0/4177673441 >> v2:172.21.15.98:3300/0 conn(0x7fe39815f3e0 0x7fe3980b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/4177673441 >> 172.21.15.70:0/4177673441 conn(0x7fe398099a60 msgr2=0x7fe398099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/4177673441 shutdown_connections 2024-04-22T01:16:54.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/4177673441 wait complete. 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 Processor -- start 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- start start 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3980b1ff0 0x7fe3980c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe39815f3e0 0x7fe398154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 0x7fe3980ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3980cd8d0 con 0x7fe398160010 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3980cd750 con 0x7fe3980b1ff0 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe398168cc0 con 0x7fe39815f3e0 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 0x7fe3980ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 0x7fe3980ae8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60734/0 (socket says 172.21.15.70:60734) 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 -- 172.21.15.70:0/2949481198 learned_addr learned my addr 172.21.15.70:0/2949481198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:54.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe38f7fe640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe39815f3e0 0x7fe398154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe397fff640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3980b1ff0 0x7fe3980c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 -- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe39815f3e0 msgr2=0x7fe398154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe39815f3e0 0x7fe398154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 -- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3980b1ff0 msgr2=0x7fe3980c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3980b1ff0 0x7fe3980c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 -- 172.21.15.70:0/2949481198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3980ce6b0 con 0x7fe398160010 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe38f7fe640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe39815f3e0 0x7fe398154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39cef3640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 0x7fe3980ae8d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe39000c770 tx=0x7fe39000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe390003b80 con 0x7fe398160010 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe390003d20 con 0x7fe398160010 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39000aa90 con 0x7fe398160010 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3980ca2d0 con 0x7fe398160010 2024-04-22T01:16:54.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.374+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3980ca890 con 0x7fe398160010 2024-04-22T01:16:54.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.378+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe398068210 con 0x7fe398160010 2024-04-22T01:16:54.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.378+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe39000ac30 con 0x7fe398160010 2024-04-22T01:16:54.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.378+0000 7fe3957fa640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3780678f0 0x7fe378069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.378+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fe390085cc0 con 0x7fe398160010 2024-04-22T01:16:54.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.378+0000 7fe397fff640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3780678f0 0x7fe378069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.378+0000 7fe397fff640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3780678f0 0x7fe378069db0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe388009a50 tx=0x7fe388008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.382+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe390056700 con 0x7fe398160010 2024-04-22T01:16:54.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.542+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe3980b1930 con 0x7fe398160010 2024-04-22T01:16:54.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.542+0000 7fe3957fa640 1 -- 172.21.15.70:0/2949481198 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7fe390056000 con 0x7fe398160010 2024-04-22T01:16:54.544 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3780678f0 msgr2=0x7fe378069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3780678f0 0x7fe378069db0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe388009a50 tx=0x7fe388008040 comp rx=0 tx=0).stop 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 msgr2=0x7fe3980ae8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 0x7fe3980ae8d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe39000c770 tx=0x7fe39000cc30 comp rx=0 tx=0).stop 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 shutdown_connections 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3780678f0 0x7fe378069db0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe398160010 0x7fe3980ae8d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe39815f3e0 0x7fe398154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 --2- 172.21.15.70:0/2949481198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3980b1ff0 0x7fe3980c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 >> 172.21.15.70:0/2949481198 conn(0x7fe398099a60 msgr2=0x7fe3980b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 shutdown_connections 2024-04-22T01:16:54.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.546+0000 7fe39e956640 1 -- 172.21.15.70:0/2949481198 wait complete. 2024-04-22T01:16:54.563 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:54.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 Processor -- start 2024-04-22T01:16:54.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 -- start start 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fca9415e6e0 0x7fca9415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca94155370 0x7fca9406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9415f000 con 0x7fca940ce920 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fca940b3800 con 0x7fca9415e6e0 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca98df1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fca940b3ae0 con 0x7fca94155370 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fca9415e6e0 0x7fca9415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fca9415e6e0 0x7fca9415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60736/0 (socket says 172.21.15.70:60736) 2024-04-22T01:16:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 -- 172.21.15.70:0/4096834072 learned_addr learned my addr 172.21.15.70:0/4096834072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca92d9d640 1 --2- 172.21.15.70:0/4096834072 >> v2:172.21.15.98:3300/0 conn(0x7fca94155370 0x7fca9406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 -- 172.21.15.70:0/4096834072 >> v2:172.21.15.98:3300/0 conn(0x7fca94155370 msgr2=0x7fca9406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 --2- 172.21.15.70:0/4096834072 >> v2:172.21.15.98:3300/0 conn(0x7fca94155370 0x7fca9406d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 -- 172.21.15.70:0/4096834072 >> v1:172.21.15.98:6789/0 conn(0x7fca940ce920 legacy=0x7fca94154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.690+0000 7fca9259c640 1 -- 172.21.15.70:0/4096834072 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca940cde40 con 0x7fca9415e6e0 2024-04-22T01:16:54.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca9259c640 1 --2- 172.21.15.70:0/4096834072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca9415e6e0 0x7fca9415eac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fca94158ce0 tx=0x7fca8c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d9743555550e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca9159a640 1 -- 172.21.15.70:0/4096834072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0aa610 con 0x7fca9415e6e0 2024-04-22T01:16:54.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca9159a640 1 -- 172.21.15.70:0/4096834072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8c0af070 con 0x7fca9415e6e0 2024-04-22T01:16:54.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca9159a640 1 -- 172.21.15.70:0/4096834072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0aa9d0 con 0x7fca9415e6e0 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 -- 172.21.15.70:0/4096834072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca9415e6e0 msgr2=0x7fca9415eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 --2- 172.21.15.70:0/4096834072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca9415e6e0 0x7fca9415eac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fca94158ce0 tx=0x7fca8c01c4b0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 -- 172.21.15.70:0/4096834072 shutdown_connections 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 --2- 172.21.15.70:0/4096834072 >> v2:172.21.15.98:3300/0 conn(0x7fca94155370 0x7fca9406d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 --2- 172.21.15.70:0/4096834072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca9415e6e0 0x7fca9415eac0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 -- 172.21.15.70:0/4096834072 >> 172.21.15.70:0/4096834072 conn(0x7fca940b1d40 msgr2=0x7fca940cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 -- 172.21.15.70:0/4096834072 shutdown_connections 2024-04-22T01:16:54.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.694+0000 7fca98df1640 1 -- 172.21.15.70:0/4096834072 wait complete. 2024-04-22T01:16:54.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 Processor -- start 2024-04-22T01:16:54.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- start start 2024-04-22T01:16:54.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 0x7fca94156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca94155370 0x7fca941549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca9415e6e0 0x7fca94150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca9406f260 con 0x7fca940ce920 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca9406f0e0 con 0x7fca94155370 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fca940b32c0 con 0x7fca9415e6e0 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 0x7fca94156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 0x7fca94156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60738/0 (socket says 172.21.15.70:60738) 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 -- 172.21.15.70:0/1377592323 learned_addr learned my addr 172.21.15.70:0/1377592323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca91d9b640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca94155370 0x7fca941549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca92d9d640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca9415e6e0 0x7fca94150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 -- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca9415e6e0 msgr2=0x7fca94150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca9415e6e0 0x7fca94150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 -- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca94155370 msgr2=0x7fca941549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca94155370 0x7fca941549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 -- 172.21.15.70:0/1377592323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca941572e0 con 0x7fca940ce920 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca91d9b640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca94155370 0x7fca941549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca92d9d640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca9415e6e0 0x7fca94150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca9259c640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 0x7fca94156a90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0ac040 tx=0x7fca8c01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0bd5f0 con 0x7fca940ce920 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8c0bd7b0 con 0x7fca940ce920 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca941575c0 con 0x7fca940ce920 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0bdba0 con 0x7fca940ce920 2024-04-22T01:16:54.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca941f55f0 con 0x7fca940ce920 2024-04-22T01:16:54.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca94068210 con 0x7fca940ce920 2024-04-22T01:16:54.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fca8c0b3600 con 0x7fca940ce920 2024-04-22T01:16:54.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca82ffd640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca740678c0 0x7fca74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fca8c120d70 con 0x7fca940ce920 2024-04-22T01:16:54.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.698+0000 7fca91d9b640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca740678c0 0x7fca74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.702+0000 7fca91d9b640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca740678c0 0x7fca74069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca88007fd0 tx=0x7fca88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:54.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.706+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fca8c0c3060 con 0x7fca940ce920 2024-04-22T01:16:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.854+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fca940d08e0 con 0x7fca940ce920 2024-04-22T01:16:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.854+0000 7fca82ffd640 1 -- 172.21.15.70:0/1377592323 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) ==== 104+0+11 (secure 0 0 0) 0x7fca8c0a7070 con 0x7fca940ce920 2024-04-22T01:16:54.856 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca740678c0 msgr2=0x7fca74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca740678c0 0x7fca74069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca88007fd0 tx=0x7fca88009040 comp rx=0 tx=0).stop 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 msgr2=0x7fca94156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 0x7fca94156a90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0ac040 tx=0x7fca8c01acf0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 shutdown_connections 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fca740678c0 0x7fca74069d80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca9415e6e0 0x7fca94150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca94155370 0x7fca941549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 --2- 172.21.15.70:0/1377592323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca940ce920 0x7fca94156a90 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 >> 172.21.15.70:0/1377592323 conn(0x7fca940b1d40 msgr2=0x7fca9415c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 shutdown_connections 2024-04-22T01:16:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.858+0000 7fca98df1640 1 -- 172.21.15.70:0/1377592323 wait complete. 2024-04-22T01:16:54.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:54.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.990+0000 7f735dbe2640 1 Processor -- start 2024-04-22T01:16:54.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f735dbe2640 1 -- start start 2024-04-22T01:16:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f735dbe2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f735815be30 0x7f73580d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f735dbe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735806b900 0x7f73580af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f735dbe2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73580b5420 con 0x7f73580cc2e0 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f735dbe2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f73580b6250 con 0x7f735815be30 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f735dbe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f73580b5d20 con 0x7f735806b900 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f734ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f73580cc2e0 0x7f735806b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57010/0 (socket says 172.21.15.70:57010) 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f734ffff640 1 -- 172.21.15.70:0/2049070468 learned_addr learned my addr 172.21.15.70:0/2049070468 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735806b900 0x7f73580af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f73577fe640 1 --2- 172.21.15.70:0/2049070468 >> v2:172.21.15.98:3300/0 conn(0x7f735815be30 0x7f73580d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2115975791 0 0) 0x7f73580b5420 con 0x7f73580cc2e0 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7344003530 con 0x7f73580cc2e0 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2995560860 0 0) 0x7f7344003530 con 0x7f73580cc2e0 2024-04-22T01:16:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735806b900 msgr2=0x7f73580af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735806b900 0x7f73580af1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 >> v2:172.21.15.98:3300/0 conn(0x7f735815be30 msgr2=0x7f73580d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 --2- 172.21.15.70:0/2049070468 >> v2:172.21.15.98:3300/0 conn(0x7f735815be30 0x7f73580d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73580b5f70 con 0x7f73580cc2e0 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7340001680 con 0x7f73580cc2e0 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 >> v1:172.21.15.70:6789/0 conn(0x7f73580cc2e0 legacy=0x7f735806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73440039c0 0x7f7344003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7344004810 0x7f7344004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7344005190 0x7f734400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73580b5d20 con 0x7f7344005190 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f734400bf10 con 0x7f73440039c0 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f734400c090 con 0x7f7344004810 2024-04-22T01:16:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7344005190 0x7f734400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f73577fe640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73440039c0 0x7f7344003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f734ffff640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7344004810 0x7f7344004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 -- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7344004810 msgr2=0x7f7344004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7344004810 0x7f7344004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 -- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73440039c0 msgr2=0x7f7344003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73440039c0 0x7f7344003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:54.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.994+0000 7f7357fff640 1 -- 172.21.15.70:0/2049070468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7344003530 con 0x7f7344005190 2024-04-22T01:16:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7344005190 0x7f734400b9d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7350009fc0 tx=0x7f735001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4fc9af01458d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73500b2070 con 0x7f7344005190 2024-04-22T01:16:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73500a5c90 con 0x7f7344005190 2024-04-22T01:16:55.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7356ffd640 1 -- 172.21.15.70:0/2049070468 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350009c10 con 0x7f7344005190 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- 172.21.15.70:0/2049070468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7344005190 msgr2=0x7f734400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 --2- 172.21.15.70:0/2049070468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7344005190 0x7f734400b9d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7350009fc0 tx=0x7f735001cc80 comp rx=0 tx=0).stop 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 -- 172.21.15.70:0/2049070468 reap_dead start 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- 172.21.15.70:0/2049070468 shutdown_connections 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- 172.21.15.70:0/2049070468 >> 172.21.15.70:0/2049070468 conn(0x7f7358099a60 msgr2=0x7f73580b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- 172.21.15.70:0/2049070468 shutdown_connections 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- 172.21.15.70:0/2049070468 wait complete. 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 Processor -- start 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- start start 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735809d5b0 0x7f735809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735809d990 0x7f73580ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 0x7f73581f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7358067630 con 0x7f73581f4140 2024-04-22T01:16:55.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73580cfe70 con 0x7f735809d990 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f735dbe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f735806c950 con 0x7f735809d5b0 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 0x7f73581f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 0x7f73581f45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60762/0 (socket says 172.21.15.70:60762) 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 -- 172.21.15.70:0/1898870045 learned_addr learned my addr 172.21.15.70:0/1898870045 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f73577fe640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735809d5b0 0x7f735809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f734ffff640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735809d990 0x7f73580ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 -- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735809d5b0 msgr2=0x7f735809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735809d5b0 0x7f735809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 -- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735809d990 msgr2=0x7f73580ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735809d990 0x7f73580ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:54.998+0000 7f7357fff640 1 -- 172.21.15.70:0/1898870045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73580cf1e0 con 0x7f73581f4140 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7357fff640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 0x7f73581f45a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7350002240 tx=0x7f7350009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73500bd070 con 0x7f73581f4140 2024-04-22T01:16:55.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73500a5a90 con 0x7f73581f4140 2024-04-22T01:16:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73581f4ae0 con 0x7f73581f4140 2024-04-22T01:16:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73500b2040 con 0x7f73581f4140 2024-04-22T01:16:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f735809bd30 con 0x7f73581f4140 2024-04-22T01:16:55.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735809c620 con 0x7f73581f4140 2024-04-22T01:16:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f73500039a0 con 0x7f73581f4140 2024-04-22T01:16:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7354ff9640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f73380678c0 0x7f7338069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f735011fb40 con 0x7f73581f4140 2024-04-22T01:16:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f73577fe640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f73380678c0 0x7f7338069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.002+0000 7f73577fe640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f73380678c0 0x7f7338069d80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7348009ab0 tx=0x7f7348008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.006+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7350016990 con 0x7f73581f4140 2024-04-22T01:16:55.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.162+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7358068920 con 0x7f73581f4140 2024-04-22T01:16:55.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f7354ff9640 1 -- 172.21.15.70:0/1898870045 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) ==== 100+0+11 (secure 0 0 0) 0x7f73500efeb0 con 0x7f73581f4140 2024-04-22T01:16:55.166 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f73380678c0 msgr2=0x7f7338069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f73380678c0 0x7f7338069d80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7348009ab0 tx=0x7f7348008040 comp rx=0 tx=0).stop 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 msgr2=0x7f73581f45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 0x7f73581f45a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7350002240 tx=0x7f7350009750 comp rx=0 tx=0).stop 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 shutdown_connections 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f73380678c0 0x7f7338069d80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73581f4140 0x7f73581f45a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735809d990 0x7f73580ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 --2- 172.21.15.70:0/1898870045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f735809d5b0 0x7f735809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 >> 172.21.15.70:0/1898870045 conn(0x7f7358099a60 msgr2=0x7f73581564e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:55.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 shutdown_connections 2024-04-22T01:16:55.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.166+0000 7f735dbe2640 1 -- 172.21.15.70:0/1898870045 wait complete. 2024-04-22T01:16:55.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:16:55.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 Processor -- start 2024-04-22T01:16:55.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 -- start start 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc58015bf30 0x7fc58006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc5801605e0 0x7fc58006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5800cd7a0 con 0x7fc58015bb50 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc5800cd580 con 0x7fc5801605e0 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc586d1b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc58015f8b0 con 0x7fc58015bf30 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc584ab7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc58015bb50 0x7fc5800cce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57020/0 (socket says 172.21.15.70:57020) 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc584ab7640 1 -- 172.21.15.70:0/1797431960 learned_addr learned my addr 172.21.15.70:0/1797431960 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5852b8640 1 --2- 172.21.15.70:0/1797431960 >> v2:172.21.15.70:3300/0 conn(0x7fc5801605e0 0x7fc58006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5777fe640 1 -- 172.21.15.70:0/1797431960 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100581450 0 0) 0x7fc5800cd7a0 con 0x7fc58015bb50 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5777fe640 1 -- 172.21.15.70:0/1797431960 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc568003530 con 0x7fc58015bb50 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5852b8640 1 -- 172.21.15.70:0/1797431960 >> v2:172.21.15.98:3300/0 conn(0x7fc58015bf30 msgr2=0x7fc58006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:55.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc577fff640 1 --2- 172.21.15.70:0/1797431960 >> v2:172.21.15.98:3300/0 conn(0x7fc58015bf30 0x7fc58006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5852b8640 1 --2- 172.21.15.70:0/1797431960 >> v2:172.21.15.98:3300/0 conn(0x7fc58015bf30 0x7fc58006ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5852b8640 1 -- 172.21.15.70:0/1797431960 >> v1:172.21.15.70:6789/0 conn(0x7fc58015bb50 legacy=0x7fc5800cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.294+0000 7fc5852b8640 1 -- 172.21.15.70:0/1797431960 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58015fb00 con 0x7fc5801605e0 2024-04-22T01:16:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc5852b8640 1 --2- 172.21.15.70:0/1797431960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5801605e0 0x7fc58006f2b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc578009d90 tx=0x7fc57801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38075f13f7c9748d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc5777fe640 1 -- 172.21.15.70:0/1797431960 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5780b2070 con 0x7fc5801605e0 2024-04-22T01:16:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc5777fe640 1 -- 172.21.15.70:0/1797431960 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5780a5870 con 0x7fc5801605e0 2024-04-22T01:16:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc5777fe640 1 -- 172.21.15.70:0/1797431960 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5780a5c20 con 0x7fc5801605e0 2024-04-22T01:16:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 -- 172.21.15.70:0/1797431960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5801605e0 msgr2=0x7fc58006f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1797431960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5801605e0 0x7fc58006f2b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc578009d90 tx=0x7fc57801c8a0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 -- 172.21.15.70:0/1797431960 shutdown_connections 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1797431960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5801605e0 0x7fc58006f2b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1797431960 >> v2:172.21.15.98:3300/0 conn(0x7fc58015bf30 0x7fc58006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 -- 172.21.15.70:0/1797431960 >> 172.21.15.70:0/1797431960 conn(0x7fc5800cf9e0 msgr2=0x7fc5800cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 -- 172.21.15.70:0/1797431960 shutdown_connections 2024-04-22T01:16:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 -- 172.21.15.70:0/1797431960 wait complete. 2024-04-22T01:16:55.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 Processor -- start 2024-04-22T01:16:55.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.298+0000 7fc586d1b640 1 -- start start 2024-04-22T01:16:55.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc58015bb50 0x7fc5801663f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 0x7fc58014e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5801605e0 0x7fc5800b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc580072690 con 0x7fc58015bf30 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc580072510 con 0x7fc5801605e0 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc580072810 con 0x7fc58015bb50 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 0x7fc58014e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 0x7fc58014e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60786/0 (socket says 172.21.15.70:60786) 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 -- 172.21.15.70:0/1558682595 learned_addr learned my addr 172.21.15.70:0/1558682595 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 -- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc58015bb50 msgr2=0x7fc5801663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc584ab7640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc58015bb50 0x7fc5801663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5852b8640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5801605e0 0x7fc5800b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc58015bb50 0x7fc5801663f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 -- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5801605e0 msgr2=0x7fc5800b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5801605e0 0x7fc5800b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 -- 172.21.15.70:0/1558682595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc580165350 con 0x7fc58015bf30 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc577fff640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 0x7fc58014e030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c7f0 tx=0x7fc57000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570015070 con 0x7fc58015bf30 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc570004240 con 0x7fc58015bf30 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570003620 con 0x7fc58015bf30 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc580165630 con 0x7fc58015bf30 2024-04-22T01:16:55.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc580165bf0 con 0x7fc58015bf30 2024-04-22T01:16:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58004f510 con 0x7fc58015bf30 2024-04-22T01:16:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5700039d0 con 0x7fc58015bf30 2024-04-22T01:16:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5757fa640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc56c0678c0 0x7fc56c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fc570085f80 con 0x7fc58015bf30 2024-04-22T01:16:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc584ab7640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc56c0678c0 0x7fc56c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.302+0000 7fc584ab7640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc56c0678c0 0x7fc56c069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.306+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc5700568a0 con 0x7fc58015bf30 2024-04-22T01:16:55.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.478+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc5800d0740 con 0x7fc58015bf30 2024-04-22T01:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.478+0000 7fc5757fa640 1 -- 172.21.15.70:0/1558682595 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v52) ==== 80+0+0 (secure 0 0 0) 0x7fc5700561a0 con 0x7fc58015bf30 2024-04-22T01:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:16:55.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc56c0678c0 msgr2=0x7fc56c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc56c0678c0 0x7fc56c069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2024-04-22T01:16:55.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 msgr2=0x7fc58014e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 0x7fc58014e030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c7f0 tx=0x7fc57000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 shutdown_connections 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc56c0678c0 0x7fc56c069d80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5801605e0 0x7fc5800b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc58015bf30 0x7fc58014e030 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 --2- 172.21.15.70:0/1558682595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc58015bb50 0x7fc5801663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 >> 172.21.15.70:0/1558682595 conn(0x7fc5800cf9e0 msgr2=0x7fc58015a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 shutdown_connections 2024-04-22T01:16:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.482+0000 7fc586d1b640 1 -- 172.21.15.70:0/1558682595 wait complete. 2024-04-22T01:16:55.500 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:16:55.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:55.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:55.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[40147]: fuse finished with error 0 and tester_r 0 2024-04-22T01:16:55.513 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:16:55.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:55.513 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:16:55.665 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:55.665 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:16:55.666 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:55.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:55.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 Processor -- start 2024-04-22T01:16:55.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 -- start start 2024-04-22T01:16:55.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15aa30 0x7f241c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f241c15bb50 0x7f241c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f241c0aec70 0x7f241c15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f241c0af0a0 con 0x7f241c15bb50 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f241c06dbd0 con 0x7f241c0aec70 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f2421e4d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f241c06de70 con 0x7f241c15aa30 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f241c15bb50 0x7f241c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f241c15bb50 0x7f241c0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60802/0 (socket says 172.21.15.70:60802) 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 -- 172.21.15.70:0/340493277 learned_addr learned my addr 172.21.15.70:0/340493277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241b7fe640 1 --2- 172.21.15.70:0/340493277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15aa30 0x7f241c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241bfff640 1 --2- 172.21.15.70:0/340493277 >> v2:172.21.15.98:3300/0 conn(0x7f241c0aec70 0x7f241c15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 -- 172.21.15.70:0/340493277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15aa30 msgr2=0x7f241c0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 --2- 172.21.15.70:0/340493277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15aa30 0x7f241c0b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 -- 172.21.15.70:0/340493277 >> v2:172.21.15.98:3300/0 conn(0x7f241c0aec70 msgr2=0x7f241c15e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 --2- 172.21.15.70:0/340493277 >> v2:172.21.15.98:3300/0 conn(0x7f241c0aec70 0x7f241c15e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.834+0000 7f241affd640 1 -- 172.21.15.70:0/340493277 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c06e150 con 0x7f241c15bb50 2024-04-22T01:16:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f241affd640 1 --2- 172.21.15.70:0/340493277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c15bb50 0x7f241c0ae730 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f241c0d05d0 tx=0x7f241001c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5794600abe0ac49d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f241a7fc640 1 -- 172.21.15.70:0/340493277 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24100b0070 con 0x7f241c15bb50 2024-04-22T01:16:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f241a7fc640 1 -- 172.21.15.70:0/340493277 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24100ab600 con 0x7f241c15bb50 2024-04-22T01:16:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f241a7fc640 1 -- 172.21.15.70:0/340493277 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24100ab9d0 con 0x7f241c15bb50 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 -- 172.21.15.70:0/340493277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c15bb50 msgr2=0x7f241c0ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 --2- 172.21.15.70:0/340493277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c15bb50 0x7f241c0ae730 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f241c0d05d0 tx=0x7f241001c230 comp rx=0 tx=0).stop 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 -- 172.21.15.70:0/340493277 shutdown_connections 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 --2- 172.21.15.70:0/340493277 >> v2:172.21.15.98:3300/0 conn(0x7f241c0aec70 0x7f241c15e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 --2- 172.21.15.70:0/340493277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c15bb50 0x7f241c0ae730 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 --2- 172.21.15.70:0/340493277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15aa30 0x7f241c0b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 -- 172.21.15.70:0/340493277 >> 172.21.15.70:0/340493277 conn(0x7f241c06b1f0 msgr2=0x7f241c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 -- 172.21.15.70:0/340493277 shutdown_connections 2024-04-22T01:16:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 -- 172.21.15.70:0/340493277 wait complete. 2024-04-22T01:16:55.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 Processor -- start 2024-04-22T01:16:55.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.838+0000 7f2421e4d640 1 -- start start 2024-04-22T01:16:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 0x7f241c150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c15aa30 0x7f241c14f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15bb50 0x7f241c1470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f241c15af90 con 0x7f241c0aec70 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f241c15ae10 con 0x7f241c15aa30 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f241c06e950 con 0x7f241c15bb50 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 0x7f241c150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 0x7f241c150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60806/0 (socket says 172.21.15.70:60806) 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 -- 172.21.15.70:0/1587410829 learned_addr learned my addr 172.21.15.70:0/1587410829 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241affd640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c15aa30 0x7f241c14f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241bfff640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15bb50 0x7f241c1470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 -- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15bb50 msgr2=0x7f241c1470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15bb50 0x7f241c1470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 -- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c15aa30 msgr2=0x7f241c14f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c15aa30 0x7f241c14f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 -- 172.21.15.70:0/1587410829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c157550 con 0x7f241c0aec70 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241b7fe640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 0x7f241c150000 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f240c00d5f0 tx=0x7f240c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c0125f0 con 0x7f241c0aec70 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f240c019070 con 0x7f241c0aec70 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241c157830 con 0x7f241c0aec70 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c0129b0 con 0x7f241c0aec70 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f241c157df0 con 0x7f241c0aec70 2024-04-22T01:16:55.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f241c068210 con 0x7f241c0aec70 2024-04-22T01:16:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f240c012b50 con 0x7f241c0aec70 2024-04-22T01:16:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2403fff640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24080678c0 0x7f2408069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f240c0861d0 con 0x7f241c0aec70 2024-04-22T01:16:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.842+0000 7f241affd640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24080678c0 0x7f2408069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.846+0000 7f241affd640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24080678c0 0x7f2408069d80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f24100ad000 tx=0x7f2410008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.850+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f240c056c10 con 0x7f241c0aec70 2024-04-22T01:16:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:55.998+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f241c06f190 con 0x7f241c0aec70 2024-04-22T01:16:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.002+0000 7f2403fff640 1 -- 172.21.15.70:0/1587410829 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f240c056510 con 0x7f241c0aec70 2024-04-22T01:16:56.004 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:56.004 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:43.266695+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24080678c0 msgr2=0x7f2408069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24080678c0 0x7f2408069d80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f24100ad000 tx=0x7f2410008040 comp rx=0 tx=0).stop 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 msgr2=0x7f241c150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 0x7f241c150000 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f240c00d5f0 tx=0x7f240c00dab0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 shutdown_connections 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f24080678c0 0x7f2408069d80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f241c15bb50 0x7f241c1470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f241c15aa30 0x7f241c14f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 --2- 172.21.15.70:0/1587410829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f241c0aec70 0x7f241c150000 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 >> 172.21.15.70:0/1587410829 conn(0x7f241c06b1f0 msgr2=0x7f241c0b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 shutdown_connections 2024-04-22T01:16:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.006+0000 7f2421e4d640 1 -- 172.21.15.70:0/1587410829 wait complete. 2024-04-22T01:16:56.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:16:39.917376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:16:40.609582+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:56.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:56.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3c05651640 1 Processor -- start 2024-04-22T01:16:56.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3c05651640 1 -- start start 2024-04-22T01:16:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3c05651640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3c00072280 0x7f3c0016d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3c05651640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c0006e340 con 0x7f3c00162350 2024-04-22T01:16:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3c05651640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c0006e620 con 0x7f3c0006f3e0 2024-04-22T01:16:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3c05651640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3c0006e900 con 0x7f3c00072280 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bfeffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3c00162350 0x7f3c0006d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34022/0 (socket says 172.21.15.70:34022) 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bfeffd640 1 -- 172.21.15.70:0/3051998586 learned_addr learned my addr 172.21.15.70:0/3051998586 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bff7fe640 1 --2- 172.21.15.70:0/3051998586 >> v2:172.21.15.70:3300/0 conn(0x7f3c00072280 0x7f3c0016d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bff7fe640 1 -- 172.21.15.70:0/3051998586 >> v1:172.21.15.98:6789/0 conn(0x7f3c0006f3e0 legacy=0x7f3c00071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bff7fe640 1 -- 172.21.15.70:0/3051998586 >> v1:172.21.15.70:6789/0 conn(0x7f3c00162350 legacy=0x7f3c0006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bfdffb640 1 -- 172.21.15.70:0/3051998586 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169412401 0 0) 0x7f3c0006e340 con 0x7f3c00162350 2024-04-22T01:16:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.146+0000 7f3bff7fe640 1 -- 172.21.15.70:0/3051998586 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0006ebe0 con 0x7f3c00072280 2024-04-22T01:16:56.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bff7fe640 1 --2- 172.21.15.70:0/3051998586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0016d120 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009d90 tx=0x7f3bf801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd72436f7bc7369 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfdffb640 1 -- 172.21.15.70:0/3051998586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80b2070 con 0x7f3c00072280 2024-04-22T01:16:56.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfdffb640 1 -- 172.21.15.70:0/3051998586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf80a5950 con 0x7f3c00072280 2024-04-22T01:16:56.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfdffb640 1 -- 172.21.15.70:0/3051998586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80a5d20 con 0x7f3c00072280 2024-04-22T01:16:56.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3051998586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 msgr2=0x7f3c0016d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 --2- 172.21.15.70:0/3051998586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0016d120 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009d90 tx=0x7f3bf801c900 comp rx=0 tx=0).stop 2024-04-22T01:16:56.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3051998586 shutdown_connections 2024-04-22T01:16:56.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 --2- 172.21.15.70:0/3051998586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0016d120 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3051998586 >> 172.21.15.70:0/3051998586 conn(0x7f3c000b1d40 msgr2=0x7f3c000cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3051998586 shutdown_connections 2024-04-22T01:16:56.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3051998586 wait complete. 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 Processor -- start 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- start start 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c0006f3e0 0x7f3c0014d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0014c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c00162350 0x7f3c00164fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00170000 con 0x7f3c00072280 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0016fe80 con 0x7f3c00162350 2024-04-22T01:16:56.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3c00170180 con 0x7f3c0006f3e0 2024-04-22T01:16:56.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0014c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0014c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45768/0 (socket says 172.21.15.70:45768) 2024-04-22T01:16:56.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 -- 172.21.15.70:0/3544990950 learned_addr learned my addr 172.21.15.70:0/3544990950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bff7fe640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c00162350 0x7f3c00164fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfeffd640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c0006f3e0 0x7f3c0014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 -- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c0006f3e0 msgr2=0x7f3c0014d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c0006f3e0 0x7f3c0014d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 -- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c00162350 msgr2=0x7f3c00164fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c00162350 0x7f3c00164fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 -- 172.21.15.70:0/3544990950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c00163700 con 0x7f3c00072280 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3bfe7fc640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0014c7f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400c450 tx=0x7f3bf400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4012830 con 0x7f3c00072280 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf4017070 con 0x7f3c00072280 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c00157a10 con 0x7f3c00072280 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4012c30 con 0x7f3c00072280 2024-04-22T01:16:56.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.150+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0014cd30 con 0x7f3c00072280 2024-04-22T01:16:56.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.154+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c00068210 con 0x7f3c00072280 2024-04-22T01:16:56.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.154+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3bf401c460 con 0x7f3c00072280 2024-04-22T01:16:56.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.154+0000 7f3be77fe640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bec0678f0 0x7f3bec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.154+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f3bf4085c20 con 0x7f3c00072280 2024-04-22T01:16:56.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.154+0000 7f3bfeffd640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bec0678f0 0x7f3bec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.154+0000 7f3bfeffd640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bec0678f0 0x7f3bec069db0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0000f70 tx=0x7f3bf0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.158+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3bf40565d0 con 0x7f3c00072280 2024-04-22T01:16:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.318+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3c00165b00 con 0x7f3c00072280 2024-04-22T01:16:56.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.318+0000 7f3be77fe640 1 -- 172.21.15.70:0/3544990950 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7f3bf4055ed0 con 0x7f3c00072280 2024-04-22T01:16:56.319 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bec0678f0 msgr2=0x7f3bec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bec0678f0 0x7f3bec069db0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0000f70 tx=0x7f3bf0009040 comp rx=0 tx=0).stop 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 msgr2=0x7f3c0014c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0014c7f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400c450 tx=0x7f3bf400c910 comp rx=0 tx=0).stop 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 shutdown_connections 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3bec0678f0 0x7f3bec069db0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c00162350 0x7f3c00164fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c00072280 0x7f3c0014c7f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 --2- 172.21.15.70:0/3544990950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c0006f3e0 0x7f3c0014d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 >> 172.21.15.70:0/3544990950 conn(0x7f3c000b1d40 msgr2=0x7f3c0015f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 shutdown_connections 2024-04-22T01:16:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.322+0000 7f3c05651640 1 -- 172.21.15.70:0/3544990950 wait complete. 2024-04-22T01:16:56.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 Processor -- start 2024-04-22T01:16:56.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 -- start start 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0b4450 0x7f70dc071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70dc0720b0 0x7f70dc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70dc15c290 con 0x7f70dc0cd050 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f70dc072510 con 0x7f70dc0720b0 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70e1fe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc0b3a50 con 0x7f70dc0b4450 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70daffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70dc0720b0 0x7f70dc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70dbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f70dc0cd050 0x7f70dc15f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34038/0 (socket says 172.21.15.70:34038) 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70daffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70dc0720b0 0x7f70dc0ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45782/0 (socket says 172.21.15.70:45782) 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70dbfff640 1 -- 172.21.15.70:0/2577056291 learned_addr learned my addr 172.21.15.70:0/2577056291 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70db7fe640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0b4450 0x7f70dc071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70daffd640 1 -- 172.21.15.70:0/2577056291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0b4450 msgr2=0x7f70dc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70daffd640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0b4450 0x7f70dc071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70daffd640 1 -- 172.21.15.70:0/2577056291 >> v1:172.21.15.70:6789/0 conn(0x7f70dc0cd050 legacy=0x7f70dc15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70da7fc640 1 -- 172.21.15.70:0/2577056291 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543017672 0 0) 0x7f70dc15c290 con 0x7f70dc0cd050 2024-04-22T01:16:56.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70daffd640 1 -- 172.21.15.70:0/2577056291 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc15c570 con 0x7f70dc0720b0 2024-04-22T01:16:56.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.462+0000 7f70db7fe640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0b4450 0x7f70dc071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:56.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70daffd640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0720b0 0x7f70dc0ccb10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f70d00097c0 tx=0x7f70d001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82161f976c91b7a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70da7fc640 1 -- 172.21.15.70:0/2577056291 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00b2070 con 0x7f70dc0720b0 2024-04-22T01:16:56.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70da7fc640 1 -- 172.21.15.70:0/2577056291 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d00ad640 con 0x7f70dc0720b0 2024-04-22T01:16:56.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70da7fc640 1 -- 172.21.15.70:0/2577056291 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00ada10 con 0x7f70dc0720b0 2024-04-22T01:16:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 -- 172.21.15.70:0/2577056291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0720b0 msgr2=0x7f70dc0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0720b0 0x7f70dc0ccb10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f70d00097c0 tx=0x7f70d001c470 comp rx=0 tx=0).stop 2024-04-22T01:16:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 -- 172.21.15.70:0/2577056291 shutdown_connections 2024-04-22T01:16:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0720b0 0x7f70dc0ccb10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/2577056291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0b4450 0x7f70dc071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 -- 172.21.15.70:0/2577056291 >> 172.21.15.70:0/2577056291 conn(0x7f70dc0b1d40 msgr2=0x7f70dc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 -- 172.21.15.70:0/2577056291 shutdown_connections 2024-04-22T01:16:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.466+0000 7f70e1fe2640 1 -- 172.21.15.70:0/2577056291 wait complete. 2024-04-22T01:16:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 Processor -- start 2024-04-22T01:16:56.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- start start 2024-04-22T01:16:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0720b0 0x7f70dc156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 0x7f70dc14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70dc0cd050 0x7f70dc14a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc0b4830 con 0x7f70dc0b4450 2024-04-22T01:16:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc0cd480 con 0x7f70dc0cd050 2024-04-22T01:16:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc0b49b0 con 0x7f70dc0720b0 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 0x7f70dc14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 0x7f70dc14a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45792/0 (socket says 172.21.15.70:45792) 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 -- 172.21.15.70:0/3215465498 learned_addr learned my addr 172.21.15.70:0/3215465498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70dbfff640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70dc0cd050 0x7f70dc14a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70db7fe640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0720b0 0x7f70dc156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 -- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0720b0 msgr2=0x7f70dc156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0720b0 0x7f70dc156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 -- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70dc0cd050 msgr2=0x7f70dc14a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70dc0cd050 0x7f70dc14a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 -- 172.21.15.70:0/3215465498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc1596f0 con 0x7f70dc0b4450 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70db7fe640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0720b0 0x7f70dc156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70daffd640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 0x7f70dc14a210 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f70d001a0a0 tx=0x7f70d0019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00b2040 con 0x7f70dc0b4450 2024-04-22T01:16:56.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d00bc070 con 0x7f70dc0b4450 2024-04-22T01:16:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70dc1599d0 con 0x7f70dc0b4450 2024-04-22T01:16:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00b6870 con 0x7f70dc0b4450 2024-04-22T01:16:56.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70dc159f90 con 0x7f70dc0b4450 2024-04-22T01:16:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.470+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70dc068210 con 0x7f70dc0b4450 2024-04-22T01:16:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.474+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70d00ad4c0 con 0x7f70dc0b4450 2024-04-22T01:16:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.474+0000 7f70c3fff640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70c80678c0 0x7f70c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.474+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f70d0120060 con 0x7f70dc0b4450 2024-04-22T01:16:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.474+0000 7f70db7fe640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70c80678c0 0x7f70c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.474+0000 7f70db7fe640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70c80678c0 0x7f70c8069d80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f70dc156b00 tx=0x7f70cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.478+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f70d0016090 con 0x7f70dc0b4450 2024-04-22T01:16:56.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70dc0d01e0 con 0x7f70dc0b4450 2024-04-22T01:16:56.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70c3fff640 1 -- 172.21.15.70:0/3215465498 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) ==== 104+0+11 (secure 0 0 0) 0x7f70d00f0b40 con 0x7f70dc0b4450 2024-04-22T01:16:56.638 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:16:56.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70c80678c0 msgr2=0x7f70c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70c80678c0 0x7f70c8069d80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f70dc156b00 tx=0x7f70cc008040 comp rx=0 tx=0).stop 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 msgr2=0x7f70dc14a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 0x7f70dc14a210 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f70d001a0a0 tx=0x7f70d0019210 comp rx=0 tx=0).stop 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 shutdown_connections 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f70c80678c0 0x7f70c8069d80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70dc0cd050 0x7f70dc14a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70dc0b4450 0x7f70dc14a210 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 --2- 172.21.15.70:0/3215465498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0720b0 0x7f70dc156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 >> 172.21.15.70:0/3215465498 conn(0x7f70dc0b1d40 msgr2=0x7f70dc0cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 shutdown_connections 2024-04-22T01:16:56.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.638+0000 7f70e1fe2640 1 -- 172.21.15.70:0/3215465498 wait complete. 2024-04-22T01:16:56.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:56.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 Processor -- start 2024-04-22T01:16:56.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 -- start start 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f537015f3e0 0x7f5370099d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53700b43e0 0x7f537009a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53700c5720 con 0x7f53700cd280 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f53700cd710 con 0x7f537015f3e0 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5378617640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f53700cd9f0 con 0x7f53700b43e0 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f537015f3e0 0x7f5370099d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f537015f3e0 0x7f5370099d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45800/0 (socket says 172.21.15.70:45800) 2024-04-22T01:16:56.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 -- 172.21.15.70:0/974504865 learned_addr learned my addr 172.21.15.70:0/974504865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:56.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f5376bb4640 1 --2- 172.21.15.70:0/974504865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53700b43e0 0x7f537009a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 -- 172.21.15.70:0/974504865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53700b43e0 msgr2=0x7f537009a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:56.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 --2- 172.21.15.70:0/974504865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53700b43e0 0x7f537009a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 -- 172.21.15.70:0/974504865 >> v1:172.21.15.98:6789/0 conn(0x7f53700cd280 legacy=0x7f53700c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:56.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.778+0000 7f53763b3640 1 -- 172.21.15.70:0/974504865 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53700cdcd0 con 0x7f537015f3e0 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f53763b3640 1 --2- 172.21.15.70:0/974504865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f5370099d30 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f53700733f0 tx=0x7f536c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356cbcca160aaa7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f53753b1640 1 -- 172.21.15.70:0/974504865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c0aa5f0 con 0x7f537015f3e0 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f53753b1640 1 -- 172.21.15.70:0/974504865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f536c0af070 con 0x7f537015f3e0 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f53753b1640 1 -- 172.21.15.70:0/974504865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c0aa9b0 con 0x7f537015f3e0 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- 172.21.15.70:0/974504865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 msgr2=0x7f5370099d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 --2- 172.21.15.70:0/974504865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f5370099d30 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f53700733f0 tx=0x7f536c01c4b0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- 172.21.15.70:0/974504865 shutdown_connections 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 --2- 172.21.15.70:0/974504865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53700b43e0 0x7f537009a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 --2- 172.21.15.70:0/974504865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f5370099d30 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- 172.21.15.70:0/974504865 >> 172.21.15.70:0/974504865 conn(0x7f53700b36c0 msgr2=0x7f53700b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- 172.21.15.70:0/974504865 shutdown_connections 2024-04-22T01:16:56.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- 172.21.15.70:0/974504865 wait complete. 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 Processor -- start 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- start start 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53700b43e0 0x7f53700b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f537014b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370150370 0x7f5370150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53700c6ed0 con 0x7f537015f3e0 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53700c6d50 con 0x7f53700b43e0 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5378617640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5370168b30 con 0x7f5370150370 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f537014b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f537014b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45802/0 (socket says 172.21.15.70:45802) 2024-04-22T01:16:56.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 -- 172.21.15.70:0/3380494658 learned_addr learned my addr 172.21.15.70:0/3380494658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f53763b3640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53700b43e0 0x7f53700b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5376bb4640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370150370 0x7f5370150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 -- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370150370 msgr2=0x7f5370150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370150370 0x7f5370150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 -- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53700b43e0 msgr2=0x7f53700b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53700b43e0 0x7f53700b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 -- 172.21.15.70:0/3380494658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5370150e30 con 0x7f537015f3e0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5376bb4640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370150370 0x7f5370150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.782+0000 7f5375bb2640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f537014b440 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f536000b410 tx=0x7f536000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360015030 con 0x7f537015f3e0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5360004020 con 0x7f537015f3e0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5370151110 con 0x7f537015f3e0 2024-04-22T01:16:56.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360004450 con 0x7f537015f3e0 2024-04-22T01:16:56.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537006c5e0 con 0x7f537015f3e0 2024-04-22T01:16:56.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5370068920 con 0x7f537015f3e0 2024-04-22T01:16:56.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5360018070 con 0x7f537015f3e0 2024-04-22T01:16:56.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5366ffd640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53540678c0 0x7f5354069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:56.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f5360085a50 con 0x7f537015f3e0 2024-04-22T01:16:56.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f53763b3640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53540678c0 0x7f5354069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:56.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.786+0000 7f53763b3640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53540678c0 0x7f5354069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f536c0ae000 tx=0x7f536c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:56.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.790+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5360056400 con 0x7f537015f3e0 2024-04-22T01:16:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f537009a390 con 0x7f537015f3e0 2024-04-22T01:16:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5366ffd640 1 -- 172.21.15.70:0/3380494658 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) ==== 100+0+11 (secure 0 0 0) 0x7f5360055d00 con 0x7f537015f3e0 2024-04-22T01:16:56.939 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53540678c0 msgr2=0x7f5354069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53540678c0 0x7f5354069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f536c0ae000 tx=0x7f536c0a7000 comp rx=0 tx=0).stop 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 msgr2=0x7f537014b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f537014b440 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f536000b410 tx=0x7f536000b8d0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 shutdown_connections 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f53540678c0 0x7f5354069d80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5370150370 0x7f5370150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f537015f3e0 0x7f537014b440 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 --2- 172.21.15.70:0/3380494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53700b43e0 0x7f53700b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 >> 172.21.15.70:0/3380494658 conn(0x7f53700b36c0 msgr2=0x7f53700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 shutdown_connections 2024-04-22T01:16:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:56.938+0000 7f5378617640 1 -- 172.21.15.70:0/3380494658 wait complete. 2024-04-22T01:16:56.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:16:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf80dc2640 1 Processor -- start 2024-04-22T01:16:57.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf80dc2640 1 -- start start 2024-04-22T01:16:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf80dc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 0x7fdf7c166c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf80dc2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c0cd7a0 con 0x7fdf7c15bb50 2024-04-22T01:16:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf80dc2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c15fb00 con 0x7fdf7c15bf30 2024-04-22T01:16:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf80dc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c0cd580 con 0x7fdf7c06f480 2024-04-22T01:16:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdf7c15bb50 0x7fdf7c0cce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34052/0 (socket says 172.21.15.70:34052) 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7a59c640 1 -- 172.21.15.70:0/1954314954 learned_addr learned my addr 172.21.15.70:0/1954314954 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7ad9d640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 0x7fdf7c166c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903559640 0 0) 0x7fdf7c0cd7a0 con 0x7fdf7c15bb50 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf64003530 con 0x7fdf7c15bb50 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 967224879 0 0) 0x7fdf64003530 con 0x7fdf7c15bb50 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 msgr2=0x7fdf7c166c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 0x7fdf7c166c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 >> v1:172.21.15.98:6789/0 conn(0x7fdf7c15bf30 legacy=0x7fdf7c06ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c15fde0 con 0x7fdf7c15bb50 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fdf6c001680 con 0x7fdf7c15bb50 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 >> v1:172.21.15.70:6789/0 conn(0x7fdf7c15bb50 legacy=0x7fdf7c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64003a60 0x7fdf64003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64004810 0x7fdf64004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 0x7fdf640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c0cd580 con 0x7fdf64004810 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6400bb00 con 0x7fdf64003a60 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.066+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6400bd00 con 0x7fdf7c06f480 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64004810 0x7fdf64004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7a59c640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64003a60 0x7fdf64003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 -- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 msgr2=0x7fdf640051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7ad9d640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 0x7fdf640051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7c06f480 0x7fdf640051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 -- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64003a60 msgr2=0x7fdf64003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64003a60 0x7fdf64003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 -- 172.21.15.70:0/1954314954 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64003530 con 0x7fdf64004810 2024-04-22T01:16:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7a59c640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf64003a60 0x7fdf64003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64004810 0x7fdf64004c90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdf70001780 tx=0x7fdf7001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5807f9fd89040e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf700013a0 con 0x7fdf64004810 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf700052f0 con 0x7fdf64004810 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7959a640 1 -- 172.21.15.70:0/1954314954 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70003840 con 0x7fdf64004810 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf80dc2640 1 -- 172.21.15.70:0/1954314954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64004810 msgr2=0x7fdf64004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/1954314954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64004810 0x7fdf64004c90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdf70001780 tx=0x7fdf7001c420 comp rx=0 tx=0).stop 2024-04-22T01:16:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf7ad9d640 1 -- 172.21.15.70:0/1954314954 reap_dead start 2024-04-22T01:16:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf80dc2640 1 -- 172.21.15.70:0/1954314954 shutdown_connections 2024-04-22T01:16:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf80dc2640 1 -- 172.21.15.70:0/1954314954 >> 172.21.15.70:0/1954314954 conn(0x7fdf7c0cf9e0 msgr2=0x7fdf7c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf80dc2640 1 -- 172.21.15.70:0/1954314954 shutdown_connections 2024-04-22T01:16:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.070+0000 7fdf80dc2640 1 -- 172.21.15.70:0/1954314954 wait complete. 2024-04-22T01:16:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 Processor -- start 2024-04-22T01:16:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- start start 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf64004810 0x7fdf7c143d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 0x7fdf7c0ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c146a50 0x7fdf7c14ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c15f8b0 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c0cf460 con 0x7fdf7c146a50 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c0cf5e0 con 0x7fdf64004810 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 0x7fdf7c0ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 0x7fdf7c0ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45814/0 (socket says 172.21.15.70:45814) 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 -- 172.21.15.70:0/426284715 learned_addr learned my addr 172.21.15.70:0/426284715 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf7a59c640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf64004810 0x7fdf7c143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf7ad9d640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c146a50 0x7fdf7c14ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 -- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf64004810 msgr2=0x7fdf7c143d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf64004810 0x7fdf7c143d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 -- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c146a50 msgr2=0x7fdf7c14ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c146a50 0x7fdf7c14ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 -- 172.21.15.70:0/426284715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c0caeb0 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf7a59c640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf64004810 0x7fdf7c143d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf7ad9d640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c146a50 0x7fdf7c14ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf79d9b640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 0x7fdf7c0ca970 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf70005040 tx=0x7fdf64005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70007a40 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf700bb070 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c0c93a0 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf700069e0 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c0c9960 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c04f510 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.074+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf700b4040 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.078+0000 7fdf6affd640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf5c0678c0 0x7fdf5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.078+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fdf70120030 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.078+0000 7fdf7a59c640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf5c0678c0 0x7fdf5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.078+0000 7fdf7a59c640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf5c0678c0 0x7fdf5c069d80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c001210 tx=0x7fdf6c000f60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.082+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdf7c0caeb0 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.238+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdf7c06f260 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.238+0000 7fdf6affd640 1 -- 172.21.15.70:0/426284715 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v52) ==== 80+0+0 (secure 0 0 0) 0x7fdf70016130 con 0x7fdf7c0c8d50 2024-04-22T01:16:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf5c0678c0 msgr2=0x7fdf5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf5c0678c0 0x7fdf5c069d80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c001210 tx=0x7fdf6c000f60 comp rx=0 tx=0).stop 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 msgr2=0x7fdf7c0ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 0x7fdf7c0ca970 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf70005040 tx=0x7fdf64005840 comp rx=0 tx=0).stop 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 shutdown_connections 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf5c0678c0 0x7fdf5c069d80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c146a50 0x7fdf7c14ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c0c8d50 0x7fdf7c0ca970 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 --2- 172.21.15.70:0/426284715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf64004810 0x7fdf7c143d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 >> 172.21.15.70:0/426284715 conn(0x7fdf7c0cf9e0 msgr2=0x7fdf7c06b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 shutdown_connections 2024-04-22T01:16:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.242+0000 7fdf80dc2640 1 -- 172.21.15.70:0/426284715 wait complete. 2024-04-22T01:16:57.258 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:16:57.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:57.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:16:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34057]: fuse finished with error 0 and tester_r 0 2024-04-22T01:16:57.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:16:57.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:57.274 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:16:57.363 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:57.363 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:16:57.363 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:57.369 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:16:57.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:16:57.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:57.369 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:16:57.418 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-22T01:16:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:57.419 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:16:57.419 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:16:57.420 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:57.425 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:16:57.425 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:16:57.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:57.426 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:16:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-22T01:16:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:16:57.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.602+0000 7ff7dd9b2640 1 Processor -- start 2024-04-22T01:16:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.602+0000 7ff7dd9b2640 1 -- start start 2024-04-22T01:16:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7dd9b2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff7d8145570 0x7ff7d80d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7dd9b2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7d8145950 con 0x7ff7d80cdee0 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7dd9b2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7d80d06f0 con 0x7ff7d80b2d70 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7dd9b2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff7d815f960 con 0x7ff7d8145570 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff7d8145570 0x7ff7d80d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff7d8145570 0x7ff7d80d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45822/0 (socket says 172.21.15.70:45822) 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 -- 172.21.15.70:0/368494493 learned_addr learned my addr 172.21.15.70:0/368494493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d67fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff7d80cdee0 0x7ff7d80b2660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34056/0 (socket says 172.21.15.70:34056) 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 -- 172.21.15.70:0/368494493 >> v1:172.21.15.98:6789/0 conn(0x7ff7d80b2d70 legacy=0x7ff7d8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 -- 172.21.15.70:0/368494493 >> v1:172.21.15.70:6789/0 conn(0x7ff7d80cdee0 legacy=0x7ff7d80b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 -- 172.21.15.70:0/368494493 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d815fc40 con 0x7ff7d8145570 2024-04-22T01:16:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d5ffb640 1 -- 172.21.15.70:0/368494493 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828730668 0 0) 0x7ff7c0001120 con 0x7ff7d80cdee0 2024-04-22T01:16:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d6ffd640 1 --2- 172.21.15.70:0/368494493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d80d0010 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009e50 tx=0x7ff7d001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c485df77d010191d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d5ffb640 1 -- 172.21.15.70:0/368494493 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d00a5880 con 0x7ff7d8145570 2024-04-22T01:16:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d5ffb640 1 -- 172.21.15.70:0/368494493 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d00a5a20 con 0x7ff7d8145570 2024-04-22T01:16:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.606+0000 7ff7d5ffb640 1 -- 172.21.15.70:0/368494493 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d00a5df0 con 0x7ff7d8145570 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/368494493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 msgr2=0x7ff7d80d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/368494493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d80d0010 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009e50 tx=0x7ff7d001c9c0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/368494493 shutdown_connections 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/368494493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d80d0010 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/368494493 >> 172.21.15.70:0/368494493 conn(0x7ff7d806c300 msgr2=0x7ff7d806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/368494493 shutdown_connections 2024-04-22T01:16:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/368494493 wait complete. 2024-04-22T01:16:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 Processor -- start 2024-04-22T01:16:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- start start 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7d80b2d70 0x7ff7d815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d80cdee0 0x7ff7d8138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d8160720 con 0x7ff7d8145570 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d8169d10 con 0x7ff7d80cdee0 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d815dfc0 con 0x7ff7d80b2d70 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d815afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45838/0 (socket says 172.21.15.70:45838) 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 -- 172.21.15.70:0/2813015718 learned_addr learned my addr 172.21.15.70:0/2813015718 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d6ffd640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7d80b2d70 0x7ff7d815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d67fc640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d80cdee0 0x7ff7d8138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 -- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7d80b2d70 msgr2=0x7ff7d815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7d80b2d70 0x7ff7d815cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 -- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d80cdee0 msgr2=0x7ff7d8138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d80cdee0 0x7ff7d8138f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 -- 172.21.15.70:0/2813015718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d81394a0 con 0x7ff7d8145570 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d6ffd640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7d80b2d70 0x7ff7d815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7d77fe640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d815afd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c440 tx=0x7ff7c800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8017070 con 0x7ff7d8145570 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c8003600 con 0x7ff7d8145570 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d815b510 con 0x7ff7d8145570 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80039f0 con 0x7ff7d8145570 2024-04-22T01:16:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.610+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d815d690 con 0x7ff7d8145570 2024-04-22T01:16:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.614+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d8068210 con 0x7ff7d8145570 2024-04-22T01:16:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.614+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff7c801a070 con 0x7ff7d8145570 2024-04-22T01:16:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.614+0000 7ff7cffff640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7b80678f0 0x7ff7b8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.614+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7ff7c8085250 con 0x7ff7d8145570 2024-04-22T01:16:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.614+0000 7ff7d6ffd640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7b80678f0 0x7ff7b8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.614+0000 7ff7d6ffd640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7b80678f0 0x7ff7b8069db0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00ae000 tx=0x7ff7d00a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.618+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff7c8055b70 con 0x7ff7d8145570 2024-04-22T01:16:57.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.806+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7d81608a0 con 0x7ff7d8145570 2024-04-22T01:16:57.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7cffff640 1 -- 172.21.15.70:0/2813015718 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5178 (secure 0 0 0) 0x7ff7c8055470 con 0x7ff7d8145570 2024-04-22T01:16:57.810 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:57.810 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:41.643333+0000","modified":"2024-04-22T01:16:44.689368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5922},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5922":{"gid":5922,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2188446179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2188446179},{"type":"v1","addr":"172.21.15.70:6837","nonce":2188446179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5922,"qdb_cluster":[5922]},"id":3}]} 2024-04-22T01:16:57.811 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 25 2024-04-22T01:16:57.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7b80678f0 msgr2=0x7ff7b8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7b80678f0 0x7ff7b8069db0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00ae000 tx=0x7ff7d00a7070 comp rx=0 tx=0).stop 2024-04-22T01:16:57.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 msgr2=0x7ff7d815afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d815afd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c440 tx=0x7ff7c800c900 comp rx=0 tx=0).stop 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 shutdown_connections 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7b80678f0 0x7ff7b8069db0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d8145570 0x7ff7d815afd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d80cdee0 0x7ff7d8138f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 --2- 172.21.15.70:0/2813015718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7d80b2d70 0x7ff7d815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 >> 172.21.15.70:0/2813015718 conn(0x7ff7d806c300 msgr2=0x7ff7d806ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 shutdown_connections 2024-04-22T01:16:57.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.810+0000 7ff7dd9b2640 1 -- 172.21.15.70:0/2813015718 wait complete. 2024-04-22T01:16:57.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:57.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 Processor -- start 2024-04-22T01:16:57.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 -- start start 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15081455b0 0x7f15080cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15080cc150 0x7f1508071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1508072060 0x7f150815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f15080724d0 con 0x7f1508072060 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f15080d0250 con 0x7f15081455b0 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150fc48640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f15080cd7b0 con 0x7f15080cc150 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1508072060 0x7f150815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1508072060 0x7f150815f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45844/0 (socket says 172.21.15.70:45844) 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 -- 172.21.15.70:0/3766765572 learned_addr learned my addr 172.21.15.70:0/3766765572 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150d1e3640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15080cc150 0x7f1508071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150d9e4640 1 --2- 172.21.15.70:0/3766765572 >> v2:172.21.15.98:3300/0 conn(0x7f15081455b0 0x7f15080cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 -- 172.21.15.70:0/3766765572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15080cc150 msgr2=0x7f1508071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15080cc150 0x7f1508071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 -- 172.21.15.70:0/3766765572 >> v2:172.21.15.98:3300/0 conn(0x7f15081455b0 msgr2=0x7f15080cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 --2- 172.21.15.70:0/3766765572 >> v2:172.21.15.98:3300/0 conn(0x7f15081455b0 0x7f15080cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 -- 172.21.15.70:0/3766765572 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15080cc710 con 0x7f1508072060 2024-04-22T01:16:57.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150d1e3640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15080cc150 0x7f1508071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:57.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.946+0000 7f150e1e5640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1508072060 0x7f150815f7a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f15080d0700 tx=0x7f14f801c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86d13a680989b95 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150c9e2640 1 -- 172.21.15.70:0/3766765572 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f80a6040 con 0x7f1508072060 2024-04-22T01:16:57.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150c9e2640 1 -- 172.21.15.70:0/3766765572 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14f80af070 con 0x7f1508072060 2024-04-22T01:16:57.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150c9e2640 1 -- 172.21.15.70:0/3766765572 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f80aa660 con 0x7f1508072060 2024-04-22T01:16:57.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- 172.21.15.70:0/3766765572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1508072060 msgr2=0x7f150815f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1508072060 0x7f150815f7a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f15080d0700 tx=0x7f14f801c610 comp rx=0 tx=0).stop 2024-04-22T01:16:57.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- 172.21.15.70:0/3766765572 shutdown_connections 2024-04-22T01:16:57.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1508072060 0x7f150815f7a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- 172.21.15.70:0/3766765572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15080cc150 0x7f1508071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- 172.21.15.70:0/3766765572 >> v2:172.21.15.98:3300/0 conn(0x7f15081455b0 0x7f15080cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- 172.21.15.70:0/3766765572 >> 172.21.15.70:0/3766765572 conn(0x7f150806b1f0 msgr2=0x7f150806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- 172.21.15.70:0/3766765572 shutdown_connections 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- 172.21.15.70:0/3766765572 wait complete. 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 Processor -- start 2024-04-22T01:16:57.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- start start 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1508072060 0x7f15080b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 0x7f15081f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15081455b0 0x7f15081f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f150806e380 con 0x7f15080cc150 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f150806e200 con 0x7f1508072060 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150fc48640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f150806e500 con 0x7f15081455b0 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 0x7f15081f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 0x7f15081f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45850/0 (socket says 172.21.15.70:45850) 2024-04-22T01:16:57.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 -- 172.21.15.70:0/2856451321 learned_addr learned my addr 172.21.15.70:0/2856451321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d9e4640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1508072060 0x7f15080b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 -- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15081455b0 msgr2=0x7f15081f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15081455b0 0x7f15081f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 -- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1508072060 msgr2=0x7f15080b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1508072060 0x7f15080b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.950+0000 7f150d1e3640 1 -- 172.21.15.70:0/2856451321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15080b1eb0 con 0x7f15080cc150 2024-04-22T01:16:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f150d1e3640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 0x7f15081f6ff0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f150400e840 tx=0x7f150400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150400f040 con 0x7f15080cc150 2024-04-22T01:16:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1504004030 con 0x7f15080cc150 2024-04-22T01:16:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15080b2190 con 0x7f15080cc150 2024-04-22T01:16:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504004400 con 0x7f15080cc150 2024-04-22T01:16:57.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15080b2750 con 0x7f15080cc150 2024-04-22T01:16:57.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1508068210 con 0x7f15080cc150 2024-04-22T01:16:57.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f150400c6d0 con 0x7f15080cc150 2024-04-22T01:16:57.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f14fe7fc640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f40678c0 0x7f14f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:57.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f1504085af0 con 0x7f15080cc150 2024-04-22T01:16:57.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f150d9e4640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f40678c0 0x7f14f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:57.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.954+0000 7f150d9e4640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f40678c0 0x7f14f4069d80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f15080af070 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:57.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:57.958+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1504017020 con 0x7f15080cc150 2024-04-22T01:16:58.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.122+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f150815f510 con 0x7f15080cc150 2024-04-22T01:16:58.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.122+0000 7f14fe7fc640 1 -- 172.21.15.70:0/2856451321 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f15040564a0 con 0x7f15080cc150 2024-04-22T01:16:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:43.266695+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:58.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.122+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f40678c0 msgr2=0x7f14f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.122+0000 7f150fc48640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f40678c0 0x7f14f4069d80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f15080af070 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 msgr2=0x7f15081f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 0x7f15081f6ff0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f150400e840 tx=0x7f150400ed00 comp rx=0 tx=0).stop 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 shutdown_connections 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f40678c0 0x7f14f4069d80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15081455b0 0x7f15081f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080cc150 0x7f15081f6ff0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 --2- 172.21.15.70:0/2856451321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1508072060 0x7f15080b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 >> 172.21.15.70:0/2856451321 conn(0x7f150806b1f0 msgr2=0x7f15080744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 shutdown_connections 2024-04-22T01:16:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.126+0000 7f150fc48640 1 -- 172.21.15.70:0/2856451321 wait complete. 2024-04-22T01:16:58.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:16:39.917376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:16:40.609582+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:16:58.142 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:58.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 Processor -- start 2024-04-22T01:16:58.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 -- start start 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83c415de20 0x7f83c40b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 0x7f83c40cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c40cf3c0 con 0x7f83c40b6db0 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f83c40b75e0 con 0x7f83c415de20 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83cb3d8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83c40b78c0 con 0x7f83c40cea10 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f83c40b6db0 0x7f83c40d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34058/0 (socket says 172.21.15.70:34058) 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 -- 172.21.15.70:0/1677448214 learned_addr learned my addr 172.21.15.70:0/1677448214 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:58.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c9975640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 0x7f83c40cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c9174640 1 --2- 172.21.15.70:0/1677448214 >> v2:172.21.15.98:3300/0 conn(0x7f83c415de20 0x7f83c40b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743545262 0 0) 0x7f83c40cf3c0 con 0x7f83c40b6db0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83b0003530 con 0x7f83c40b6db0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1543304322 0 0) 0x7f83b0003530 con 0x7f83c40b6db0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 msgr2=0x7f83c40cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 0x7f83c40cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 >> v2:172.21.15.98:3300/0 conn(0x7f83c415de20 msgr2=0x7f83c40b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 --2- 172.21.15.70:0/1677448214 >> v2:172.21.15.98:3300/0 conn(0x7f83c415de20 0x7f83c40b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c40b7ba0 con 0x7f83c40b6db0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f83b4001680 con 0x7f83c40b6db0 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 >> v1:172.21.15.70:6789/0 conn(0x7f83c40b6db0 legacy=0x7f83c40d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b00039c0 0x7f83b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b0004810 0x7f83b0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 0x7f83b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c40b75e0 con 0x7f83b0004810 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83c40b78c0 con 0x7f83b00039c0 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83b000b9d0 con 0x7f83c40cea10 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b0004810 0x7f83b0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c9174640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b00039c0 0x7f83b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c9975640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 0x7f83b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 -- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 msgr2=0x7f83b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83c40cea10 0x7f83b00051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 -- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b00039c0 msgr2=0x7f83b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b00039c0 0x7f83b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c8973640 1 -- 172.21.15.70:0/1677448214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b0003530 con 0x7f83b0004810 2024-04-22T01:16:58.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.254+0000 7f83c9174640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83b00039c0 0x7f83b0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:58.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c8973640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b0004810 0x7f83b0004c90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f83b400c210 tx=0x7f83b401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e094acbcff7599d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b4007d60 con 0x7f83b0004810 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b40a9020 con 0x7f83b0004810 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83bbfff640 1 -- 172.21.15.70:0/1677448214 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b4005b20 con 0x7f83b0004810 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- 172.21.15.70:0/1677448214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b0004810 msgr2=0x7f83b0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/1677448214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b0004810 0x7f83b0004c90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f83b400c210 tx=0x7f83b401cca0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 -- 172.21.15.70:0/1677448214 reap_dead start 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- 172.21.15.70:0/1677448214 shutdown_connections 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- 172.21.15.70:0/1677448214 >> 172.21.15.70:0/1677448214 conn(0x7f83c4072d20 msgr2=0x7f83c4073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- 172.21.15.70:0/1677448214 shutdown_connections 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- 172.21.15.70:0/1677448214 wait complete. 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 Processor -- start 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- start start 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83b0004810 0x7f83c4152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40d44b0 0x7f83c4152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 0x7f83c40ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c415e200 con 0x7f83c40cc660 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83c40b8160 con 0x7f83c40d44b0 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f83c40b71f0 con 0x7f83b0004810 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 0x7f83c40ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 0x7f83c40ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45868/0 (socket says 172.21.15.70:45868) 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 -- 172.21.15.70:0/2459698951 learned_addr learned my addr 172.21.15.70:0/2459698951 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c8973640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40d44b0 0x7f83c4152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9174640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83b0004810 0x7f83c4152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 -- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83b0004810 msgr2=0x7f83c4152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83b0004810 0x7f83c4152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 -- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40d44b0 msgr2=0x7f83c4152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40d44b0 0x7f83c4152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 -- 172.21.15.70:0/2459698951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c4153200 con 0x7f83c40cc660 2024-04-22T01:16:58.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c8973640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40d44b0 0x7f83c4152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83c9975640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 0x7f83c40ccb10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f83c40d4d00 tx=0x7f83bc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc01c070 con 0x7f83c40cc660 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc017400 con 0x7f83c40cc660 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c41534a0 con 0x7f83c40cc660 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.258+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0177f0 con 0x7f83c40cc660 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c4153a60 con 0x7f83c40cc660 2024-04-22T01:16:58.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c4068210 con 0x7f83c40cc660 2024-04-22T01:16:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83bc01f070 con 0x7f83c40cc660 2024-04-22T01:16:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83b9ffb640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83a80678f0 0x7f83a8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f83bc085dd0 con 0x7f83c40cc660 2024-04-22T01:16:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83c9174640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83a80678f0 0x7f83a8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.262+0000 7f83c9174640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83a80678f0 0x7f83a8069db0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f83c40ca760 tx=0x7f83c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.266+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f83bc012070 con 0x7f83c40cc660 2024-04-22T01:16:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.422+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f83c415c920 con 0x7f83c40cc660 2024-04-22T01:16:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.422+0000 7f83b9ffb640 1 -- 172.21.15.70:0/2459698951 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7f83bc056810 con 0x7f83c40cc660 2024-04-22T01:16:58.424 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83a80678f0 msgr2=0x7f83a8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83a80678f0 0x7f83a8069db0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f83c40ca760 tx=0x7f83c0008040 comp rx=0 tx=0).stop 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 msgr2=0x7f83c40ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 0x7f83c40ccb10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f83c40d4d00 tx=0x7f83bc00f4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 shutdown_connections 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83a80678f0 0x7f83a8069db0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c40cc660 0x7f83c40ccb10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40d44b0 0x7f83c4152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 --2- 172.21.15.70:0/2459698951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83b0004810 0x7f83c4152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 >> 172.21.15.70:0/2459698951 conn(0x7f83c4072d20 msgr2=0x7f83c40b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 shutdown_connections 2024-04-22T01:16:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.426+0000 7f83cb3d8640 1 -- 172.21.15.70:0/2459698951 wait complete. 2024-04-22T01:16:58.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:58.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.550+0000 7fe5dfeb1640 1 Processor -- start 2024-04-22T01:16:58.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.550+0000 7fe5dfeb1640 1 -- start start 2024-04-22T01:16:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dfeb1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5d815de20 0x7fe5d80b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dfeb1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5d80cea10 0x7fe5d80cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dfeb1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5d80b75e0 con 0x7fe5d80b6db0 2024-04-22T01:16:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dfeb1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe5d80b78c0 con 0x7fe5d80cea10 2024-04-22T01:16:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dfeb1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe5d80b7ba0 con 0x7fe5d815de20 2024-04-22T01:16:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5d80cea10 0x7fe5d80cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5d80cea10 0x7fe5d80cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45882/0 (socket says 172.21.15.70:45882) 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 -- 172.21.15.70:0/1259183793 learned_addr learned my addr 172.21.15.70:0/1259183793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5ddc4d640 1 --2- 172.21.15.70:0/1259183793 >> v2:172.21.15.98:3300/0 conn(0x7fe5d815de20 0x7fe5d80b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 -- 172.21.15.70:0/1259183793 >> v2:172.21.15.98:3300/0 conn(0x7fe5d815de20 msgr2=0x7fe5d80b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 --2- 172.21.15.70:0/1259183793 >> v2:172.21.15.98:3300/0 conn(0x7fe5d815de20 0x7fe5d80b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 -- 172.21.15.70:0/1259183793 >> v1:172.21.15.98:6789/0 conn(0x7fe5d80b6db0 legacy=0x7fe5d80d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:58.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 -- 172.21.15.70:0/1259183793 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d80b7e80 con 0x7fe5d80cea10 2024-04-22T01:16:58.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5de44e640 1 --2- 172.21.15.70:0/1259183793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d80cee60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80d1ac0 tx=0x7fe5d001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f118ab4b5244e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dcc4b640 1 -- 172.21.15.70:0/1259183793 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d00a5820 con 0x7fe5d80cea10 2024-04-22T01:16:58.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dcc4b640 1 -- 172.21.15.70:0/1259183793 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d00a59c0 con 0x7fe5d80cea10 2024-04-22T01:16:58.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.554+0000 7fe5dcc4b640 1 -- 172.21.15.70:0/1259183793 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d00a5d90 con 0x7fe5d80cea10 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/1259183793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 msgr2=0x7fe5d80cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/1259183793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d80cee60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80d1ac0 tx=0x7fe5d001c980 comp rx=0 tx=0).stop 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/1259183793 shutdown_connections 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/1259183793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d80cee60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/1259183793 >> v2:172.21.15.98:3300/0 conn(0x7fe5d815de20 0x7fe5d80b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/1259183793 >> 172.21.15.70:0/1259183793 conn(0x7fe5d8072d20 msgr2=0x7fe5d8073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/1259183793 shutdown_connections 2024-04-22T01:16:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/1259183793 wait complete. 2024-04-22T01:16:58.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 Processor -- start 2024-04-22T01:16:58.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- start start 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5d80b6db0 0x7fe5d81544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d8159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d815de20 0x7fe5d8159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d80b7370 con 0x7fe5d80cea10 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d80b71f0 con 0x7fe5d815de20 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe5d80b8e90 con 0x7fe5d80b6db0 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d8159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d8159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45892/0 (socket says 172.21.15.70:45892) 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 -- 172.21.15.70:0/2316913871 learned_addr learned my addr 172.21.15.70:0/2316913871 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5ddc4d640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5d80b6db0 0x7fe5d81544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5de44e640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d815de20 0x7fe5d8159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 -- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5d80b6db0 msgr2=0x7fe5d81544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5d80b6db0 0x7fe5d81544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 -- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d815de20 msgr2=0x7fe5d8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d815de20 0x7fe5d8159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 -- 172.21.15.70:0/2316913871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d8159cb0 con 0x7fe5d80cea10 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dd44c640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d8159140 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800c450 tx=0x7fe5c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8012830 con 0x7fe5d80cea10 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8017070 con 0x7fe5d80cea10 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5d80c7830 con 0x7fe5d80cea10 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8012c30 con 0x7fe5d80cea10 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.558+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5d80c7df0 con 0x7fe5d80cea10 2024-04-22T01:16:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.562+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5d8068210 con 0x7fe5d80cea10 2024-04-22T01:16:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.562+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5c801c460 con 0x7fe5d80cea10 2024-04-22T01:16:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.562+0000 7fe5ce7fc640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5bc0678c0 0x7fe5bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.562+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fe5c8085c00 con 0x7fe5d80cea10 2024-04-22T01:16:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.562+0000 7fe5ddc4d640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5bc0678c0 0x7fe5bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.562+0000 7fe5ddc4d640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5bc0678c0 0x7fe5bc069d80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe5d81568d0 tx=0x7fe5d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.566+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe5c808c080 con 0x7fe5d80cea10 2024-04-22T01:16:58.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.714+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5d80b6420 con 0x7fe5d80cea10 2024-04-22T01:16:58.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.714+0000 7fe5ce7fc640 1 -- 172.21.15.70:0/2316913871 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) ==== 104+0+11 (secure 0 0 0) 0x7fe5c80565b0 con 0x7fe5d80cea10 2024-04-22T01:16:58.715 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:16:58.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5bc0678c0 msgr2=0x7fe5bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5bc0678c0 0x7fe5bc069d80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe5d81568d0 tx=0x7fe5d4008040 comp rx=0 tx=0).stop 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 msgr2=0x7fe5d8159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d8159140 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800c450 tx=0x7fe5c800c910 comp rx=0 tx=0).stop 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 shutdown_connections 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5bc0678c0 0x7fe5bc069d80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d815de20 0x7fe5d8159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d80cea10 0x7fe5d8159140 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 --2- 172.21.15.70:0/2316913871 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5d80b6db0 0x7fe5d81544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 >> 172.21.15.70:0/2316913871 conn(0x7fe5d8072d20 msgr2=0x7fe5d80cb250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 shutdown_connections 2024-04-22T01:16:58.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.718+0000 7fe5dfeb1640 1 -- 172.21.15.70:0/2316913871 wait complete. 2024-04-22T01:16:58.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:58.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.842+0000 7fea7f638640 1 Processor -- start 2024-04-22T01:16:58.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.842+0000 7fea7f638640 1 -- start start 2024-04-22T01:16:58.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7f638640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fea780b4450 0x7fea78071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7f638640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7815c290 con 0x7fea780720b0 2024-04-22T01:16:58.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7f638640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7815c570 con 0x7fea780cd220 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7f638640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fea7815c850 con 0x7fea780b4450 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7cbd3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fea780720b0 0x7fea780ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34072/0 (socket says 172.21.15.70:34072) 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7cbd3640 1 -- 172.21.15.70:0/3880450603 learned_addr learned my addr 172.21.15.70:0/3880450603 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7d3d4640 1 --2- 172.21.15.70:0/3880450603 >> v2:172.21.15.70:3300/0 conn(0x7fea780b4450 0x7fea78071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea6ffff640 1 -- 172.21.15.70:0/3880450603 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256428715 0 0) 0x7fea7815c290 con 0x7fea780720b0 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea6ffff640 1 -- 172.21.15.70:0/3880450603 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea64003530 con 0x7fea780720b0 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7d3d4640 1 -- 172.21.15.70:0/3880450603 >> v1:172.21.15.98:6789/0 conn(0x7fea780cd220 legacy=0x7fea780b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea6ffff640 1 -- 172.21.15.70:0/3880450603 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 392951713 0 0) 0x7fea64003530 con 0x7fea780720b0 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7d3d4640 1 -- 172.21.15.70:0/3880450603 >> v1:172.21.15.70:6789/0 conn(0x7fea780720b0 legacy=0x7fea780ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.846+0000 7fea7d3d4640 1 -- 172.21.15.70:0/3880450603 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7815cb30 con 0x7fea780b4450 2024-04-22T01:16:58.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7d3d4640 1 --2- 172.21.15.70:0/3880450603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780b4450 0x7fea78071b70 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fea740097c0 tx=0x7fea7401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e9a458fca6d701 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea6ffff640 1 -- 172.21.15.70:0/3880450603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740b0070 con 0x7fea780b4450 2024-04-22T01:16:58.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea6ffff640 1 -- 172.21.15.70:0/3880450603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea740ab400 con 0x7fea780b4450 2024-04-22T01:16:58.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea6ffff640 1 -- 172.21.15.70:0/3880450603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68001680 con 0x7fea780b4450 2024-04-22T01:16:58.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- 172.21.15.70:0/3880450603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780b4450 msgr2=0x7fea78071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 --2- 172.21.15.70:0/3880450603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780b4450 0x7fea78071b70 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fea740097c0 tx=0x7fea7401c3c0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- 172.21.15.70:0/3880450603 shutdown_connections 2024-04-22T01:16:58.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 --2- 172.21.15.70:0/3880450603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780b4450 0x7fea78071b70 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- 172.21.15.70:0/3880450603 >> 172.21.15.70:0/3880450603 conn(0x7fea780b1d40 msgr2=0x7fea780cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:58.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- 172.21.15.70:0/3880450603 shutdown_connections 2024-04-22T01:16:58.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- 172.21.15.70:0/3880450603 wait complete. 2024-04-22T01:16:58.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 Processor -- start 2024-04-22T01:16:58.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- start start 2024-04-22T01:16:58.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea780720b0 0x7fea7806e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea780b4450 0x7fea78157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 0x7fea781583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fea78169ca0 con 0x7fea780cd220 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fea78169b20 con 0x7fea780720b0 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7f638640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fea78169e20 con 0x7fea780b4450 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 0x7fea781583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 0x7fea781583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45902/0 (socket says 172.21.15.70:45902) 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 -- 172.21.15.70:0/2355539155 learned_addr learned my addr 172.21.15.70:0/2355539155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7d3d4640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea780720b0 0x7fea7806e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7cbd3640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea780b4450 0x7fea78157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 -- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea780b4450 msgr2=0x7fea78157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea780b4450 0x7fea78157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 -- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea780720b0 msgr2=0x7fea7806e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea780720b0 0x7fea7806e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 -- 172.21.15.70:0/2355539155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea78155870 con 0x7fea780cd220 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea7dbd5640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 0x7fea781583a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fea7000c7a0 tx=0x7fea7000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70003b80 con 0x7fea780cd220 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea70003d20 con 0x7fea780cd220 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.850+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7000aa90 con 0x7fea780cd220 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea78155590 con 0x7fea780cd220 2024-04-22T01:16:58.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7806dce0 con 0x7fea780cd220 2024-04-22T01:16:58.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea78068210 con 0x7fea780cd220 2024-04-22T01:16:58.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fea7000ac30 con 0x7fea780cd220 2024-04-22T01:16:58.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea6dffb640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fea5c0678f0 0x7fea5c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:58.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea7d3d4640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fea5c0678f0 0x7fea5c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:58.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fea70085d60 con 0x7fea780cd220 2024-04-22T01:16:58.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.854+0000 7fea7d3d4640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fea5c0678f0 0x7fea5c069db0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea740ad000 tx=0x7fea740a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:58.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:58.858+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fea70056710 con 0x7fea780cd220 2024-04-22T01:16:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.018+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fea780d01e0 con 0x7fea780cd220 2024-04-22T01:16:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.018+0000 7fea6dffb640 1 -- 172.21.15.70:0/2355539155 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) ==== 100+0+11 (secure 0 0 0) 0x7fea70056010 con 0x7fea780cd220 2024-04-22T01:16:59.020 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:16:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fea5c0678f0 msgr2=0x7fea5c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fea5c0678f0 0x7fea5c069db0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea740ad000 tx=0x7fea740a6000 comp rx=0 tx=0).stop 2024-04-22T01:16:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 msgr2=0x7fea781583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 0x7fea781583a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fea7000c7a0 tx=0x7fea7000cc60 comp rx=0 tx=0).stop 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 shutdown_connections 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fea5c0678f0 0x7fea5c069db0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea780cd220 0x7fea781583a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea780b4450 0x7fea78157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 --2- 172.21.15.70:0/2355539155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea780720b0 0x7fea7806e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 >> 172.21.15.70:0/2355539155 conn(0x7fea780b1d40 msgr2=0x7fea7815dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 shutdown_connections 2024-04-22T01:16:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.022+0000 7fea7f638640 1 -- 172.21.15.70:0/2355539155 wait complete. 2024-04-22T01:16:59.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:59.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.154+0000 7f45c7ac5640 1 Processor -- start 2024-04-22T01:16:59.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.154+0000 7f45c7ac5640 1 -- start start 2024-04-22T01:16:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c7ac5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45c015dd90 0x7f45c0099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c7ac5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45c009a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c7ac5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c00d03a0 con 0x7f45c00cfe50 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c7ac5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f45c00d0fb0 con 0x7f45c015dd90 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c7ac5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45c00d0680 con 0x7f45c006d790 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c6062640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f45c00cfe50 0x7f45c0161b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34086/0 (socket says 172.21.15.70:34086) 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c6062640 1 -- 172.21.15.70:0/3092555642 learned_addr learned my addr 172.21.15.70:0/3092555642 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5060640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45c009a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 --2- 172.21.15.70:0/3092555642 >> v2:172.21.15.98:3300/0 conn(0x7f45c015dd90 0x7f45c0099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418201348 0 0) 0x7f45c00d03a0 con 0x7f45c00cfe50 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45ac003530 con 0x7f45c00cfe50 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2199264339 0 0) 0x7f45ac003530 con 0x7f45c00cfe50 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 msgr2=0x7f45c009a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45c009a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 >> v2:172.21.15.98:3300/0 conn(0x7f45c015dd90 msgr2=0x7f45c0099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 --2- 172.21.15.70:0/3092555642 >> v2:172.21.15.98:3300/0 conn(0x7f45c015dd90 0x7f45c0099cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c00d0960 con 0x7f45c00cfe50 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f45b8001a00 con 0x7f45c00cfe50 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 >> v1:172.21.15.70:6789/0 conn(0x7f45c00cfe50 legacy=0x7f45c0161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac004810 0x7f45ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c00d0fb0 con 0x7f45ac0039c0 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c00d0680 con 0x7f45ac004810 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac00b9d0 con 0x7f45c006d790 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5060640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac004810 0x7f45ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c6062640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 -- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 msgr2=0x7f45ac0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45ac0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 -- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac004810 msgr2=0x7f45ac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac004810 0x7f45ac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 -- 172.21.15.70:0/3092555642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac003530 con 0x7f45ac0039c0 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c6062640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c006d790 0x7f45ac0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5060640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac004810 0x7f45ac004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c5861640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45ac003de0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009ae0 tx=0x7f45bc01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c4ecc01a167ec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.158+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0b2070 con 0x7f45ac0039c0 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc0a5590 con 0x7f45ac0039c0 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c485f640 1 -- 172.21.15.70:0/3092555642 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0096b0 con 0x7f45ac0039c0 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3092555642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 msgr2=0x7f45ac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3092555642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45ac003de0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009ae0 tx=0x7f45bc01ca80 comp rx=0 tx=0).stop 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c6062640 1 -- 172.21.15.70:0/3092555642 reap_dead start 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3092555642 shutdown_connections 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3092555642 >> 172.21.15.70:0/3092555642 conn(0x7f45c00b35d0 msgr2=0x7f45c00b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3092555642 shutdown_connections 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3092555642 wait complete. 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 Processor -- start 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- start start 2024-04-22T01:16:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45c015a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c015af90 0x7f45c0160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00b5db0 0x7f45c00cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c009a590 con 0x7f45ac0039c0 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c009a410 con 0x7f45c015af90 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45c0168c80 con 0x7f45c00b5db0 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45c015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45c015a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45926/0 (socket says 172.21.15.70:45926) 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 -- 172.21.15.70:0/3726214797 learned_addr learned my addr 172.21.15.70:0/3726214797 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c6062640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00b5db0 0x7f45c00cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5060640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c015af90 0x7f45c0160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 -- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00b5db0 msgr2=0x7f45c00cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00b5db0 0x7f45c00cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 -- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c015af90 msgr2=0x7f45c0160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c015af90 0x7f45c0160500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 -- 172.21.15.70:0/3726214797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c00b55f0 con 0x7f45ac0039c0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c5861640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45c015a980 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f45bc01c8b0 tx=0x7f45ac005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0bc020 con 0x7f45ac0039c0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc0a5d40 con 0x7f45ac0039c0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c01612f0 con 0x7f45ac0039c0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0b2040 con 0x7f45ac0039c0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.162+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c0161010 con 0x7f45ac0039c0 2024-04-22T01:16:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.166+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c0068920 con 0x7f45ac0039c0 2024-04-22T01:16:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.166+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f45bc004220 con 0x7f45ac0039c0 2024-04-22T01:16:59.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.166+0000 7f45b67fc640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45a40678f0 0x7f45a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.166+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f45bc11ff90 con 0x7f45ac0039c0 2024-04-22T01:16:59.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.166+0000 7f45c5060640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45a40678f0 0x7f45a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.166+0000 7f45c5060640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45a40678f0 0x7f45a4069db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f45c01fa710 tx=0x7f45b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.170+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f45bc0c6020 con 0x7f45ac0039c0 2024-04-22T01:16:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.346+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45c0162eb0 con 0x7f45ac0039c0 2024-04-22T01:16:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45b67fc640 1 -- 172.21.15.70:0/3726214797 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5178 (secure 0 0 0) 0x7f45bc0167c0 con 0x7f45ac0039c0 2024-04-22T01:16:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:59.349 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:41.643333+0000","modified":"2024-04-22T01:16:44.689368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5922},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5922":{"gid":5922,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2188446179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2188446179},{"type":"v1","addr":"172.21.15.70:6837","nonce":2188446179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5922,"qdb_cluster":[5922]},"id":3}]} 2024-04-22T01:16:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 25 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45a40678f0 msgr2=0x7f45a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45a40678f0 0x7f45a4069db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f45c01fa710 tx=0x7f45b0008040 comp rx=0 tx=0).stop 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 msgr2=0x7f45c015a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45c015a980 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f45bc01c8b0 tx=0x7f45ac005840 comp rx=0 tx=0).stop 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 shutdown_connections 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f45a40678f0 0x7f45a4069db0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00b5db0 0x7f45c00cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c015af90 0x7f45c0160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 --2- 172.21.15.70:0/3726214797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ac0039c0 0x7f45c015a980 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 >> 172.21.15.70:0/3726214797 conn(0x7f45c00b35d0 msgr2=0x7f45c0167c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 shutdown_connections 2024-04-22T01:16:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.350+0000 7f45c7ac5640 1 -- 172.21.15.70:0/3726214797 wait complete. 2024-04-22T01:16:59.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:59.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86aa89c640 1 Processor -- start 2024-04-22T01:16:59.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86aa89c640 1 -- start start 2024-04-22T01:16:59.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86aa89c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86a40b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86aa89c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a406dbd0 con 0x7f86a40aee40 2024-04-22T01:16:59.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86aa89c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a406de70 con 0x7f86a415bb50 2024-04-22T01:16:59.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86aa89c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f86a406e150 con 0x7f86a415aa30 2024-04-22T01:16:59.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a8e39640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f86a40aee40 0x7f86a40b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34090/0 (socket says 172.21.15.70:34090) 2024-04-22T01:16:59.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a8e39640 1 -- 172.21.15.70:0/3046184948 learned_addr learned my addr 172.21.15.70:0/3046184948 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86a40b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015768527 0 0) 0x7f86a406dbd0 con 0x7f86a40aee40 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8690003530 con 0x7f86a40aee40 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266663495 0 0) 0x7f8690003530 con 0x7f86a40aee40 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 msgr2=0x7f86a40b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86a40b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 >> v1:172.21.15.98:6789/0 conn(0x7f86a415bb50 legacy=0x7f86a40ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a406e430 con 0x7f86a40aee40 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f869c001a00 con 0x7f86a40aee40 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 >> v1:172.21.15.70:6789/0 conn(0x7f86a40aee40 legacy=0x7f86a40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86900039f0 0x7f8690003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8690004810 0x7f8690004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86900051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86a406e150 con 0x7f86900039f0 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f869000ba80 con 0x7f8690004810 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f869000de90 con 0x7f86a415aa30 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86900039f0 0x7f8690003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f869bfff640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8690004810 0x7f8690004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a8e39640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86900051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 -- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 msgr2=0x7f86900051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86900051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 -- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8690004810 msgr2=0x7f8690004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8690004810 0x7f8690004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a3fff640 1 -- 172.21.15.70:0/3046184948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8690003530 con 0x7f86900039f0 2024-04-22T01:16:59.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.474+0000 7f86a8e39640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a415aa30 0x7f86900051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86900039f0 0x7f8690003e10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8694009ab0 tx=0x7f869401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18bd293672c52848 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86940b2070 con 0x7f86900039f0 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86940a5520 con 0x7f86900039f0 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a37fe640 1 -- 172.21.15.70:0/3046184948 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86940096b0 con 0x7f86900039f0 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- 172.21.15.70:0/3046184948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86900039f0 msgr2=0x7f8690003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3046184948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86900039f0 0x7f8690003e10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8694009ab0 tx=0x7f869401c9e0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 -- 172.21.15.70:0/3046184948 reap_dead start 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- 172.21.15.70:0/3046184948 shutdown_connections 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- 172.21.15.70:0/3046184948 >> 172.21.15.70:0/3046184948 conn(0x7f86a406b1f0 msgr2=0x7f86a406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- 172.21.15.70:0/3046184948 shutdown_connections 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- 172.21.15.70:0/3046184948 wait complete. 2024-04-22T01:16:59.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 Processor -- start 2024-04-22T01:16:59.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- start start 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86900039f0 0x7f86a4142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a4145890 0x7f86a40b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 0x7f86a40a8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86a406ead0 con 0x7f86a40a84c0 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86a406e950 con 0x7f86a4145890 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86aa89c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f86a4071b20 con 0x7f86900039f0 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 0x7f86a40a8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 0x7f86a40a8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45942/0 (socket says 172.21.15.70:45942) 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 -- 172.21.15.70:0/3571526635 learned_addr learned my addr 172.21.15.70:0/3571526635 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f869bfff640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a4145890 0x7f86a40b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86900039f0 0x7f86a4142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 -- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86900039f0 msgr2=0x7f86a4142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86900039f0 0x7f86a4142230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 -- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a4145890 msgr2=0x7f86a40b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a4145890 0x7f86a40b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f86a8e39640 1 -- 172.21.15.70:0/3571526635 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a40b0f20 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.478+0000 7f869bfff640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a4145890 0x7f86a40b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a8e39640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 0x7f86a40a8970 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f86a4072ea0 tx=0x7f869c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c00bb80 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869c00bd40 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a40b1200 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c0072b0 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a40b17c0 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a4068210 con 0x7f86a40a84c0 2024-04-22T01:16:59.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f869c01b020 con 0x7f86a40a84c0 2024-04-22T01:16:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a17fa640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86840678f0 0x7f8684069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f869c0860b0 con 0x7f86a40a84c0 2024-04-22T01:16:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86840678f0 0x7f8684069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.482+0000 7f86a3fff640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86840678f0 0x7f8684069db0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f86940af000 tx=0x7f86940046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.486+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f869c0569e0 con 0x7f86a40a84c0 2024-04-22T01:16:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.634+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86a40aec20 con 0x7f86a40a84c0 2024-04-22T01:16:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.634+0000 7f86a17fa640 1 -- 172.21.15.70:0/3571526635 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f869c0562e0 con 0x7f86a40a84c0 2024-04-22T01:16:59.637 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:59.637 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:43.266695+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86840678f0 msgr2=0x7f8684069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86840678f0 0x7f8684069db0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f86940af000 tx=0x7f86940046d0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 msgr2=0x7f86a40a8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 0x7f86a40a8970 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f86a4072ea0 tx=0x7f869c00b4a0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 shutdown_connections 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86840678f0 0x7f8684069db0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40a84c0 0x7f86a40a8970 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a4145890 0x7f86a40b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 --2- 172.21.15.70:0/3571526635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86900039f0 0x7f86a4142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 >> 172.21.15.70:0/3571526635 conn(0x7f86a406b1f0 msgr2=0x7f86a4071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 shutdown_connections 2024-04-22T01:16:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.638+0000 7f86aa89c640 1 -- 172.21.15.70:0/3571526635 wait complete. 2024-04-22T01:16:59.655 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:16:59.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:59.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.762+0000 7f05ad6a6640 1 Processor -- start 2024-04-22T01:16:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.762+0000 7f05ad6a6640 1 -- start start 2024-04-22T01:16:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05ad6a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05ad6a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05a80d2110 0x7f05a80d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05ad6a6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a815e390 con 0x7f05a815dfb0 2024-04-22T01:16:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05ad6a6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f05a80cf420 con 0x7f05a80d2110 2024-04-22T01:16:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05ad6a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05a8161e00 con 0x7f05a80cea80 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05a80d2110 0x7f05a80d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a80ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:56810/0 (socket says 172.21.15.70:56810) 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05a80d2110 0x7f05a80d2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:56218/0 (socket says 172.21.15.70:56218) 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a67fc640 1 -- 172.21.15.70:0/1266984739 learned_addr learned my addr 172.21.15.70:0/1266984739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a6ffd640 1 --1- 172.21.15.70:0/1266984739 >> v1:172.21.15.98:6789/0 conn(0x7f05a815dfb0 0x7f05a80b95d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:59652/0 (socket says 172.21.15.70:59652) 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a77fe640 1 -- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 msgr2=0x7f05a80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a77fe640 1 --2- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a77fe640 1 -- 172.21.15.70:0/1266984739 >> v1:172.21.15.98:6789/0 conn(0x7f05a815dfb0 legacy=0x7f05a80b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.766+0000 7f05a77fe640 1 -- 172.21.15.70:0/1266984739 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a81620e0 con 0x7f05a80d2110 2024-04-22T01:16:59.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a77fe640 1 --2- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80d2110 0x7f05a80d2570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009d90 tx=0x7f05a001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c1dbce9b86ffe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a5ffb640 1 -- 172.21.15.70:0/1266984739 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a00b2070 con 0x7f05a80d2110 2024-04-22T01:16:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a5ffb640 1 -- 172.21.15.70:0/1266984739 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a00a5940 con 0x7f05a80d2110 2024-04-22T01:16:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a5ffb640 1 -- 172.21.15.70:0/1266984739 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a00a5d10 con 0x7f05a80d2110 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80d2110 msgr2=0x7f05a80d2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80d2110 0x7f05a80d2570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009d90 tx=0x7f05a001c970 comp rx=0 tx=0).stop 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- 172.21.15.70:0/1266984739 shutdown_connections 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80d2110 0x7f05a80d2570 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/1266984739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- 172.21.15.70:0/1266984739 >> 172.21.15.70:0/1266984739 conn(0x7f05a806c9b0 msgr2=0x7f05a806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- 172.21.15.70:0/1266984739 shutdown_connections 2024-04-22T01:16:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- 172.21.15.70:0/1266984739 wait complete. 2024-04-22T01:16:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 Processor -- start 2024-04-22T01:16:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- start start 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a81605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 0x7f05a815d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a815dfb0 0x7f05a815bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05a816af50 con 0x7f05a80d2110 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05a816add0 con 0x7f05a815dfb0 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05ad6a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05a816b0d0 con 0x7f05a80cea80 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 0x7f05a815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 0x7f05a815d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45950/0 (socket says 172.21.15.70:45950) 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 -- 172.21.15.70:0/370826963 learned_addr learned my addr 172.21.15.70:0/370826963 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a6ffd640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a81605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a77fe640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a815dfb0 0x7f05a815bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 -- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 msgr2=0x7f05a81605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a81605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 -- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a815dfb0 msgr2=0x7f05a815bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a815dfb0 0x7f05a815bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 -- 172.21.15.70:0/370826963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a815c2d0 con 0x7f05a80d2110 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a6ffd640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a81605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a77fe640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a815dfb0 0x7f05a815bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f05a67fc640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 0x7f05a815d400 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f059c00e7f0 tx=0x7f059c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.770+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c017070 con 0x7f05a80d2110 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059c00c940 con 0x7f05a80d2110 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c00cd30 con 0x7f05a80d2110 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a8160c80 con 0x7f05a80d2110 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a815b2b0 con 0x7f05a80d2110 2024-04-22T01:16:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a8068210 con 0x7f05a80d2110 2024-04-22T01:16:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f059c01a070 con 0x7f05a80d2110 2024-04-22T01:16:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f058f7fe640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05940678c0 0x7f0594069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f059c085db0 con 0x7f05a80d2110 2024-04-22T01:16:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f05a6ffd640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05940678c0 0x7f0594069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.774+0000 7f05a6ffd640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05940678c0 0x7f0594069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0598007fd0 tx=0x7f0598009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:59.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.778+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f059c0566e0 con 0x7f05a80d2110 2024-04-22T01:16:59.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05a80d1950 con 0x7f05a80d2110 2024-04-22T01:16:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f058f7fe640 1 -- 172.21.15.70:0/370826963 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5178 (secure 0 0 0) 0x7f059c055fe0 con 0x7f05a80d2110 2024-04-22T01:16:59.958 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:16:59.958 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:41.643333+0000","modified":"2024-04-22T01:16:44.689368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5922},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5922":{"gid":5922,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2188446179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2188446179},{"type":"v1","addr":"172.21.15.70:6837","nonce":2188446179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5922,"qdb_cluster":[5922]},"id":3}]} 2024-04-22T01:16:59.958 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 25 2024-04-22T01:16:59.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05940678c0 msgr2=0x7f0594069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05940678c0 0x7f0594069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0598007fd0 tx=0x7f0598009040 comp rx=0 tx=0).stop 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 msgr2=0x7f05a815d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 0x7f05a815d400 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f059c00e7f0 tx=0x7f059c00ecb0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 shutdown_connections 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05940678c0 0x7f0594069d80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a815dfb0 0x7f05a815bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a80d2110 0x7f05a815d400 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 --2- 172.21.15.70:0/370826963 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a80cea80 0x7f05a81605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 >> 172.21.15.70:0/370826963 conn(0x7f05a806c9b0 msgr2=0x7f05a8156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 shutdown_connections 2024-04-22T01:16:59.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:16:59.958+0000 7f05ad6a6640 1 -- 172.21.15.70:0/370826963 wait complete. 2024-04-22T01:16:59.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.086+0000 7f6d1e866640 1 Processor -- start 2024-04-22T01:17:00.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1e866640 1 -- start start 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1e866640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d180cc150 0x7f6d18071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1e866640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d18072060 0x7f6d1806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1e866640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d180cd7b0 con 0x7f6d181455b0 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1e866640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6d180724d0 con 0x7f6d180cc150 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1e866640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6d180cc710 con 0x7f6d18072060 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d17fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6d181455b0 0x7f6d180cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34106/0 (socket says 172.21.15.70:34106) 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d17fff640 1 -- 172.21.15.70:0/3717201363 learned_addr learned my addr 172.21.15.70:0/3717201363 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1ce03640 1 --2- 172.21.15.70:0/3717201363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d18072060 0x7f6d1806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d177fe640 1 --2- 172.21.15.70:0/3717201363 >> v2:172.21.15.98:3300/0 conn(0x7f6d180cc150 0x7f6d18071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d16ffd640 1 -- 172.21.15.70:0/3717201363 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847983370 0 0) 0x7f6d180cd7b0 con 0x7f6d181455b0 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d16ffd640 1 -- 172.21.15.70:0/3717201363 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d00003530 con 0x7f6d181455b0 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1ce03640 1 -- 172.21.15.70:0/3717201363 >> v2:172.21.15.98:3300/0 conn(0x7f6d180cc150 msgr2=0x7f6d18071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1ce03640 1 --2- 172.21.15.70:0/3717201363 >> v2:172.21.15.98:3300/0 conn(0x7f6d180cc150 0x7f6d18071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1ce03640 1 -- 172.21.15.70:0/3717201363 >> v1:172.21.15.70:6789/0 conn(0x7f6d181455b0 legacy=0x7f6d180cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d1ce03640 1 -- 172.21.15.70:0/3717201363 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d180cc9f0 con 0x7f6d18072060 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.090+0000 7f6d177fe640 1 --2- 172.21.15.70:0/3717201363 >> v2:172.21.15.98:3300/0 conn(0x7f6d180cc150 0x7f6d18071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1ce03640 1 --2- 172.21.15.70:0/3717201363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d18072060 0x7f6d1806e200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009d90 tx=0x7f6d1001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33d30c1ffa7194e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d16ffd640 1 -- 172.21.15.70:0/3717201363 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d100b2070 con 0x7f6d18072060 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d16ffd640 1 -- 172.21.15.70:0/3717201363 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d100a5710 con 0x7f6d18072060 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d16ffd640 1 -- 172.21.15.70:0/3717201363 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d100a5ae0 con 0x7f6d18072060 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- 172.21.15.70:0/3717201363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d18072060 msgr2=0x7f6d1806e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 --2- 172.21.15.70:0/3717201363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d18072060 0x7f6d1806e200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009d90 tx=0x7f6d1001c850 comp rx=0 tx=0).stop 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- 172.21.15.70:0/3717201363 shutdown_connections 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 --2- 172.21.15.70:0/3717201363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d18072060 0x7f6d1806e200 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 --2- 172.21.15.70:0/3717201363 >> v2:172.21.15.98:3300/0 conn(0x7f6d180cc150 0x7f6d18071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- 172.21.15.70:0/3717201363 >> 172.21.15.70:0/3717201363 conn(0x7f6d1806b1f0 msgr2=0x7f6d1806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- 172.21.15.70:0/3717201363 shutdown_connections 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- 172.21.15.70:0/3717201363 wait complete. 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 Processor -- start 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- start start 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d18072060 0x7f6d18140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 0x7f6d18159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d181455b0 0x7f6d18165b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1806f320 con 0x7f6d180cc150 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1806f1a0 con 0x7f6d18072060 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d1e866640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6d18168b40 con 0x7f6d181455b0 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 0x7f6d18159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 0x7f6d18159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45962/0 (socket says 172.21.15.70:45962) 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d177fe640 1 -- 172.21.15.70:0/1013522756 learned_addr learned my addr 172.21.15.70:0/1013522756 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.094+0000 7f6d17fff640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d18072060 0x7f6d18140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d1ce03640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d181455b0 0x7f6d18165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d177fe640 1 -- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d181455b0 msgr2=0x7f6d18165b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d177fe640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d181455b0 0x7f6d18165b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d177fe640 1 -- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d18072060 msgr2=0x7f6d18140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d177fe640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d18072060 0x7f6d18140620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d177fe640 1 -- 172.21.15.70:0/1013522756 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d18166570 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d177fe640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 0x7f6d18159460 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00e980 tx=0x7f6d0c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c017070 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d0c00c9a0 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d180b2a90 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c00cd70 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d180b3050 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d18068210 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d0c01a070 con 0x7f6d180cc150 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d14ff9640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d040678c0 0x7f6d04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f6d0c086060 con 0x7f6d180cc150 2024-04-22T01:17:00.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d17fff640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d040678c0 0x7f6d04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.098+0000 7f6d17fff640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d040678c0 0x7f6d04069d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005890 tx=0x7f6d08009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.102+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6d0c056a10 con 0x7f6d180cc150 2024-04-22T01:17:00.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.298+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d1815f510 con 0x7f6d180cc150 2024-04-22T01:17:00.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.298+0000 7f6d14ff9640 1 -- 172.21.15.70:0/1013522756 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5178 (secure 0 0 0) 0x7f6d0c056310 con 0x7f6d180cc150 2024-04-22T01:17:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:00.300 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:16:41.643333+0000","modified":"2024-04-22T01:16:44.689368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5922},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5922":{"gid":5922,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2188446179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2188446179},{"type":"v1","addr":"172.21.15.70:6837","nonce":2188446179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5922,"qdb_cluster":[5922]},"id":3}]} 2024-04-22T01:17:00.300 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 25 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d040678c0 msgr2=0x7f6d04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d040678c0 0x7f6d04069d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005890 tx=0x7f6d08009040 comp rx=0 tx=0).stop 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 msgr2=0x7f6d18159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 0x7f6d18159460 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00e980 tx=0x7f6d0c00ee40 comp rx=0 tx=0).stop 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 shutdown_connections 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6d040678c0 0x7f6d04069d80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d181455b0 0x7f6d18165b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d180cc150 0x7f6d18159460 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 --2- 172.21.15.70:0/1013522756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d18072060 0x7f6d18140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 >> 172.21.15.70:0/1013522756 conn(0x7f6d1806b1f0 msgr2=0x7f6d1813fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:00.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 shutdown_connections 2024-04-22T01:17:00.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.302+0000 7f6d1e866640 1 -- 172.21.15.70:0/1013522756 wait complete. 2024-04-22T01:17:00.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 Processor -- start 2024-04-22T01:17:00.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 -- start start 2024-04-22T01:17:00.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f148415be40 0x7f1484099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14840ad880 0x7f148415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f148415c220 con 0x7f1484066f50 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f148415c570 con 0x7f148415be40 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1489025640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f14840674a0 con 0x7f14840ad880 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f148415be40 0x7f1484099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f148415be40 0x7f1484099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45972/0 (socket says 172.21.15.70:45972) 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 -- 172.21.15.70:0/1106852959 learned_addr learned my addr 172.21.15.70:0/1106852959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f148359e640 1 --2- 172.21.15.70:0/1106852959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14840ad880 0x7f148415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 -- 172.21.15.70:0/1106852959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14840ad880 msgr2=0x7f148415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 --2- 172.21.15.70:0/1106852959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14840ad880 0x7f148415ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 -- 172.21.15.70:0/1106852959 >> v1:172.21.15.98:6789/0 conn(0x7f1484066f50 legacy=0x7f14840ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.438+0000 7f1482d9d640 1 -- 172.21.15.70:0/1106852959 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148406c4b0 con 0x7f148415be40 2024-04-22T01:17:00.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1482d9d640 1 --2- 172.21.15.70:0/1106852959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484099b80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1474009870 tx=0x7f147401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57ec83ed720e692 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f148259c640 1 -- 172.21.15.70:0/1106852959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740a95f0 con 0x7f148415be40 2024-04-22T01:17:00.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f148259c640 1 -- 172.21.15.70:0/1106852959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14740a9790 con 0x7f148415be40 2024-04-22T01:17:00.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f148259c640 1 -- 172.21.15.70:0/1106852959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740a9b60 con 0x7f148415be40 2024-04-22T01:17:00.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 -- 172.21.15.70:0/1106852959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 msgr2=0x7f1484099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 --2- 172.21.15.70:0/1106852959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484099b80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1474009870 tx=0x7f147401c530 comp rx=0 tx=0).stop 2024-04-22T01:17:00.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 -- 172.21.15.70:0/1106852959 shutdown_connections 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 --2- 172.21.15.70:0/1106852959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14840ad880 0x7f148415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 --2- 172.21.15.70:0/1106852959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484099b80 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 -- 172.21.15.70:0/1106852959 >> 172.21.15.70:0/1106852959 conn(0x7f14840b3d90 msgr2=0x7f14840b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 -- 172.21.15.70:0/1106852959 shutdown_connections 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 -- 172.21.15.70:0/1106852959 wait complete. 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 Processor -- start 2024-04-22T01:17:00.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.442+0000 7f1489025640 1 -- start start 2024-04-22T01:17:00.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1484066f50 0x7f1484153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14840ad880 0x7f1484136890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f148415d8d0 con 0x7f148415be40 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f148415d750 con 0x7f14840ad880 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f148415da50 con 0x7f1484066f50 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45984/0 (socket says 172.21.15.70:45984) 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 -- 172.21.15.70:0/2850731702 learned_addr learned my addr 172.21.15.70:0/2850731702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147bfff640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14840ad880 0x7f1484136890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1482d9d640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1484066f50 0x7f1484153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 -- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1484066f50 msgr2=0x7f1484153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1484066f50 0x7f1484153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 -- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14840ad880 msgr2=0x7f1484136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14840ad880 0x7f1484136890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 -- 172.21.15.70:0/2850731702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1484154d10 con 0x7f148415be40 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f148359e640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484136dd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f147c007de0 tx=0x7f147c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c014620 con 0x7f148415be40 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147c019070 con 0x7f148415be40 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1484134990 con 0x7f148415be40 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c014a20 con 0x7f148415be40 2024-04-22T01:17:00.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1484154750 con 0x7f148415be40 2024-04-22T01:17:00.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1484068920 con 0x7f148415be40 2024-04-22T01:17:00.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f147c014bc0 con 0x7f148415be40 2024-04-22T01:17:00.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147b7fe640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14640678f0 0x7f1464069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1482d9d640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14640678f0 0x7f1464069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f147c0866b0 con 0x7f148415be40 2024-04-22T01:17:00.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.446+0000 7f1482d9d640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14640678f0 0x7f1464069db0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f14740ae000 tx=0x7f14740b3040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.450+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f147c0570f0 con 0x7f148415be40 2024-04-22T01:17:00.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.614+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1484152410 con 0x7f148415be40 2024-04-22T01:17:00.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.618+0000 7f147b7fe640 1 -- 172.21.15.70:0/2850731702 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f147c0569f0 con 0x7f148415be40 2024-04-22T01:17:00.619 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:00.619 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:16:43.266695+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:16:39.917376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-22T01:16:40.609582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14640678f0 msgr2=0x7f1464069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14640678f0 0x7f1464069db0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f14740ae000 tx=0x7f14740b3040 comp rx=0 tx=0).stop 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 msgr2=0x7f1484136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484136dd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f147c007de0 tx=0x7f147c00e4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 shutdown_connections 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14640678f0 0x7f1464069db0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f148415be40 0x7f1484136dd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14840ad880 0x7f1484136890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 --2- 172.21.15.70:0/2850731702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1484066f50 0x7f1484153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 >> 172.21.15.70:0/2850731702 conn(0x7f14840b3d90 msgr2=0x7f1484152930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:00.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 shutdown_connections 2024-04-22T01:17:00.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.622+0000 7f1489025640 1 -- 172.21.15.70:0/2850731702 wait complete. 2024-04-22T01:17:00.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 Processor -- start 2024-04-22T01:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 -- start start 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd8dc099cc0 0x7fd8dc09a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8dc0ccb50 0x7fd8dc0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8dc06d850 con 0x7fd8dc15e160 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd8dc15e540 con 0x7fd8dc099cc0 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8e09c5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd8dc0cc6c0 con 0x7fd8dc0ccb50 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8da59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd8dc15e160 0x7fd8dc09ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34118/0 (socket says 172.21.15.70:34118) 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8da59c640 1 -- 172.21.15.70:0/984008837 learned_addr learned my addr 172.21.15.70:0/984008837 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8d9d9b640 1 --2- 172.21.15.70:0/984008837 >> v2:172.21.15.70:3300/0 conn(0x7fd8dc099cc0 0x7fd8dc09a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8dad9d640 1 --2- 172.21.15.70:0/984008837 >> v2:172.21.15.98:3300/0 conn(0x7fd8dc0ccb50 0x7fd8dc0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8d9d9b640 1 -- 172.21.15.70:0/984008837 >> v2:172.21.15.98:3300/0 conn(0x7fd8dc0ccb50 msgr2=0x7fd8dc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8d9d9b640 1 --2- 172.21.15.70:0/984008837 >> v2:172.21.15.98:3300/0 conn(0x7fd8dc0ccb50 0x7fd8dc0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8d9d9b640 1 -- 172.21.15.70:0/984008837 >> v1:172.21.15.70:6789/0 conn(0x7fd8dc15e160 legacy=0x7fd8dc09ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8d9d9b640 1 -- 172.21.15.70:0/984008837 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8dc09b250 con 0x7fd8dc099cc0 2024-04-22T01:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.762+0000 7fd8d959a640 1 -- 172.21.15.70:0/984008837 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 62009816 0 0) 0x7fd8dc06d850 con 0x7fd8dc15e160 2024-04-22T01:17:00.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8d9d9b640 1 --2- 172.21.15.70:0/984008837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc09a100 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0d14d0 tx=0x7fd8c401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1440a0f4ed43e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8d959a640 1 -- 172.21.15.70:0/984008837 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c40b1070 con 0x7fd8dc099cc0 2024-04-22T01:17:00.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8d959a640 1 -- 172.21.15.70:0/984008837 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c40ac5f0 con 0x7fd8dc099cc0 2024-04-22T01:17:00.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8d959a640 1 -- 172.21.15.70:0/984008837 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c40ac9c0 con 0x7fd8dc099cc0 2024-04-22T01:17:00.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- 172.21.15.70:0/984008837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc099cc0 msgr2=0x7fd8dc09a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/984008837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc09a100 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0d14d0 tx=0x7fd8c401c480 comp rx=0 tx=0).stop 2024-04-22T01:17:00.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- 172.21.15.70:0/984008837 shutdown_connections 2024-04-22T01:17:00.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/984008837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc09a100 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/984008837 >> v2:172.21.15.98:3300/0 conn(0x7fd8dc0ccb50 0x7fd8dc0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- 172.21.15.70:0/984008837 >> 172.21.15.70:0/984008837 conn(0x7fd8dc0b3690 msgr2=0x7fd8dc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:00.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- 172.21.15.70:0/984008837 shutdown_connections 2024-04-22T01:17:00.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- 172.21.15.70:0/984008837 wait complete. 2024-04-22T01:17:00.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 Processor -- start 2024-04-22T01:17:00.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- start start 2024-04-22T01:17:00.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc15ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8dc0ccb50 0x7fd8dc1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 0x7fd8dc150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8dc0d0a70 con 0x7fd8dc15e160 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8dc0d08f0 con 0x7fd8dc099cc0 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8e09c5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd8dc16ac40 con 0x7fd8dc0ccb50 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 0x7fd8dc150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 0x7fd8dc150a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46004/0 (socket says 172.21.15.70:46004) 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8dad9d640 1 -- 172.21.15.70:0/3878484615 learned_addr learned my addr 172.21.15.70:0/3878484615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.766+0000 7fd8d9d9b640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8dc0ccb50 0x7fd8dc1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8da59c640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc15ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8dad9d640 1 -- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8dc0ccb50 msgr2=0x7fd8dc1504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8dad9d640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8dc0ccb50 0x7fd8dc1504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8dad9d640 1 -- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8dc099cc0 msgr2=0x7fd8dc15ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8dad9d640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc15ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8dad9d640 1 -- 172.21.15.70:0/3878484615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8dc151fd0 con 0x7fd8dc15e160 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8dad9d640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 0x7fd8dc150a30 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00d830 tx=0x7fd8cc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00b6c0 con 0x7fd8dc15e160 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8cc019070 con 0x7fd8dc15e160 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8dc15b510 con 0x7fd8dc15e160 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00baa0 con 0x7fd8dc15e160 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8dc1561b0 con 0x7fd8dc15e160 2024-04-22T01:17:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8dc1521a0 con 0x7fd8dc15e160 2024-04-22T01:17:00.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd8cc01e460 con 0x7fd8dc15e160 2024-04-22T01:17:00.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8d2ffd640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8bc0678c0 0x7fd8bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:00.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fd8cc0864d0 con 0x7fd8dc15e160 2024-04-22T01:17:00.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8da59c640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8bc0678c0 0x7fd8bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:00.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.770+0000 7fd8da59c640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8bc0678c0 0x7fd8bc069d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4005fd0 tx=0x7fd8d4005b80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:00.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.774+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd8cc056f10 con 0x7fd8dc15e160 2024-04-22T01:17:00.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.962+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd8dc0c5480 con 0x7fd8dc15e160 2024-04-22T01:17:00.962 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:17:00.962+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:17:00.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.962+0000 7fd8d2ffd640 1 -- 172.21.15.70:0/3878484615 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7fd8cc056810 con 0x7fd8dc15e160 2024-04-22T01:17:00.965 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8bc0678c0 msgr2=0x7fd8bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8bc0678c0 0x7fd8bc069d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4005fd0 tx=0x7fd8d4005b80 comp rx=0 tx=0).stop 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 msgr2=0x7fd8dc150a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 0x7fd8dc150a30 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00d830 tx=0x7fd8cc00dcf0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 shutdown_connections 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd8bc0678c0 0x7fd8bc069d80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8dc15e160 0x7fd8dc150a30 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8dc0ccb50 0x7fd8dc1504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 --2- 172.21.15.70:0/3878484615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8dc099cc0 0x7fd8dc15ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:00.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 >> 172.21.15.70:0/3878484615 conn(0x7fd8dc0b3690 msgr2=0x7fd8dc09a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:00.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 shutdown_connections 2024-04-22T01:17:00.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:00.966+0000 7fd8e09c5640 1 -- 172.21.15.70:0/3878484615 wait complete. 2024-04-22T01:17:00.971 INFO:tasks.ceph.mds.a.smithi070.stderr: -1558> 2024-04-22T01:16:47.910+0000 7f831935f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:17:00.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:17:01.014 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:17:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba5241a640 1 Processor -- start 2024-04-22T01:17:01.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba5241a640 1 -- start start 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba5241a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fba4c15a360 0x7fba4c0d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba5241a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4c0b54d0 con 0x7fba4c06c200 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba5241a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4c0d0c20 con 0x7fba4c15a740 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba5241a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fba4c0b0880 con 0x7fba4c15a360 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba4bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fba4c15a360 0x7fba4c0d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba509b7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fba4c06c200 0x7fba4c0aea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34128/0 (socket says 172.21.15.70:34128) 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba4bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fba4c15a360 0x7fba4c0d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46040/0 (socket says 172.21.15.70:46040) 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba509b7640 1 -- 172.21.15.70:0/432547744 learned_addr learned my addr 172.21.15.70:0/432547744 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba4bfff640 1 -- 172.21.15.70:0/432547744 >> v1:172.21.15.98:6789/0 conn(0x7fba4c15a740 legacy=0x7fba4c0ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba4bfff640 1 -- 172.21.15.70:0/432547744 >> v1:172.21.15.70:6789/0 conn(0x7fba4c06c200 legacy=0x7fba4c0aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:01.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba4affd640 1 -- 172.21.15.70:0/432547744 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704322845 0 0) 0x7fba4c0b54d0 con 0x7fba4c06c200 2024-04-22T01:17:01.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.102+0000 7fba4bfff640 1 -- 172.21.15.70:0/432547744 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c0d0f00 con 0x7fba4c15a360 2024-04-22T01:17:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4bfff640 1 --2- 172.21.15.70:0/432547744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c15a360 0x7fba4c0d0050 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fba340097c0 tx=0x7fba3401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1861212ec03b5c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4affd640 1 -- 172.21.15.70:0/432547744 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba340b5070 con 0x7fba4c15a360 2024-04-22T01:17:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4affd640 1 -- 172.21.15.70:0/432547744 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba340a8950 con 0x7fba4c15a360 2024-04-22T01:17:01.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4affd640 1 -- 172.21.15.70:0/432547744 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba340a8d20 con 0x7fba4c15a360 2024-04-22T01:17:01.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- 172.21.15.70:0/432547744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c15a360 msgr2=0x7fba4c0d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:01.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 --2- 172.21.15.70:0/432547744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c15a360 0x7fba4c0d0050 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fba340097c0 tx=0x7fba3401ec50 comp rx=0 tx=0).stop 2024-04-22T01:17:01.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- 172.21.15.70:0/432547744 shutdown_connections 2024-04-22T01:17:01.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 --2- 172.21.15.70:0/432547744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c15a360 0x7fba4c0d0050 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- 172.21.15.70:0/432547744 >> 172.21.15.70:0/432547744 conn(0x7fba4c099a60 msgr2=0x7fba4c0b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:01.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- 172.21.15.70:0/432547744 shutdown_connections 2024-04-22T01:17:01.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- 172.21.15.70:0/432547744 wait complete. 2024-04-22T01:17:01.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 Processor -- start 2024-04-22T01:17:01.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- start start 2024-04-22T01:17:01.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 0x7fba4c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:01.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c15a360 0x7fba4c141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:01.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba4c15a740 0x7fba4c152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:01.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c06c630 con 0x7fba4c06c200 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c0b0b60 con 0x7fba4c15a360 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba5241a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fba4c06ca10 con 0x7fba4c15a740 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 0x7fba4c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 0x7fba4c0ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46046/0 (socket says 172.21.15.70:46046) 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.106+0000 7fba4bfff640 1 -- 172.21.15.70:0/1485689619 learned_addr learned my addr 172.21.15.70:0/1485689619 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba509b7640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba4c15a740 0x7fba4c152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4b7fe640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c15a360 0x7fba4c141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4bfff640 1 -- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba4c15a740 msgr2=0x7fba4c152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4bfff640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba4c15a740 0x7fba4c152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4bfff640 1 -- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c15a360 msgr2=0x7fba4c141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4bfff640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c15a360 0x7fba4c141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4bfff640 1 -- 172.21.15.70:0/1485689619 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c0ae110 con 0x7fba4c06c200 2024-04-22T01:17:01.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4bfff640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 0x7fba4c0ad8e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fba3401b860 tx=0x7fba3401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:01.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba340b5040 con 0x7fba4c06c200 2024-04-22T01:17:01.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba340bf070 con 0x7fba4c06c200 2024-04-22T01:17:01.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba340b0a80 con 0x7fba4c06c200 2024-04-22T01:17:01.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c0ae3f0 con 0x7fba4c06c200 2024-04-22T01:17:01.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c0ae9b0 con 0x7fba4c06c200 2024-04-22T01:17:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4c068920 con 0x7fba4c06c200 2024-04-22T01:17:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fba340b0c20 con 0x7fba4c06c200 2024-04-22T01:17:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba48ff9640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fba300678f0 0x7fba30069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6187+0+0 (secure 0 0 0) 0x7fba34123490 con 0x7fba4c06c200 2024-04-22T01:17:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4b7fe640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fba300678f0 0x7fba30069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.110+0000 7fba4b7fe640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fba300678f0 0x7fba30069db0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fba4c1f73d0 tx=0x7fba44009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:01.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.114+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fba340187b0 con 0x7fba4c06c200 2024-04-22T01:17:01.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.298+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fba4c14ff20 con 0x7fba4c06c200 2024-04-22T01:17:01.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.966+0000 7fba48ff9640 1 -- 172.21.15.70:0/1485689619 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fba340af3a0 con 0x7fba4c06c200 2024-04-22T01:17:01.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fba300678f0 msgr2=0x7fba30069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:01.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fba300678f0 0x7fba30069db0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fba4c1f73d0 tx=0x7fba44009040 comp rx=0 tx=0).stop 2024-04-22T01:17:01.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 msgr2=0x7fba4c0ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 0x7fba4c0ad8e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fba3401b860 tx=0x7fba3401b9f0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 shutdown_connections 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fba300678f0 0x7fba30069db0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba4c15a740 0x7fba4c152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c15a360 0x7fba4c141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 --2- 172.21.15.70:0/1485689619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba4c06c200 0x7fba4c0ad8e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 >> 172.21.15.70:0/1485689619 conn(0x7fba4c099a60 msgr2=0x7fba4c1530c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 shutdown_connections 2024-04-22T01:17:01.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:01.970+0000 7fba5241a640 1 -- 172.21.15.70:0/1485689619 wait complete. 2024-04-22T01:17:01.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:17:02.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 Processor -- start 2024-04-22T01:17:02.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 -- start start 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff6440ce920 0x7ff644154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff6440cedd0 0x7ff6441551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6440b3800 con 0x7ff64415e6e0 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff6440b3ae0 con 0x7ff6440ce920 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff64ac11640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff6440cde40 con 0x7ff6440cedd0 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff6440ce920 0x7ff644154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff6440ce920 0x7ff644154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46058/0 (socket says 172.21.15.70:46058) 2024-04-22T01:17:02.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 -- 172.21.15.70:0/1817643777 learned_addr learned my addr 172.21.15.70:0/1817643777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:02.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff6437fe640 1 -- 172.21.15.70:0/1817643777 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183887382 0 0) 0x7ff6440b3800 con 0x7ff64415e6e0 2024-04-22T01:17:02.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff6437fe640 1 -- 172.21.15.70:0/1817643777 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff62c003530 con 0x7ff64415e6e0 2024-04-22T01:17:02.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff6491ae640 1 --2- 172.21.15.70:0/1817643777 >> v2:172.21.15.98:3300/0 conn(0x7ff6440cedd0 0x7ff6441551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:02.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 -- 172.21.15.70:0/1817643777 >> v2:172.21.15.98:3300/0 conn(0x7ff6440cedd0 msgr2=0x7ff6441551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:02.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 --2- 172.21.15.70:0/1817643777 >> v2:172.21.15.98:3300/0 conn(0x7ff6440cedd0 0x7ff6441551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 -- 172.21.15.70:0/1817643777 >> v1:172.21.15.70:6789/0 conn(0x7ff64415e6e0 legacy=0x7ff64415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:02.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.158+0000 7ff643fff640 1 -- 172.21.15.70:0/1817643777 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6440ce120 con 0x7ff6440ce920 2024-04-22T01:17:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff643fff640 1 --2- 172.21.15.70:0/1817643777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440ce920 0x7ff644154c60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff644158cb0 tx=0x7ff63801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45efe1c1f807df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff6437fe640 1 -- 172.21.15.70:0/1817643777 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6380aa670 con 0x7ff6440ce920 2024-04-22T01:17:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff6437fe640 1 -- 172.21.15.70:0/1817643777 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6380af070 con 0x7ff6440ce920 2024-04-22T01:17:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff6437fe640 1 -- 172.21.15.70:0/1817643777 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6380aaa30 con 0x7ff6440ce920 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 -- 172.21.15.70:0/1817643777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440ce920 msgr2=0x7ff644154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 --2- 172.21.15.70:0/1817643777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440ce920 0x7ff644154c60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff644158cb0 tx=0x7ff63801c4f0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 -- 172.21.15.70:0/1817643777 shutdown_connections 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 --2- 172.21.15.70:0/1817643777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440ce920 0x7ff644154c60 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 --2- 172.21.15.70:0/1817643777 >> v2:172.21.15.98:3300/0 conn(0x7ff6440cedd0 0x7ff6441551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 -- 172.21.15.70:0/1817643777 >> 172.21.15.70:0/1817643777 conn(0x7ff6440b1d40 msgr2=0x7ff6440cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 -- 172.21.15.70:0/1817643777 shutdown_connections 2024-04-22T01:17:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 -- 172.21.15.70:0/1817643777 wait complete. 2024-04-22T01:17:02.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 Processor -- start 2024-04-22T01:17:02.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.162+0000 7ff64ac11640 1 -- start start 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6440ce920 0x7ff64415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 0x7ff6440afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff64415e6e0 0x7ff6441f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6440b2e80 con 0x7ff6440cedd0 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6440b2d00 con 0x7ff64415e6e0 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff6440b3000 con 0x7ff6440ce920 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 0x7ff6440afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 0x7ff6440afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46066/0 (socket says 172.21.15.70:46066) 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 -- 172.21.15.70:0/3449237769 learned_addr learned my addr 172.21.15.70:0/3449237769 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6491ae640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff64415e6e0 0x7ff6441f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6489ad640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6440ce920 0x7ff64415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:02.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 -- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6440ce920 msgr2=0x7ff64415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6440ce920 0x7ff64415a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 -- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff64415e6e0 msgr2=0x7ff6441f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff64415e6e0 0x7ff6441f8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 -- 172.21.15.70:0/3449237769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64414f720 con 0x7ff6440cedd0 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6491ae640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff64415e6e0 0x7ff6441f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff643fff640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 0x7ff6440afdc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff638019100 tx=0x7ff63801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6380bd5f0 con 0x7ff6440cedd0 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6380bd790 con 0x7ff6440cedd0 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64415a290 con 0x7ff6440cedd0 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6380bdb40 con 0x7ff6440cedd0 2024-04-22T01:17:02.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff644159fb0 con 0x7ff6440cedd0 2024-04-22T01:17:02.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff644068210 con 0x7ff6440cedd0 2024-04-22T01:17:02.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6380b33f0 con 0x7ff6440cedd0 2024-04-22T01:17:02.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6417fa640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6300678c0 0x7ff630069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:02.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6187+0+0 (secure 0 0 0) 0x7ff638120dd0 con 0x7ff6440cedd0 2024-04-22T01:17:02.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6489ad640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6300678c0 0x7ff630069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:02.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.166+0000 7ff6489ad640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6300678c0 0x7ff630069d80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff6440b0810 tx=0x7ff634009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:02.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.170+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff6380c3060 con 0x7ff6440cedd0 2024-04-22T01:17:02.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.322+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff6440d08e0 con 0x7ff6440cedd0 2024-04-22T01:17:02.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.978+0000 7ff6417fa640 1 -- 172.21.15.70:0/3449237769 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v54) ==== 181+0+0 (secure 0 0 0) 0x7ff638016110 con 0x7ff6440cedd0 2024-04-22T01:17:02.978 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6300678c0 msgr2=0x7ff630069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6300678c0 0x7ff630069d80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff6440b0810 tx=0x7ff634009040 comp rx=0 tx=0).stop 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 msgr2=0x7ff6440afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 0x7ff6440afdc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff638019100 tx=0x7ff63801ad30 comp rx=0 tx=0).stop 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 shutdown_connections 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6300678c0 0x7ff630069d80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff64415e6e0 0x7ff6441f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440cedd0 0x7ff6440afdc0 secure :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff638019100 tx=0x7ff63801ad30 comp rx=0 tx=0).stop 2024-04-22T01:17:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 --2- 172.21.15.70:0/3449237769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6440ce920 0x7ff64415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 >> 172.21.15.70:0/3449237769 conn(0x7ff6440b1d40 msgr2=0x7ff64415c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 shutdown_connections 2024-04-22T01:17:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:02.982+0000 7ff64ac11640 1 -- 172.21.15.70:0/3449237769 wait complete. 2024-04-22T01:17:03.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:17:03.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 Processor -- start 2024-04-22T01:17:03.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 -- start start 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9b98162350 0x7f9b9806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9b98072280 0x7f9b980ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b9806e340 con 0x7f9b9806de30 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9806e620 con 0x7f9b98072280 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b9dd66640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9806e900 con 0x7f9b98162350 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b97fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9b98072280 0x7f9b980ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b8effd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9b9806de30 0x7f9b98071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34144/0 (socket says 172.21.15.70:34144) 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b8effd640 1 -- 172.21.15.70:0/610708378 learned_addr learned my addr 172.21.15.70:0/610708378 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:03.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b977fe640 1 --2- 172.21.15.70:0/610708378 >> v2:172.21.15.98:3300/0 conn(0x7f9b98162350 0x7f9b9806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:03.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b97fff640 1 -- 172.21.15.70:0/610708378 >> v2:172.21.15.98:3300/0 conn(0x7f9b98162350 msgr2=0x7f9b9806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:03.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b97fff640 1 --2- 172.21.15.70:0/610708378 >> v2:172.21.15.98:3300/0 conn(0x7f9b98162350 0x7f9b9806d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b97fff640 1 -- 172.21.15.70:0/610708378 >> v1:172.21.15.70:6789/0 conn(0x7f9b9806de30 legacy=0x7f9b98071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:03.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.126+0000 7f9b97fff640 1 -- 172.21.15.70:0/610708378 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9806ebe0 con 0x7f9b98072280 2024-04-22T01:17:03.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b97fff640 1 --2- 172.21.15.70:0/610708378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98072280 0x7f9b980ccb90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9b98073010 tx=0x7f9b9001c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82bc063801b40f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b96ffd640 1 -- 172.21.15.70:0/610708378 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b900b2070 con 0x7f9b98072280 2024-04-22T01:17:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b96ffd640 1 -- 172.21.15.70:0/610708378 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b900a5820 con 0x7f9b98072280 2024-04-22T01:17:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b96ffd640 1 -- 172.21.15.70:0/610708378 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b900a5bf0 con 0x7f9b98072280 2024-04-22T01:17:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 -- 172.21.15.70:0/610708378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98072280 msgr2=0x7f9b980ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/610708378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98072280 0x7f9b980ccb90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9b98073010 tx=0x7f9b9001c7c0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 -- 172.21.15.70:0/610708378 shutdown_connections 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/610708378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98072280 0x7f9b980ccb90 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/610708378 >> v2:172.21.15.98:3300/0 conn(0x7f9b98162350 0x7f9b9806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 -- 172.21.15.70:0/610708378 >> 172.21.15.70:0/610708378 conn(0x7f9b980b1d40 msgr2=0x7f9b980cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 -- 172.21.15.70:0/610708378 shutdown_connections 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 -- 172.21.15.70:0/610708378 wait complete. 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 Processor -- start 2024-04-22T01:17:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.130+0000 7f9b9dd66640 1 -- start start 2024-04-22T01:17:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b9806de30 0x7f9b981ebbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b98072280 0x7f9b98162730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 0x7f9b98146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b980b3870 con 0x7f9b98162350 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b980b36f0 con 0x7f9b9806de30 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9806f3e0 con 0x7f9b98072280 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 0x7f9b98146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 0x7f9b98146570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46080/0 (socket says 172.21.15.70:46080) 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 -- 172.21.15.70:0/2035534323 learned_addr learned my addr 172.21.15.70:0/2035534323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b8effd640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b98072280 0x7f9b98162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b977fe640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b9806de30 0x7f9b981ebbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 -- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b98072280 msgr2=0x7f9b98162730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b98072280 0x7f9b98162730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 -- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b9806de30 msgr2=0x7f9b981ebbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b9806de30 0x7f9b981ebbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 -- 172.21.15.70:0/2035534323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b98146ab0 con 0x7f9b98162350 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b97fff640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 0x7f9b98146570 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9b900b0040 tx=0x7f9b90019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b900b2050 con 0x7f9b98162350 2024-04-22T01:17:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b900be070 con 0x7f9b98162350 2024-04-22T01:17:03.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b98146e80 con 0x7f9b98162350 2024-04-22T01:17:03.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90009450 con 0x7f9b98162350 2024-04-22T01:17:03.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b98147440 con 0x7f9b98162350 2024-04-22T01:17:03.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b98068210 con 0x7f9b98162350 2024-04-22T01:17:03.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b900c3460 con 0x7f9b98162350 2024-04-22T01:17:03.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b94ff9640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b780678c0 0x7f9b78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:03.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5749+0+0 (secure 0 0 0) 0x7f9b90121d50 con 0x7f9b98162350 2024-04-22T01:17:03.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.134+0000 7f9b977fe640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b780678c0 0x7f9b78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:03.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.138+0000 7f9b977fe640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b780678c0 0x7f9b78069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b930 tx=0x7f9b88008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:03.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.138+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9b90016440 con 0x7f9b98162350 2024-04-22T01:17:03.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.298+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9b98165b00 con 0x7f9b98162350 2024-04-22T01:17:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.986+0000 7f9b94ff9640 1 -- 172.21.15.70:0/2035534323 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v55) ==== 169+0+0 (secure 0 0 0) 0x7f9b900f28d0 con 0x7f9b98162350 2024-04-22T01:17:03.989 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b780678c0 msgr2=0x7f9b78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b780678c0 0x7f9b78069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b930 tx=0x7f9b88008040 comp rx=0 tx=0).stop 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 msgr2=0x7f9b98146570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 0x7f9b98146570 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9b900b0040 tx=0x7f9b90019560 comp rx=0 tx=0).stop 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 shutdown_connections 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b780678c0 0x7f9b78069d80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b98162350 0x7f9b98146570 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b98072280 0x7f9b98162730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 --2- 172.21.15.70:0/2035534323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b9806de30 0x7f9b981ebbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 >> 172.21.15.70:0/2035534323 conn(0x7f9b980b1d40 msgr2=0x7f9b980b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:03.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 shutdown_connections 2024-04-22T01:17:03.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:03.990+0000 7f9b9dd66640 1 -- 172.21.15.70:0/2035534323 wait complete. 2024-04-22T01:17:04.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-22T01:17:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 Processor -- start 2024-04-22T01:17:04.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 -- start start 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe01409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe0140d0900 0x7fe0140cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe01415e540 con 0x7fe01415e160 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe01406c4a0 con 0x7fe0140d0900 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe018ceb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe01406c780 con 0x7fe0140b4a60 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe012d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe0140d0900 0x7fe0140cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe01259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe01415e160 0x7fe0140d01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34156/0 (socket says 172.21.15.70:34156) 2024-04-22T01:17:04.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe01259c640 1 -- 172.21.15.70:0/470042753 learned_addr learned my addr 172.21.15.70:0/470042753 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:04.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe011d9b640 1 --2- 172.21.15.70:0/470042753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe01409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe012d9d640 1 -- 172.21.15.70:0/470042753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 msgr2=0x7fe01409acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:04.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe012d9d640 1 --2- 172.21.15.70:0/470042753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe01409acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe012d9d640 1 -- 172.21.15.70:0/470042753 >> v1:172.21.15.70:6789/0 conn(0x7fe01415e160 legacy=0x7fe0140d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.142+0000 7fe012d9d640 1 -- 172.21.15.70:0/470042753 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01406ca60 con 0x7fe0140d0900 2024-04-22T01:17:04.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe012d9d640 1 --2- 172.21.15.70:0/470042753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140cfb40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009d90 tx=0x7fe00c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8477c7c295da9925 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:04.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe01159a640 1 -- 172.21.15.70:0/470042753 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c0b2070 con 0x7fe0140d0900 2024-04-22T01:17:04.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe01159a640 1 -- 172.21.15.70:0/470042753 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00c0a5830 con 0x7fe0140d0900 2024-04-22T01:17:04.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe01159a640 1 -- 172.21.15.70:0/470042753 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c0a5c00 con 0x7fe0140d0900 2024-04-22T01:17:04.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 -- 172.21.15.70:0/470042753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 msgr2=0x7fe0140cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 --2- 172.21.15.70:0/470042753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140cfb40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009d90 tx=0x7fe00c01c970 comp rx=0 tx=0).stop 2024-04-22T01:17:04.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 -- 172.21.15.70:0/470042753 shutdown_connections 2024-04-22T01:17:04.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 --2- 172.21.15.70:0/470042753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140cfb40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 --2- 172.21.15.70:0/470042753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe01409acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 -- 172.21.15.70:0/470042753 >> 172.21.15.70:0/470042753 conn(0x7fe0140b3700 msgr2=0x7fe0140b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:04.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 -- 172.21.15.70:0/470042753 shutdown_connections 2024-04-22T01:17:04.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.146+0000 7fe018ceb640 1 -- 172.21.15.70:0/470042753 wait complete. 2024-04-22T01:17:04.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 Processor -- start 2024-04-22T01:17:04.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- start start 2024-04-22T01:17:04.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe0141501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe01415e160 0x7fe014160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0140b33f0 con 0x7fe0140d0900 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0140b3270 con 0x7fe01415e160 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe0140c7ac0 con 0x7fe0140b4a60 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46092/0 (socket says 172.21.15.70:46092) 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 -- 172.21.15.70:0/2125819953 learned_addr learned my addr 172.21.15.70:0/2125819953 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 -- 172.21.15.70:0/2125819953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 msgr2=0x7fe0141501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe0141501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 -- 172.21.15.70:0/2125819953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe01415e160 msgr2=0x7fe014160770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe01415e160 0x7fe014160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 -- 172.21.15.70:0/2125819953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0140c6eb0 con 0x7fe0140d0900 2024-04-22T01:17:04.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe011d9b640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140c7f50 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe00800d7d0 tx=0x7fe00800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:04.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00800b6c0 con 0x7fe0140d0900 2024-04-22T01:17:04.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe008019070 con 0x7fe0140d0900 2024-04-22T01:17:04.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0140c7190 con 0x7fe0140d0900 2024-04-22T01:17:04.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00800ba80 con 0x7fe0140d0900 2024-04-22T01:17:04.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0140c7750 con 0x7fe0140d0900 2024-04-22T01:17:04.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe014068920 con 0x7fe0140d0900 2024-04-22T01:17:04.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe00800bc20 con 0x7fe0140d0900 2024-04-22T01:17:04.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe002ffd640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdff40678c0 0x7fdff4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7fe008086050 con 0x7fe0140d0900 2024-04-22T01:17:04.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.150+0000 7fe01259c640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdff40678c0 0x7fdff4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.154+0000 7fe01259c640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdff40678c0 0x7fdff4069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe004000f70 tx=0x7fe004007a90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:04.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.154+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe008056c70 con 0x7fe0140d0900 2024-04-22T01:17:04.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.318+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fe0140cfba0 con 0x7fe0140d0900 2024-04-22T01:17:04.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.322+0000 7fe002ffd640 1 -- 172.21.15.70:0/2125819953 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7fe008056570 con 0x7fe0140d0900 2024-04-22T01:17:04.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdff40678c0 msgr2=0x7fdff4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdff40678c0 0x7fdff4069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe004000f70 tx=0x7fe004007a90 comp rx=0 tx=0).stop 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 msgr2=0x7fe0140c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140c7f50 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe00800d7d0 tx=0x7fe00800dc90 comp rx=0 tx=0).stop 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 shutdown_connections 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdff40678c0 0x7fdff4069d80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe01415e160 0x7fe014160770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0140d0900 0x7fe0140c7f50 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 --2- 172.21.15.70:0/2125819953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0140b4a60 0x7fe0141501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 >> 172.21.15.70:0/2125819953 conn(0x7fe0140b3700 msgr2=0x7fe0140cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 shutdown_connections 2024-04-22T01:17:04.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.326+0000 7fe018ceb640 1 -- 172.21.15.70:0/2125819953 wait complete. 2024-04-22T01:17:04.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values' 2024-04-22T01:17:04.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 Processor -- start 2024-04-22T01:17:04.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 -- start start 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9fe815a550 0x7f9fe80aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe80aeae0 0x7f9fe80cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe806f010 con 0x7f9fe815a550 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe80cf4a0 con 0x7f9fe80aeae0 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fef35d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe806f260 con 0x7f9fe80cd940 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9fe815a550 0x7f9fe80aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9fe815a550 0x7f9fe80aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46100/0 (socket says 172.21.15.70:46100) 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2406863503 learned_addr learned my addr 172.21.15.70:0/2406863503 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed8fa640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fec8f8640 1 --2- 172.21.15.70:0/2406863503 >> v2:172.21.15.98:3300/0 conn(0x7f9fe80aeae0 0x7f9fe80cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2406863503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 msgr2=0x7f9fe8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe8071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2406863503 >> v2:172.21.15.98:3300/0 conn(0x7f9fe80aeae0 msgr2=0x7f9fe80cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 --2- 172.21.15.70:0/2406863503 >> v2:172.21.15.98:3300/0 conn(0x7f9fe80aeae0 0x7f9fe80cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2406863503 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe806f540 con 0x7f9fe815a550 2024-04-22T01:17:04.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.470+0000 7f9fed8fa640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:04.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe815a550 0x7f9fe80aef50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80b3e80 tx=0x7f9fe401d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a5ac78e7282e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fdffff640 1 -- 172.21.15.70:0/2406863503 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe400b040 con 0x7f9fe815a550 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fdffff640 1 -- 172.21.15.70:0/2406863503 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe40a7710 con 0x7f9fe815a550 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fdffff640 1 -- 172.21.15.70:0/2406863503 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40a7ae0 con 0x7f9fe815a550 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- 172.21.15.70:0/2406863503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe815a550 msgr2=0x7f9fe80aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe815a550 0x7f9fe80aef50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80b3e80 tx=0x7f9fe401d8f0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- 172.21.15.70:0/2406863503 shutdown_connections 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2406863503 >> v2:172.21.15.98:3300/0 conn(0x7f9fe80aeae0 0x7f9fe80cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2406863503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe815a550 0x7f9fe80aef50 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- 172.21.15.70:0/2406863503 >> 172.21.15.70:0/2406863503 conn(0x7f9fe806b1f0 msgr2=0x7f9fe806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:04.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- 172.21.15.70:0/2406863503 shutdown_connections 2024-04-22T01:17:04.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- 172.21.15.70:0/2406863503 wait complete. 2024-04-22T01:17:04.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 Processor -- start 2024-04-22T01:17:04.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- start start 2024-04-22T01:17:04.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 0x7f9fe813d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe814cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe815a550 0x7f9fe814d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe80b30d0 con 0x7f9fe80aeae0 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe80b2f50 con 0x7f9fe815a550 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fef35d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe80b3250 con 0x7f9fe80cd940 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 0x7f9fe813d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 0x7f9fe813d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46114/0 (socket says 172.21.15.70:46114) 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2475126698 learned_addr learned my addr 172.21.15.70:0/2475126698 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fec8f8640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe814cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed8fa640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe815a550 0x7f9fe814d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 msgr2=0x7f9fe814cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe814cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe815a550 msgr2=0x7f9fe814d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe815a550 0x7f9fe814d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fed0f9640 1 -- 172.21.15.70:0/2475126698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe814fc50 con 0x7f9fe80aeae0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.474+0000 7f9fec8f8640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe814cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fed0f9640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 0x7f9fe813d4a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400a1c0 tx=0x7f9fe401c130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40be070 con 0x7f9fe80aeae0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe40afa60 con 0x7f9fe80aeae0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe814ff30 con 0x7f9fe80aeae0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40b9410 con 0x7f9fe80aeae0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe813e030 con 0x7f9fe80aeae0 2024-04-22T01:17:04.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe8068210 con 0x7f9fe80aeae0 2024-04-22T01:17:04.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9fe40c1070 con 0x7f9fe80aeae0 2024-04-22T01:17:04.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fddffb640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9fcc0678c0 0x7f9fcc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:04.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7f9fe411fe70 con 0x7f9fe80aeae0 2024-04-22T01:17:04.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fec8f8640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9fcc0678c0 0x7f9fcc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:04.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.478+0000 7f9fec8f8640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9fcc0678c0 0x7f9fcc069d80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9fe814cbe0 tx=0x7f9fd8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:04.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.482+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9fe40174e0 con 0x7f9fe80aeae0 2024-04-22T01:17:04.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.690+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]} v 0) -- 0x7f9fe806cb00 con 0x7f9fe80aeae0 2024-04-22T01:17:04.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.990+0000 7f9fddffb640 1 -- 172.21.15.70:0/2475126698 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]}]=0 v149) ==== 153+0+0 (secure 0 0 0) 0x7f9fe40f0cb0 con 0x7f9fe80aeae0 2024-04-22T01:17:04.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9fcc0678c0 msgr2=0x7f9fcc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9fcc0678c0 0x7f9fcc069d80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9fe814cbe0 tx=0x7f9fd8008040 comp rx=0 tx=0).stop 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 msgr2=0x7f9fe813d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 0x7f9fe813d4a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400a1c0 tx=0x7f9fe401c130 comp rx=0 tx=0).stop 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 shutdown_connections 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9fcc0678c0 0x7f9fcc069d80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe815a550 0x7f9fe814d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9fe80cd940 0x7f9fe814cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 --2- 172.21.15.70:0/2475126698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fe80aeae0 0x7f9fe813d4a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 >> 172.21.15.70:0/2475126698 conn(0x7f9fe806b1f0 msgr2=0x7f9fe81529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 shutdown_connections 2024-04-22T01:17:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:04.994+0000 7f9fef35d640 1 -- 172.21.15.70:0/2475126698 wait complete. 2024-04-22T01:17:05.016 INFO:tasks.cephfs_test_runner:test invalid values for ceph.quota.max_bytes ... ok 2024-04-22T01:17:05.016 INFO:tasks.cephfs_test_runner:Starting test: test_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) test human-readable values for setting ceph.quota.max_bytes 2024-04-22T01:17:05.016 INFO:tasks.cephfs_test_runner:test_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2024-04-22T01:17:05.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.138+0000 7fed1505d640 1 Processor -- start 2024-04-22T01:17:05.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.138+0000 7fed1505d640 1 -- start start 2024-04-22T01:17:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed1505d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015aa30 0x7fed100b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed1505d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed100aee40 0x7fed100b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed1505d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed1006dbd0 con 0x7fed1015bb50 2024-04-22T01:17:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed1505d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fed1006de70 con 0x7fed100aee40 2024-04-22T01:17:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed1505d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed1006e150 con 0x7fed1015aa30 2024-04-22T01:17:05.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed100aee40 0x7fed100b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed100aee40 0x7fed100b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46126/0 (socket says 172.21.15.70:46126) 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 -- 172.21.15.70:0/276171499 learned_addr learned my addr 172.21.15.70:0/276171499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0ed9d640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015aa30 0x7fed100b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 -- 172.21.15.70:0/276171499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015aa30 msgr2=0x7fed100b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015aa30 0x7fed100b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 -- 172.21.15.70:0/276171499 >> v1:172.21.15.98:6789/0 conn(0x7fed1015bb50 legacy=0x7fed100ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 -- 172.21.15.70:0/276171499 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1006e430 con 0x7fed100aee40 2024-04-22T01:17:05.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0ed9d640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015aa30 0x7fed100b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0f59e640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100b28f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fed0800d150 tx=0x7fed0801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71a6a73d21f9f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0e59c640 1 -- 172.21.15.70:0/276171499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed080a7ce0 con 0x7fed100aee40 2024-04-22T01:17:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0e59c640 1 -- 172.21.15.70:0/276171499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0800b0d0 con 0x7fed100aee40 2024-04-22T01:17:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.142+0000 7fed0e59c640 1 -- 172.21.15.70:0/276171499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08009600 con 0x7fed100aee40 2024-04-22T01:17:05.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/276171499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 msgr2=0x7fed100b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100b28f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fed0800d150 tx=0x7fed0801ec30 comp rx=0 tx=0).stop 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/276171499 shutdown_connections 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100b28f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 --2- 172.21.15.70:0/276171499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015aa30 0x7fed100b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/276171499 >> 172.21.15.70:0/276171499 conn(0x7fed1006b1f0 msgr2=0x7fed1006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/276171499 shutdown_connections 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/276171499 wait complete. 2024-04-22T01:17:05.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 Processor -- start 2024-04-22T01:17:05.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- start start 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed1015aa30 0x7fed100b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015bb50 0x7fed100b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed1006ead0 con 0x7fed100aee40 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed1006e950 con 0x7fed1015aa30 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed10071b20 con 0x7fed1015bb50 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46142/0 (socket says 172.21.15.70:46142) 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 -- 172.21.15.70:0/529430493 learned_addr learned my addr 172.21.15.70:0/529430493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0f59e640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015bb50 0x7fed100b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed07fff640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed1015aa30 0x7fed100b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 -- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015bb50 msgr2=0x7fed100b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015bb50 0x7fed100b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 -- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed1015aa30 msgr2=0x7fed100b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed1015aa30 0x7fed100b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 -- 172.21.15.70:0/529430493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed10164b40 con 0x7fed100aee40 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed0ed9d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100ce990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e840 tx=0x7fed0000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00017070 con 0x7fed100aee40 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0000c8d0 con 0x7fed100aee40 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed100af560 con 0x7fed100aee40 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0000f040 con 0x7fed100aee40 2024-04-22T01:17:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.146+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed100afb20 con 0x7fed100aee40 2024-04-22T01:17:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.150+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed10068210 con 0x7fed100aee40 2024-04-22T01:17:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.150+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed0001a070 con 0x7fed100aee40 2024-04-22T01:17:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.150+0000 7fed077fe640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fecf00678c0 0x7fecf0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.150+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7fed000856f0 con 0x7fed100aee40 2024-04-22T01:17:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.150+0000 7fed07fff640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fecf00678c0 0x7fecf0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.150+0000 7fed07fff640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fecf00678c0 0x7fecf0069d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fecf80015b0 tx=0x7fecf8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.154+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fed00056430 con 0x7fed100aee40 2024-04-22T01:17:05.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.314+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed100aec20 con 0x7fed100aee40 2024-04-22T01:17:05.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed077fe640 1 -- 172.21.15.70:0/529430493 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+13257 (secure 0 0 0) 0x7fed00085070 con 0x7fed100aee40 2024-04-22T01:17:05.317 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:05.317 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:03.980707+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:6837/2188446179":"2024-04-23T01:17:00.964431+0000","172.21.15.70:6835/2188446179":"2024-04-23T01:17:00.964431+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fecf00678c0 msgr2=0x7fecf0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fecf00678c0 0x7fecf0069d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fecf80015b0 tx=0x7fecf8009040 comp rx=0 tx=0).stop 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 msgr2=0x7fed100ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100ce990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e840 tx=0x7fed0000ed00 comp rx=0 tx=0).stop 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 shutdown_connections 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fecf00678c0 0x7fecf0069d80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1015bb50 0x7fed100b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed1015aa30 0x7fed100b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 --2- 172.21.15.70:0/529430493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed100aee40 0x7fed100ce990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 >> 172.21.15.70:0/529430493 conn(0x7fed1006b1f0 msgr2=0x7fed1015d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 shutdown_connections 2024-04-22T01:17:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.318+0000 7fed1505d640 1 -- 172.21.15.70:0/529430493 wait complete. 2024-04-22T01:17:05.337 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:05.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:05.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.450+0000 7ff88bc76640 1 Processor -- start 2024-04-22T01:17:05.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.450+0000 7ff88bc76640 1 -- start start 2024-04-22T01:17:05.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8840b98a0 0x7ff8840d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff8840cea10 0x7ff8840cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8840b9ce0 con 0x7ff88415de20 2024-04-22T01:17:05.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff8840b75e0 con 0x7ff8840cea10 2024-04-22T01:17:05.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff8840b78c0 con 0x7ff8840b98a0 2024-04-22T01:17:05.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff8840cea10 0x7ff8840cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff8840cea10 0x7ff8840cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46150/0 (socket says 172.21.15.70:46150) 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 -- 172.21.15.70:0/1524436206 learned_addr learned my addr 172.21.15.70:0/1524436206 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff889211640 1 --2- 172.21.15.70:0/1524436206 >> v2:172.21.15.98:3300/0 conn(0x7ff8840b98a0 0x7ff8840d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 -- 172.21.15.70:0/1524436206 >> v2:172.21.15.98:3300/0 conn(0x7ff8840b98a0 msgr2=0x7ff8840d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 --2- 172.21.15.70:0/1524436206 >> v2:172.21.15.98:3300/0 conn(0x7ff8840b98a0 0x7ff8840d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 -- 172.21.15.70:0/1524436206 >> v1:172.21.15.70:6789/0 conn(0x7ff88415de20 legacy=0x7ff8840b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff888a10640 1 -- 172.21.15.70:0/1524436206 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232872178 0 0) 0x7ff8840b9ce0 con 0x7ff88415de20 2024-04-22T01:17:05.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 -- 172.21.15.70:0/1524436206 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8840b7ba0 con 0x7ff8840cea10 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88a213640 1 --2- 172.21.15.70:0/1524436206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840cee60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff8840d4e10 tx=0x7ff87c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70051d36ffaa0b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff888a10640 1 -- 172.21.15.70:0/1524436206 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0a5880 con 0x7ff8840cea10 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff888a10640 1 -- 172.21.15.70:0/1524436206 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff87c0a5a20 con 0x7ff8840cea10 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff888a10640 1 -- 172.21.15.70:0/1524436206 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0a5df0 con 0x7ff8840cea10 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- 172.21.15.70:0/1524436206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 msgr2=0x7ff8840cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1524436206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840cee60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff8840d4e10 tx=0x7ff87c01c9c0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- 172.21.15.70:0/1524436206 shutdown_connections 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1524436206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840cee60 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1524436206 >> v2:172.21.15.98:3300/0 conn(0x7ff8840b98a0 0x7ff8840d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- 172.21.15.70:0/1524436206 >> 172.21.15.70:0/1524436206 conn(0x7ff884072d20 msgr2=0x7ff884073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:05.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- 172.21.15.70:0/1524436206 shutdown_connections 2024-04-22T01:17:05.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.454+0000 7ff88bc76640 1 -- 172.21.15.70:0/1524436206 wait complete. 2024-04-22T01:17:05.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 Processor -- start 2024-04-22T01:17:05.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- start start 2024-04-22T01:17:05.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8840b98a0 0x7ff88414ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840d2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff88415de20 0x7ff88415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8840d22c0 con 0x7ff8840cea10 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff88415e200 con 0x7ff8840b98a0 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff8840b8e90 con 0x7ff88415de20 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840d2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840d2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46166/0 (socket says 172.21.15.70:46166) 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 -- 172.21.15.70:0/1619280275 learned_addr learned my addr 172.21.15.70:0/1619280275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889a12640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8840b98a0 0x7ff88414ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88a213640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff88415de20 0x7ff88415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 -- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff88415de20 msgr2=0x7ff88415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff88415de20 0x7ff88415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 -- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8840b98a0 msgr2=0x7ff88414ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8840b98a0 0x7ff88414ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 -- 172.21.15.70:0/1619280275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8840d3490 con 0x7ff8840cea10 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88a213640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff88415de20 0x7ff88415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:05.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889a12640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8840b98a0 0x7ff88414ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:05.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff889211640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840d2f50 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff87400d830 tx=0x7ff87400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87400b6c0 con 0x7ff8840cea10 2024-04-22T01:17:05.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff874019070 con 0x7ff8840cea10 2024-04-22T01:17:05.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff884159010 con 0x7ff8840cea10 2024-04-22T01:17:05.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87400ba60 con 0x7ff8840cea10 2024-04-22T01:17:05.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8841595d0 con 0x7ff8840cea10 2024-04-22T01:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.458+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff884068210 con 0x7ff8840cea10 2024-04-22T01:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.462+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff87401e460 con 0x7ff8840cea10 2024-04-22T01:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.462+0000 7ff87a7fc640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8680678c0 0x7ff868069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.462+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7ff874086220 con 0x7ff8840cea10 2024-04-22T01:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.462+0000 7ff889a12640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8680678c0 0x7ff868069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.462+0000 7ff889a12640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8680678c0 0x7ff868069d80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff880000f70 tx=0x7ff880009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.466+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff874056ed0 con 0x7ff8840cea10 2024-04-22T01:17:05.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff884159f00 con 0x7ff8840cea10 2024-04-22T01:17:05.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff87a7fc640 1 -- 172.21.15.70:0/1619280275 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7ff8740567d0 con 0x7ff8840cea10 2024-04-22T01:17:05.615 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:05.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8680678c0 msgr2=0x7ff868069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8680678c0 0x7ff868069d80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff880000f70 tx=0x7ff880009040 comp rx=0 tx=0).stop 2024-04-22T01:17:05.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 msgr2=0x7ff8840d2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840d2f50 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff87400d830 tx=0x7ff87400dcf0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 shutdown_connections 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8680678c0 0x7ff868069d80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff88415de20 0x7ff88415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8840cea10 0x7ff8840d2f50 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 --2- 172.21.15.70:0/1619280275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8840b98a0 0x7ff88414ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 >> 172.21.15.70:0/1619280275 conn(0x7ff884072d20 msgr2=0x7ff8840b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 shutdown_connections 2024-04-22T01:17:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.614+0000 7ff88bc76640 1 -- 172.21.15.70:0/1619280275 wait complete. 2024-04-22T01:17:05.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values' 2024-04-22T01:17:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 Processor -- start 2024-04-22T01:17:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 -- start start 2024-04-22T01:17:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9dc40cc150 0x7f9dc4071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc4072060 0x7f9dc406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc40cd7b0 con 0x7f9dc41455b0 2024-04-22T01:17:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9dc40724d0 con 0x7f9dc40cc150 2024-04-22T01:17:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcc50b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc40cc710 con 0x7f9dc4072060 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dc9aa6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9dc40cc150 0x7f9dc4071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dca2a7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9dc41455b0 0x7f9dc40cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:59696/0 (socket says 172.21.15.70:59696) 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dca2a7640 1 -- 172.21.15.70:0/1421073391 learned_addr learned my addr 172.21.15.70:0/1421073391 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcaaa8640 1 --2- 172.21.15.70:0/1421073391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc4072060 0x7f9dc406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/1421073391 >> v2:172.21.15.98:3300/0 conn(0x7f9dc40cc150 msgr2=0x7f9dc4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcaaa8640 1 --2- 172.21.15.70:0/1421073391 >> v2:172.21.15.98:3300/0 conn(0x7f9dc40cc150 0x7f9dc4071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/1421073391 >> v1:172.21.15.98:6789/0 conn(0x7f9dc41455b0 legacy=0x7f9dc40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/1421073391 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc40cc9f0 con 0x7f9dc4072060 2024-04-22T01:17:05.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.738+0000 7f9dc9aa6640 1 --2- 172.21.15.70:0/1421073391 >> v2:172.21.15.98:3300/0 conn(0x7f9dc40cc150 0x7f9dc4071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:17:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcaaa8640 1 --2- 172.21.15.70:0/1421073391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc4072060 0x7f9dc406e200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009d90 tx=0x7f9dbc01c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e9d0f3038411a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dc92a5640 1 -- 172.21.15.70:0/1421073391 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0b2070 con 0x7f9dc4072060 2024-04-22T01:17:05.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dc92a5640 1 -- 172.21.15.70:0/1421073391 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0a5710 con 0x7f9dc4072060 2024-04-22T01:17:05.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dc92a5640 1 -- 172.21.15.70:0/1421073391 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0a5ae0 con 0x7f9dc4072060 2024-04-22T01:17:05.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 -- 172.21.15.70:0/1421073391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc4072060 msgr2=0x7f9dc406e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/1421073391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc4072060 0x7f9dc406e200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009d90 tx=0x7f9dbc01c850 comp rx=0 tx=0).stop 2024-04-22T01:17:05.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 -- 172.21.15.70:0/1421073391 shutdown_connections 2024-04-22T01:17:05.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/1421073391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc4072060 0x7f9dc406e200 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/1421073391 >> v2:172.21.15.98:3300/0 conn(0x7f9dc40cc150 0x7f9dc4071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 -- 172.21.15.70:0/1421073391 >> 172.21.15.70:0/1421073391 conn(0x7f9dc406b1f0 msgr2=0x7f9dc406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 -- 172.21.15.70:0/1421073391 shutdown_connections 2024-04-22T01:17:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 -- 172.21.15.70:0/1421073391 wait complete. 2024-04-22T01:17:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 Processor -- start 2024-04-22T01:17:05.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.742+0000 7f9dcc50b640 1 -- start start 2024-04-22T01:17:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dc4072060 0x7f9dc41575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc40cc150 0x7f9dc40adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 0x7f9dc40b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc406f320 con 0x7f9dc41455b0 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc406f1a0 con 0x7f9dc4072060 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc4168b40 con 0x7f9dc40cc150 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 0x7f9dc40b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 0x7f9dc40b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46174/0 (socket says 172.21.15.70:46174) 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/763089010 learned_addr learned my addr 172.21.15.70:0/763089010 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dc9aa6640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc40cc150 0x7f9dc40adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc40cc150 msgr2=0x7f9dc40adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc40cc150 0x7f9dc40adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dc4072060 msgr2=0x7f9dc41575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dc4072060 0x7f9dc41575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 -- 172.21.15.70:0/763089010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc40ac8d0 con 0x7f9dc41455b0 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcaaa8640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 0x7f9dc40b4190 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0b0040 tx=0x7f9dbc0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0b2050 con 0x7f9dc41455b0 2024-04-22T01:17:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0be070 con 0x7f9dc41455b0 2024-04-22T01:17:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc40acb70 con 0x7f9dc41455b0 2024-04-22T01:17:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0ad890 con 0x7f9dc41455b0 2024-04-22T01:17:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc40ad130 con 0x7f9dc41455b0 2024-04-22T01:17:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc4068210 con 0x7f9dc41455b0 2024-04-22T01:17:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9dbc0ada30 con 0x7f9dc41455b0 2024-04-22T01:17:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dbaffd640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9da80678c0 0x7f9da8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7f9dbc121420 con 0x7f9dc41455b0 2024-04-22T01:17:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dca2a7640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9da80678c0 0x7f9da8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.746+0000 7f9dca2a7640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9da80678c0 0x7f9da8069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0000f70 tx=0x7f9dc0007a90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.750+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9dbc016560 con 0x7f9dc41455b0 2024-04-22T01:17:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.954+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]} v 0) -- 0x7f9dc415f510 con 0x7f9dc41455b0 2024-04-22T01:17:05.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.990+0000 7f9dbaffd640 1 -- 172.21.15.70:0/763089010 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]}]=0 v150) ==== 148+0+0 (secure 0 0 0) 0x7f9dbc0f2240 con 0x7f9dc41455b0 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9da80678c0 msgr2=0x7f9da8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9da80678c0 0x7f9da8069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0000f70 tx=0x7f9dc0007a90 comp rx=0 tx=0).stop 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 msgr2=0x7f9dc40b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 0x7f9dc40b4190 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0b0040 tx=0x7f9dbc0195f0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 shutdown_connections 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9da80678c0 0x7f9da8069d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9dc41455b0 0x7f9dc40b4190 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dc40cc150 0x7f9dc40adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 --2- 172.21.15.70:0/763089010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dc4072060 0x7f9dc41575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:05.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 >> 172.21.15.70:0/763089010 conn(0x7f9dc406b1f0 msgr2=0x7f9dc413fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:05.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 shutdown_connections 2024-04-22T01:17:05.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:05.994+0000 7f9dcc50b640 1 -- 172.21.15.70:0/763089010 wait complete. 2024-04-22T01:17:06.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-22T01:17:06.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.126+0000 7fc01af95640 1 Processor -- start 2024-04-22T01:17:06.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.126+0000 7fc01af95640 1 -- start start 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc01af95640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc01415e6e0 0x7fc01415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc01af95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc014155370 0x7fc01406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc01af95640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01415f000 con 0x7fc0140ce920 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc01af95640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc0140b3800 con 0x7fc01415e6e0 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc01af95640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0140b3ae0 con 0x7fc014155370 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc01415e6e0 0x7fc01415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc01415e6e0 0x7fc01415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41788/0 (socket says 172.21.15.70:41788) 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 -- 172.21.15.70:0/2459722158 learned_addr learned my addr 172.21.15.70:0/2459722158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 -- 172.21.15.70:0/2459722158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc014155370 msgr2=0x7fc01406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc019532640 1 --2- 172.21.15.70:0/2459722158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc014155370 0x7fc01406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 --2- 172.21.15.70:0/2459722158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc014155370 0x7fc01406d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 -- 172.21.15.70:0/2459722158 >> v1:172.21.15.70:6789/0 conn(0x7fc0140ce920 legacy=0x7fc014154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc00b7fe640 1 -- 172.21.15.70:0/2459722158 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761474299 0 0) 0x7fc01415f000 con 0x7fc0140ce920 2024-04-22T01:17:06.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 -- 172.21.15.70:0/2459722158 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0140cde40 con 0x7fc01415e6e0 2024-04-22T01:17:06.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc018d31640 1 --2- 172.21.15.70:0/2459722158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415eac0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc010009870 tx=0x7fc01001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dfa779db59d829c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:06.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc00b7fe640 1 -- 172.21.15.70:0/2459722158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0100a95f0 con 0x7fc01415e6e0 2024-04-22T01:17:06.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc00b7fe640 1 -- 172.21.15.70:0/2459722158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0100a9790 con 0x7fc01415e6e0 2024-04-22T01:17:06.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.130+0000 7fc00b7fe640 1 -- 172.21.15.70:0/2459722158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0100a9b60 con 0x7fc01415e6e0 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2459722158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 msgr2=0x7fc01415eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 --2- 172.21.15.70:0/2459722158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415eac0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc010009870 tx=0x7fc01001c530 comp rx=0 tx=0).stop 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2459722158 shutdown_connections 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 --2- 172.21.15.70:0/2459722158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc014155370 0x7fc01406d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 --2- 172.21.15.70:0/2459722158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415eac0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2459722158 >> 172.21.15.70:0/2459722158 conn(0x7fc0140b1d40 msgr2=0x7fc0140cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2459722158 shutdown_connections 2024-04-22T01:17:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2459722158 wait complete. 2024-04-22T01:17:06.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 Processor -- start 2024-04-22T01:17:06.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- start start 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0140ce920 0x7fc014141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc014155370 0x7fc014159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc01406f260 con 0x7fc01415e6e0 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc01406f0e0 con 0x7fc014155370 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0140b32c0 con 0x7fc0140ce920 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41794/0 (socket says 172.21.15.70:41794) 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 -- 172.21.15.70:0/2469174996 learned_addr learned my addr 172.21.15.70:0/2469174996 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc018d31640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0140ce920 0x7fc014141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 -- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0140ce920 msgr2=0x7fc014141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0140ce920 0x7fc014141680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 -- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc014155370 msgr2=0x7fc014159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc014155370 0x7fc014159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 -- 172.21.15.70:0/2469174996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc014141bc0 con 0x7fc01415e6e0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc019532640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415a260 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00cd70 tx=0x7fc00c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c003ba0 con 0x7fc01415e6e0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c003d40 con 0x7fc01415e6e0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc014158f60 con 0x7fc01415e6e0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c004240 con 0x7fc01415e6e0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.134+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc014159520 con 0x7fc01415e6e0 2024-04-22T01:17:06.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.138+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc014068210 con 0x7fc01415e6e0 2024-04-22T01:17:06.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.138+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc00c0043e0 con 0x7fc01415e6e0 2024-04-22T01:17:06.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.138+0000 7fc0097fa640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0000678f0 0x7fc000069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.138+0000 7fc018d31640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0000678f0 0x7fc000069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.138+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7fc00c086510 con 0x7fc01415e6e0 2024-04-22T01:17:06.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.138+0000 7fc018d31640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0000678f0 0x7fc000069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc0100ae000 tx=0x7fc0100a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.142+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc014141bc0 con 0x7fc01415e6e0 2024-04-22T01:17:06.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.298+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc01406e910 con 0x7fc01415e6e0 2024-04-22T01:17:06.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc0097fa640 1 -- 172.21.15.70:0/2469174996 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7fc00c01d340 con 0x7fc01415e6e0 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0000678f0 msgr2=0x7fc000069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0000678f0 0x7fc000069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc0100ae000 tx=0x7fc0100a7000 comp rx=0 tx=0).stop 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 msgr2=0x7fc01415a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415a260 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00cd70 tx=0x7fc00c00a660 comp rx=0 tx=0).stop 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 shutdown_connections 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0000678f0 0x7fc000069db0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc01415e6e0 0x7fc01415a260 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc014155370 0x7fc014159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 --2- 172.21.15.70:0/2469174996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0140ce920 0x7fc014141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 >> 172.21.15.70:0/2469174996 conn(0x7fc0140b1d40 msgr2=0x7fc014153250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:06.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 shutdown_connections 2024-04-22T01:17:06.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.306+0000 7fc01af95640 1 -- 172.21.15.70:0/2469174996 wait complete. 2024-04-22T01:17:06.324 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:17:06.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:17:06.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:17:06.419 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:17:06.419 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:17:06.420 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:06.424 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:17:06.424 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:06.424 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:06.430 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:17:06.431 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:17:06.431 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:06.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:17:06.470 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:06.470 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:06.478 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:17:06.478 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-22T01:17:06.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:06.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 Processor -- start 2024-04-22T01:17:06.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 -- start start 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1df81455b0 0x7f1df80cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cc150 0x7f1df8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df80d0250 con 0x7f1df8072060 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1df80724d0 con 0x7f1df81455b0 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dff7b0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1df80cd7b0 con 0x7f1df80cc150 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1df81455b0 0x7f1df80cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1df81455b0 0x7f1df80cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41804/0 (socket says 172.21.15.70:41804) 2024-04-22T01:17:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 -- 172.21.15.70:0/3765594812 learned_addr learned my addr 172.21.15.70:0/3765594812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:06.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfcd4b640 1 --2- 172.21.15.70:0/3765594812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cc150 0x7f1df8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 -- 172.21.15.70:0/3765594812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cc150 msgr2=0x7f1df8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:06.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 --2- 172.21.15.70:0/3765594812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cc150 0x7f1df8071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 -- 172.21.15.70:0/3765594812 >> v1:172.21.15.98:6789/0 conn(0x7f1df8072060 legacy=0x7f1df815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:06.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.634+0000 7f1dfd54c640 1 -- 172.21.15.70:0/3765594812 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df80cc710 con 0x7f1df81455b0 2024-04-22T01:17:06.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dfd54c640 1 --2- 172.21.15.70:0/3765594812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df81455b0 0x7f1df80cfd10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009870 tx=0x7f1de801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341a373af9d7e0cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:06.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1deffff640 1 -- 172.21.15.70:0/3765594812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de80b0070 con 0x7f1df81455b0 2024-04-22T01:17:06.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1deffff640 1 -- 172.21.15.70:0/3765594812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1de80ab600 con 0x7f1df81455b0 2024-04-22T01:17:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1deffff640 1 -- 172.21.15.70:0/3765594812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de80ab9d0 con 0x7f1df81455b0 2024-04-22T01:17:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 -- 172.21.15.70:0/3765594812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df81455b0 msgr2=0x7f1df80cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/3765594812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df81455b0 0x7f1df80cfd10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009870 tx=0x7f1de801c400 comp rx=0 tx=0).stop 2024-04-22T01:17:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 -- 172.21.15.70:0/3765594812 shutdown_connections 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/3765594812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cc150 0x7f1df8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/3765594812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df81455b0 0x7f1df80cfd10 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 -- 172.21.15.70:0/3765594812 >> 172.21.15.70:0/3765594812 conn(0x7f1df806b1f0 msgr2=0x7f1df806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 -- 172.21.15.70:0/3765594812 shutdown_connections 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.638+0000 7f1dff7b0640 1 -- 172.21.15.70:0/3765594812 wait complete. 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 Processor -- start 2024-04-22T01:17:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- start start 2024-04-22T01:17:06.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df8072060 0x7f1df8159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 0x7f1df813c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df81455b0 0x7f1df813c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8168b40 con 0x7f1df80cc150 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1df806f210 con 0x7f1df8072060 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1df8168cc0 con 0x7f1df81455b0 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 0x7f1df813c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 0x7f1df813c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41810/0 (socket says 172.21.15.70:41810) 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 -- 172.21.15.70:0/236098017 learned_addr learned my addr 172.21.15.70:0/236098017 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfd54c640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df8072060 0x7f1df8159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfdd4d640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df81455b0 0x7f1df813c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 -- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df81455b0 msgr2=0x7f1df813c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df81455b0 0x7f1df813c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 -- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df8072060 msgr2=0x7f1df8159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df8072060 0x7f1df8159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 -- 172.21.15.70:0/236098017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df815e1c0 con 0x7f1df80cc150 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfdd4d640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df81455b0 0x7f1df813c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dfcd4b640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 0x7f1df813c3b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1df400d800 tx=0x7f1df400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df400b6c0 con 0x7f1df80cc150 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df4019070 con 0x7f1df80cc150 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df400bae0 con 0x7f1df80cc150 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df813e580 con 0x7f1df80cc150 2024-04-22T01:17:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df813e820 con 0x7f1df80cc150 2024-04-22T01:17:06.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df8068210 con 0x7f1df80cc150 2024-04-22T01:17:06.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.642+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1df400bc80 con 0x7f1df80cc150 2024-04-22T01:17:06.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.646+0000 7f1dedffb640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ddc0678c0 0x7f1ddc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:06.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.646+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7f1df4086000 con 0x7f1df80cc150 2024-04-22T01:17:06.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.646+0000 7f1dfd54c640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ddc0678c0 0x7f1ddc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:06.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.646+0000 7f1dfd54c640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ddc0678c0 0x7f1ddc069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1de80ad000 tx=0x7f1de80a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.650+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1df4056d40 con 0x7f1df80cc150 2024-04-22T01:17:06.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.846+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1df815f510 con 0x7f1df80cc150 2024-04-22T01:17:06.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.846+0000 7f1dedffb640 1 -- 172.21.15.70:0/236098017 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3866 (secure 0 0 0) 0x7f1df4056640 con 0x7f1df80cc150 2024-04-22T01:17:06.849 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:06.849 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1767569310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1767569310},{"type":"v1","addr":"172.21.15.98:6833","nonce":1767569310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/685654503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":685654503},{"type":"v1","addr":"172.21.15.98:6835","nonce":685654503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5919,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/2450783154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2450783154},{"type":"v1","addr":"172.21.15.70:6836","nonce":2450783154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/1421560606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":1421560606},{"type":"v1","addr":"172.21.15.70:6837","nonce":1421560606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-04-22T01:17:06.849 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 27 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ddc0678c0 msgr2=0x7f1ddc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ddc0678c0 0x7f1ddc069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1de80ad000 tx=0x7f1de80a6000 comp rx=0 tx=0).stop 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 msgr2=0x7f1df813c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 0x7f1df813c3b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1df400d800 tx=0x7f1df400dcc0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 shutdown_connections 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ddc0678c0 0x7f1ddc069d80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df81455b0 0x7f1df813c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80cc150 0x7f1df813c3b0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 --2- 172.21.15.70:0/236098017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df8072060 0x7f1df8159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:06.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 >> 172.21.15.70:0/236098017 conn(0x7f1df806b1f0 msgr2=0x7f1df80b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:06.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 shutdown_connections 2024-04-22T01:17:06.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:06.850+0000 7f1dff7b0640 1 -- 172.21.15.70:0/236098017 wait complete. 2024-04-22T01:17:06.867 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:17:06.867 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-22T01:17:06.868 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:17:06.868 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:06.868 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:17:06.868 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-22T01:17:06.868 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:17:06.868 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:06.869 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:17:06.869 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-22T01:17:06.869 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:17:06.869 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:06.869 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:17:06.869 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-22T01:17:06.869 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:17:06.869 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:06.879 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:17:06.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:17:06.880 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:17:06.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:17:06.892 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:17:06.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:17:06.893 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:17:06.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:17:06.922 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:17:06.923 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:06.923 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:17:06.923 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:06.935 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:17:06.935 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:06.936 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:17:06.936 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:06.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-22T01:17:06.990 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:17:06.991 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:17:07.004 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:17:07.007 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:17:07.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 Processor -- start 2024-04-22T01:17:07.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 -- start start 2024-04-22T01:17:07.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 0x7f349c0b9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f349c0b9a50 0x7f349c0b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f349c0b6ec0 con 0x7f349c0b6360 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f349c0b6c90 con 0x7f349c0b9a50 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f34a120c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f349c0b7ad0 con 0x7f349c15ddb0 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.054+0000 7f349b59e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f349c0b6360 0x7f349c1619b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34796/0 (socket says 172.21.15.70:34796) 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 -- 172.21.15.70:0/560827808 learned_addr learned my addr 172.21.15.70:0/560827808 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349ad9d640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 0x7f349c0b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349a59c640 1 --2- 172.21.15.70:0/560827808 >> v2:172.21.15.98:3300/0 conn(0x7f349c0b9a50 0x7f349c0b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803117643 0 0) 0x7f349c0b6ec0 con 0x7f349c0b6360 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3484003530 con 0x7f349c0b6360 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2424658679 0 0) 0x7f3484003530 con 0x7f349c0b6360 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 msgr2=0x7f349c0b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:07.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 0x7f349c0b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 >> v2:172.21.15.98:3300/0 conn(0x7f349c0b9a50 msgr2=0x7f349c0b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 --2- 172.21.15.70:0/560827808 >> v2:172.21.15.98:3300/0 conn(0x7f349c0b9a50 0x7f349c0b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349c0b71a0 con 0x7f349c0b6360 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3494001a00 con 0x7f349c0b6360 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 >> v1:172.21.15.70:6789/0 conn(0x7f349c0b6360 legacy=0x7f349c1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 0x7f3484003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3484003fb0 0x7f3484004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3484004d50 0x7f34840051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f349c0b7ad0 con 0x7f3484004d50 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f348400ba00 con 0x7f3484003fb0 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f348400bc00 con 0x7f349c15ddb0 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3484004d50 0x7f34840051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349ad9d640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 0x7f3484003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349a59c640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3484003fb0 0x7f3484004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 -- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 msgr2=0x7f3484003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:07.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c15ddb0 0x7f3484003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 -- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3484003fb0 msgr2=0x7f3484004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:07.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3484003fb0 0x7f3484004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 -- 172.21.15.70:0/560827808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3484003530 con 0x7f3484004d50 2024-04-22T01:17:07.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3484004d50 0x7f34840051d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f349400a880 tx=0x7f349401d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce1c48260d1f480 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:07.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349400a4a0 con 0x7f3484004d50 2024-04-22T01:17:07.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34940ad630 con 0x7f3484004d50 2024-04-22T01:17:07.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f3499d9b640 1 -- 172.21.15.70:0/560827808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34940adaa0 con 0x7f3484004d50 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f34a120c640 1 -- 172.21.15.70:0/560827808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3484004d50 msgr2=0x7f34840051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f34a120c640 1 --2- 172.21.15.70:0/560827808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3484004d50 0x7f34840051d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f349400a880 tx=0x7f349401d3a0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f349b59e640 1 -- 172.21.15.70:0/560827808 reap_dead start 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f34a120c640 1 -- 172.21.15.70:0/560827808 shutdown_connections 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f34a120c640 1 -- 172.21.15.70:0/560827808 >> 172.21.15.70:0/560827808 conn(0x7f349c072d20 msgr2=0x7f349c073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.058+0000 7f34a120c640 1 -- 172.21.15.70:0/560827808 shutdown_connections 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- 172.21.15.70:0/560827808 wait complete. 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 Processor -- start 2024-04-22T01:17:07.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- start start 2024-04-22T01:17:07.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 0x7f349c0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c0cd280 0x7f349c0cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f349c0cc620 0x7f349c151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f349c15e190 con 0x7f349c0ccea0 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f349c0b7760 con 0x7f349c0cc620 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f349c0b8f70 con 0x7f349c0cd280 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 0x7f349c0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 0x7f349c0c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41866/0 (socket says 172.21.15.70:41866) 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 -- 172.21.15.70:0/2723538136 learned_addr learned my addr 172.21.15.70:0/2723538136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349a59c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c0cd280 0x7f349c0cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349b59e640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f349c0cc620 0x7f349c151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 -- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c0cd280 msgr2=0x7f349c0cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:07.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c0cd280 0x7f349c0cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 -- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f349c0cc620 msgr2=0x7f349c151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f349c0cc620 0x7f349c151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 -- 172.21.15.70:0/2723538136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349c0cada0 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f349ad9d640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 0x7f349c0c95a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f348c0119d0 tx=0x7f348c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c00fb50 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348c00fcf0 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349c0cb040 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c00fea0 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f349c0cb8e0 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.062+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f349c04f510 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.066+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f348c012040 con 0x7f349c0ccea0 2024-04-22T01:17:07.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.066+0000 7f34837fe640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34880678c0 0x7f3488069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:07.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.066+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5335+0+0 (secure 0 0 0) 0x7f348c0861b0 con 0x7f349c0ccea0 2024-04-22T01:17:07.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.066+0000 7f349a59c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34880678c0 0x7f3488069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:07.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.066+0000 7f349a59c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34880678c0 0x7f3488069d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f349c15bfd0 tx=0x7f3490008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:07.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.070+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f348c019030 con 0x7f349c0ccea0 2024-04-22T01:17:07.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-22T01:17:07.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:07.242+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f349c0d18a0 con 0x7f349c0ccea0 2024-04-22T01:17:08.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.338+0000 7f34837fe640 1 -- 172.21.15.70:0/2723538136 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v60) ==== 97+0+0 (secure 0 0 0) 0x7f348c0568d0 con 0x7f349c0ccea0 2024-04-22T01:17:08.339 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34880678c0 msgr2=0x7f3488069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34880678c0 0x7f3488069d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f349c15bfd0 tx=0x7f3490008040 comp rx=0 tx=0).stop 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 msgr2=0x7f349c0c95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 0x7f349c0c95a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f348c0119d0 tx=0x7f348c011e90 comp rx=0 tx=0).stop 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 shutdown_connections 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34880678c0 0x7f3488069d80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f349c0cc620 0x7f349c151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f349c0cd280 0x7f349c0cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 --2- 172.21.15.70:0/2723538136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f349c0ccea0 0x7f349c0c95a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 >> 172.21.15.70:0/2723538136 conn(0x7f349c072d20 msgr2=0x7f349c15d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 shutdown_connections 2024-04-22T01:17:08.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.342+0000 7f34a120c640 1 -- 172.21.15.70:0/2723538136 wait complete. 2024-04-22T01:17:08.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-22T01:17:08.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.482+0000 7fc9eb1cf640 1 Processor -- start 2024-04-22T01:17:08.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.482+0000 7fc9eb1cf640 1 -- start start 2024-04-22T01:17:08.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9eb1cf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9e40aeae0 0x7fc9e40aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9eb1cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e4071b20 0x7fc9e4071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9eb1cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40cf4a0 con 0x7fc9e415a550 2024-04-22T01:17:08.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9eb1cf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e406f010 con 0x7fc9e40aeae0 2024-04-22T01:17:08.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9eb1cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e406f260 con 0x7fc9e4071b20 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e976c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e4071b20 0x7fc9e4071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e8f6b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc9e415a550 0x7fc9e40cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:53974/0 (socket says 172.21.15.70:53974) 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9e40aeae0 0x7fc9e40aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e8f6b640 1 -- 172.21.15.70:0/2688362498 learned_addr learned my addr 172.21.15.70:0/2688362498 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e976c640 1 -- 172.21.15.70:0/2688362498 >> v2:172.21.15.98:3300/0 conn(0x7fc9e40aeae0 msgr2=0x7fc9e40aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e976c640 1 --2- 172.21.15.70:0/2688362498 >> v2:172.21.15.98:3300/0 conn(0x7fc9e40aeae0 0x7fc9e40aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e976c640 1 -- 172.21.15.70:0/2688362498 >> v1:172.21.15.98:6789/0 conn(0x7fc9e415a550 legacy=0x7fc9e40cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.486+0000 7fc9e976c640 1 -- 172.21.15.70:0/2688362498 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e406f540 con 0x7fc9e4071b20 2024-04-22T01:17:08.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e976c640 1 --2- 172.21.15.70:0/2688362498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e4071b20 0x7fc9e4071fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009d90 tx=0x7fc9dc01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a7442c703dc762 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e37fe640 1 -- 172.21.15.70:0/2688362498 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0b2070 con 0x7fc9e4071b20 2024-04-22T01:17:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e37fe640 1 -- 172.21.15.70:0/2688362498 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9dc0a57b0 con 0x7fc9e4071b20 2024-04-22T01:17:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e37fe640 1 -- 172.21.15.70:0/2688362498 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0a5b80 con 0x7fc9e4071b20 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2688362498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e4071b20 msgr2=0x7fc9e4071fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2688362498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e4071b20 0x7fc9e4071fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009d90 tx=0x7fc9dc01c970 comp rx=0 tx=0).stop 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2688362498 shutdown_connections 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2688362498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e4071b20 0x7fc9e4071fd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2688362498 >> v2:172.21.15.98:3300/0 conn(0x7fc9e40aeae0 0x7fc9e40aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2688362498 >> 172.21.15.70:0/2688362498 conn(0x7fc9e406b1f0 msgr2=0x7fc9e406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2688362498 shutdown_connections 2024-04-22T01:17:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2688362498 wait complete. 2024-04-22T01:17:08.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 Processor -- start 2024-04-22T01:17:08.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- start start 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 0x7fc9e4142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e40aeae0 0x7fc9e414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e415a550 0x7fc9e41f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e40b30d0 con 0x7fc9e4071b20 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e40b2f50 con 0x7fc9e40aeae0 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9eb1cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e40b3250 con 0x7fc9e415a550 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e8f6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 0x7fc9e4142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e8f6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 0x7fc9e4142320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41868/0 (socket says 172.21.15.70:41868) 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e8f6b640 1 -- 172.21.15.70:0/2076882400 learned_addr learned my addr 172.21.15.70:0/2076882400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:08.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e976c640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e415a550 0x7fc9e41f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.490+0000 7fc9e3fff640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e40aeae0 0x7fc9e414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e8f6b640 1 -- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e415a550 msgr2=0x7fc9e41f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e8f6b640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e415a550 0x7fc9e41f5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e8f6b640 1 -- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e40aeae0 msgr2=0x7fc9e414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e8f6b640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e40aeae0 0x7fc9e414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e8f6b640 1 -- 172.21.15.70:0/2076882400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e41444c0 con 0x7fc9e4071b20 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e8f6b640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 0x7fc9e4142320 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c370 tx=0x7fc9d800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8017070 con 0x7fc9e4071b20 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d8005cf0 con 0x7fc9e4071b20 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d80072f0 con 0x7fc9e4071b20 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e41447a0 con 0x7fc9e4071b20 2024-04-22T01:17:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e414b4d0 con 0x7fc9e4071b20 2024-04-22T01:17:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e4068210 con 0x7fc9e4071b20 2024-04-22T01:17:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc9d801a070 con 0x7fc9e4071b20 2024-04-22T01:17:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e17fa640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9c80678c0 0x7fc9c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5249+0+0 (secure 0 0 0) 0x7fc9d8084dc0 con 0x7fc9e4071b20 2024-04-22T01:17:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e3fff640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9c80678c0 0x7fc9c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.494+0000 7fc9e3fff640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9c80678c0 0x7fc9c8069d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc9e41523b0 tx=0x7fc9d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:08.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.498+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc9d8055b50 con 0x7fc9e4071b20 2024-04-22T01:17:08.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.702+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc9e40cbd90 con 0x7fc9e4071b20 2024-04-22T01:17:08.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.702+0000 7fc9e17fa640 1 -- 172.21.15.70:0/2076882400 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7fc9d8055450 con 0x7fc9e4071b20 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw==", 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:08.705 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w==", 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.706 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ==", 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:08.707 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA==", 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.708 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA==", 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.709 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA==", 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg==", 2024-04-22T01:17:08.710 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg==", 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.711 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong==", 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ==", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA==", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.713 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ==", 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.714 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw==", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow r", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ==", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:17:08.715 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA==", 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:17:08.716 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw==", 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.717 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm2O5GBRAA/JxaogwD4Hjmtr9eQvMY0g==", 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-22T01:17:08.718 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw==", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg==", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:08.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:08.720 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-22T01:17:08.720 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9c80678c0 msgr2=0x7fc9c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9c80678c0 0x7fc9c8069d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc9e41523b0 tx=0x7fc9d0008040 comp rx=0 tx=0).stop 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 msgr2=0x7fc9e4142320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 0x7fc9e4142320 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c370 tx=0x7fc9d800c830 comp rx=0 tx=0).stop 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 shutdown_connections 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc9c80678c0 0x7fc9c8069d80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9e415a550 0x7fc9e41f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e40aeae0 0x7fc9e414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 --2- 172.21.15.70:0/2076882400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9e4071b20 0x7fc9e4142320 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 >> 172.21.15.70:0/2076882400 conn(0x7fc9e406b1f0 msgr2=0x7fc9e4151830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 shutdown_connections 2024-04-22T01:17:08.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.706+0000 7fc9eb1cf640 1 -- 172.21.15.70:0/2076882400 wait complete. 2024-04-22T01:17:08.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:08.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 Processor -- start 2024-04-22T01:17:08.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 -- start start 2024-04-22T01:17:08.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f71b815dfb0 0x7f71b80b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80d3120 0x7f71b80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b815e390 con 0x7f71b80cea80 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f71b80cf420 con 0x7f71b815dfb0 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.842+0000 7f71bd521640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f71b8161e00 con 0x7f71b80d3120 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f71b815dfb0 0x7f71b80b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f71b815dfb0 0x7f71b80b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41878/0 (socket says 172.21.15.70:41878) 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 -- 172.21.15.70:0/1035373235 learned_addr learned my addr 172.21.15.70:0/1035373235 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b77fe640 1 --2- 172.21.15.70:0/1035373235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80d3120 0x7f71b80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 -- 172.21.15.70:0/1035373235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80d3120 msgr2=0x7f71b80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 --2- 172.21.15.70:0/1035373235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80d3120 0x7f71b80ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 -- 172.21.15.70:0/1035373235 >> v1:172.21.15.70:6789/0 conn(0x7f71b80cea80 legacy=0x7f71b80b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 -- 172.21.15.70:0/1035373235 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b81620e0 con 0x7f71b815dfb0 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b6ffd640 1 --2- 172.21.15.70:0/1035373235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b815dfb0 0x7f71b80b95d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009870 tx=0x7f71a801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ea67f5a6f85e1a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b5ffb640 1 -- 172.21.15.70:0/1035373235 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a80b2070 con 0x7f71b815dfb0 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b5ffb640 1 -- 172.21.15.70:0/1035373235 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a80ad640 con 0x7f71b815dfb0 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71b5ffb640 1 -- 172.21.15.70:0/1035373235 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a80ada10 con 0x7f71b815dfb0 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 -- 172.21.15.70:0/1035373235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b815dfb0 msgr2=0x7f71b80b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 --2- 172.21.15.70:0/1035373235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b815dfb0 0x7f71b80b95d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009870 tx=0x7f71a801c560 comp rx=0 tx=0).stop 2024-04-22T01:17:08.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 -- 172.21.15.70:0/1035373235 shutdown_connections 2024-04-22T01:17:08.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 --2- 172.21.15.70:0/1035373235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80d3120 0x7f71b80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 --2- 172.21.15.70:0/1035373235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b815dfb0 0x7f71b80b95d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 -- 172.21.15.70:0/1035373235 >> 172.21.15.70:0/1035373235 conn(0x7f71b806c9b0 msgr2=0x7f71b806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:08.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 -- 172.21.15.70:0/1035373235 shutdown_connections 2024-04-22T01:17:08.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.846+0000 7f71bd521640 1 -- 172.21.15.70:0/1035373235 wait complete. 2024-04-22T01:17:08.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 Processor -- start 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- start start 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80cea80 0x7f71b80ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 0x7f71b8142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b815dfb0 0x7f71b81432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f71b816add0 con 0x7f71b80d3120 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71b8161a90 con 0x7f71b815dfb0 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f71b816af50 con 0x7f71b80cea80 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 0x7f71b8142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 0x7f71b8142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41882/0 (socket says 172.21.15.70:41882) 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 -- 172.21.15.70:0/51198071 learned_addr learned my addr 172.21.15.70:0/51198071 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b6ffd640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80cea80 0x7f71b80ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 -- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80cea80 msgr2=0x7f71b80ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80cea80 0x7f71b80ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b77fe640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b815dfb0 0x7f71b81432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 -- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b815dfb0 msgr2=0x7f71b81432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b815dfb0 0x7f71b81432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 -- 172.21.15.70:0/51198071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b8156690 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71b67fc640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 0x7f71b8142da0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c3a0 tx=0x7f71ac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac017070 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71ac0052f0 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b80cd860 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac003860 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b80cde20 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b8068210 con 0x7f71b80d3120 2024-04-22T01:17:08.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.850+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f71ac01a070 con 0x7f71b80d3120 2024-04-22T01:17:08.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.854+0000 7f719f7fe640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71a40678f0 0x7f71a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:08.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.854+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5249+0+0 (secure 0 0 0) 0x7f71ac084e20 con 0x7f71b80d3120 2024-04-22T01:17:08.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.854+0000 7f71b6ffd640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71a40678f0 0x7f71a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:08.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.854+0000 7f71b6ffd640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71a40678f0 0x7f71a4069db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f71a80af000 tx=0x7f71a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:08.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:08.858+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f71ac055b20 con 0x7f71b80d3120 2024-04-22T01:17:09.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.002+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71b80d1950 con 0x7f71b80d3120 2024-04-22T01:17:09.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f719f7fe640 1 -- 172.21.15.70:0/51198071 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+13128 (secure 0 0 0) 0x7f71ac055420 con 0x7f71b80d3120 2024-04-22T01:17:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:09.011 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":60,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:08.338404+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71a40678f0 msgr2=0x7f71a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71a40678f0 0x7f71a4069db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f71a80af000 tx=0x7f71a8008040 comp rx=0 tx=0).stop 2024-04-22T01:17:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 msgr2=0x7f71b8142da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 0x7f71b8142da0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c3a0 tx=0x7f71ac00c860 comp rx=0 tx=0).stop 2024-04-22T01:17:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 shutdown_connections 2024-04-22T01:17:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71a40678f0 0x7f71a4069db0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71b815dfb0 0x7f71b81432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71b80d3120 0x7f71b8142da0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 --2- 172.21.15.70:0/51198071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71b80cea80 0x7f71b80ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 >> 172.21.15.70:0/51198071 conn(0x7f71b806c9b0 msgr2=0x7f71b815a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:09.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.010+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 shutdown_connections 2024-04-22T01:17:09.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.014+0000 7f71bd521640 1 -- 172.21.15.70:0/51198071 wait complete. 2024-04-22T01:17:09.030 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:09.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:09.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e546a4640 1 Processor -- start 2024-04-22T01:17:09.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e546a4640 1 -- start start 2024-04-22T01:17:09.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e546a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e4c0cd220 0x7f8e4c166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e546a4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4c15c290 con 0x7f8e4c0b4450 2024-04-22T01:17:09.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e546a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4c15c570 con 0x7f8e4c072280 2024-04-22T01:17:09.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e546a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8e4c0b3a50 con 0x7f8e4c0cd220 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52440640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8e4c0b4450 0x7f8e4c071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34822/0 (socket says 172.21.15.70:34822) 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52440640 1 -- 172.21.15.70:0/1652874156 learned_addr learned my addr 172.21.15.70:0/1652874156 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133348984 0 0) 0x7f8e4c15c290 con 0x7f8e4c0b4450 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e38003530 con 0x7f8e4c0b4450 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52c41640 1 --2- 172.21.15.70:0/1652874156 >> v2:172.21.15.98:3300/0 conn(0x7f8e4c0cd220 0x7f8e4c166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2960398232 0 0) 0x7f8e38003530 con 0x7f8e4c0b4450 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 >> v2:172.21.15.98:3300/0 conn(0x7f8e4c0cd220 msgr2=0x7f8e4c166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 --2- 172.21.15.70:0/1652874156 >> v2:172.21.15.98:3300/0 conn(0x7f8e4c0cd220 0x7f8e4c166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 >> v1:172.21.15.98:6789/0 conn(0x7f8e4c072280 legacy=0x7f8e4c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c15c850 con 0x7f8e4c0b4450 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f8e48001680 con 0x7f8e4c0b4450 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 >> v1:172.21.15.70:6789/0 conn(0x7f8e4c0b4450 legacy=0x7f8e4c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e380039c0 0x7f8e38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e38004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e380051d0 0x7f8e3800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0b3a50 con 0x7f8e38004810 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3800c120 con 0x7f8e380039c0 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3800c320 con 0x7f8e380051d0 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e38004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52440640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e380039c0 0x7f8e38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e51c3f640 1 -- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e380051d0 msgr2=0x7f8e3800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52c41640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e380051d0 0x7f8e3800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e380051d0 0x7f8e3800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e51c3f640 1 -- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e380039c0 msgr2=0x7f8e38003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e380039c0 0x7f8e38003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e51c3f640 1 -- 172.21.15.70:0/1652874156 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e38003530 con 0x7f8e38004810 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52c41640 1 -- 172.21.15.70:0/1652874156 reap_dead start 2024-04-22T01:17:09.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.142+0000 7f8e52440640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e380039c0 0x7f8e38003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e38004c90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001780 tx=0x7f8e3c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80231b7f013a3eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0013a0 con 0x7f8e38004810 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3c0052f0 con 0x7f8e38004810 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e5143e640 1 -- 172.21.15.70:0/1652874156 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c003840 con 0x7f8e38004810 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- 172.21.15.70:0/1652874156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 msgr2=0x7f8e38004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e38004c90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001780 tx=0x7f8e3c01c400 comp rx=0 tx=0).stop 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- 172.21.15.70:0/1652874156 shutdown_connections 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1652874156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e38004c90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- 172.21.15.70:0/1652874156 >> 172.21.15.70:0/1652874156 conn(0x7f8e4c0b1d40 msgr2=0x7f8e4c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- 172.21.15.70:0/1652874156 shutdown_connections 2024-04-22T01:17:09.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- 172.21.15.70:0/1652874156 wait complete. 2024-04-22T01:17:09.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 Processor -- start 2024-04-22T01:17:09.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- start start 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e4c1407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4c14e7b0 0x7f8e4c15e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4c14f270 0x7f8e4c13efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0b4830 con 0x7f8e38004810 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c15d050 con 0x7f8e4c14f270 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e546a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4c0b49b0 con 0x7f8e4c14e7b0 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e4c1407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e4c1407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41900/0 (socket says 172.21.15.70:41900) 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 -- 172.21.15.70:0/1842849741 learned_addr learned my addr 172.21.15.70:0/1842849741 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4c14e7b0 0x7f8e4c15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52c41640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4c14f270 0x7f8e4c13efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 -- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4c14e7b0 msgr2=0x7f8e4c15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4c14e7b0 0x7f8e4c15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 -- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4c14f270 msgr2=0x7f8e4c13efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4c14f270 0x7f8e4c13efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 -- 172.21.15.70:0/1842849741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c13df40 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.146+0000 7f8e52440640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e4c1407e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8e48010b10 tx=0x7f8e480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4801b070 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e48011040 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800ee40 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c13e1e0 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c13e7a0 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4c068210 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8e480143a0 con 0x7f8e38004810 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e42ffd640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e300678f0 0x7f8e30069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e300678f0 0x7f8e30069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5249+0+0 (secure 0 0 0) 0x7f8e48088430 con 0x7f8e38004810 2024-04-22T01:17:09.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.150+0000 7f8e51c3f640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e300678f0 0x7f8e30069db0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c004000 tx=0x7f8e3c0011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.154+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8e480591c0 con 0x7f8e38004810 2024-04-22T01:17:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8e4c0d01e0 con 0x7f8e38004810 2024-04-22T01:17:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e42ffd640 1 -- 172.21.15.70:0/1842849741 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) ==== 93+0+10 (secure 0 0 0) 0x7f8e48058ac0 con 0x7f8e38004810 2024-04-22T01:17:09.319 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e300678f0 msgr2=0x7f8e30069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e300678f0 0x7f8e30069db0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c004000 tx=0x7f8e3c0011a0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 msgr2=0x7f8e4c1407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e4c1407e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8e48010b10 tx=0x7f8e480057f0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 shutdown_connections 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8e300678f0 0x7f8e30069db0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e4c14f270 0x7f8e4c13efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e4c14e7b0 0x7f8e4c15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 --2- 172.21.15.70:0/1842849741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e38004810 0x7f8e4c1407e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 >> 172.21.15.70:0/1842849741 conn(0x7f8e4c0b1d40 msgr2=0x7f8e4c0d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 shutdown_connections 2024-04-22T01:17:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.318+0000 7f8e546a4640 1 -- 172.21.15.70:0/1842849741 wait complete. 2024-04-22T01:17:09.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:17:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.446+0000 7f235ddc6640 1 Processor -- start 2024-04-22T01:17:09.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.446+0000 7f235ddc6640 1 -- start start 2024-04-22T01:17:09.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f235ddc6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23581455b0 0x7f23580cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f235ddc6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f2358071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f235ddc6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2358072060 0x7f235815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f235ddc6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f23580724d0 con 0x7f2358072060 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f235ddc6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f23580d0250 con 0x7f23581455b0 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f235ddc6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f23580cd7b0 con 0x7f23580cc150 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2358072060 0x7f235815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2358072060 0x7f235815f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41914/0 (socket says 172.21.15.70:41914) 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 -- 172.21.15.70:0/2507835969 learned_addr learned my addr 172.21.15.70:0/2507835969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2356ffd640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f2358071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f23577fe640 1 --2- 172.21.15.70:0/2507835969 >> v2:172.21.15.98:3300/0 conn(0x7f23581455b0 0x7f23580cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 -- 172.21.15.70:0/2507835969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 msgr2=0x7f2358071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f2358071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 -- 172.21.15.70:0/2507835969 >> v2:172.21.15.98:3300/0 conn(0x7f23581455b0 msgr2=0x7f23580cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 --2- 172.21.15.70:0/2507835969 >> v2:172.21.15.98:3300/0 conn(0x7f23581455b0 0x7f23580cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 -- 172.21.15.70:0/2507835969 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23580cc710 con 0x7f2358072060 2024-04-22T01:17:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2356ffd640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f2358071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f23577fe640 1 --2- 172.21.15.70:0/2507835969 >> v2:172.21.15.98:3300/0 conn(0x7f23581455b0 0x7f23580cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f2357fff640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f235815f7a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f23580d0700 tx=0x7f234801c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b2f9d28f96ae26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f23567fc640 1 -- 172.21.15.70:0/2507835969 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23480b1070 con 0x7f2358072060 2024-04-22T01:17:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f23567fc640 1 -- 172.21.15.70:0/2507835969 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23480ac440 con 0x7f2358072060 2024-04-22T01:17:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.450+0000 7f23567fc640 1 -- 172.21.15.70:0/2507835969 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23480ac810 con 0x7f2358072060 2024-04-22T01:17:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/2507835969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 msgr2=0x7f235815f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f235815f7a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f23580d0700 tx=0x7f234801c140 comp rx=0 tx=0).stop 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/2507835969 shutdown_connections 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f235815f7a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- 172.21.15.70:0/2507835969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f2358071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- 172.21.15.70:0/2507835969 >> v2:172.21.15.98:3300/0 conn(0x7f23581455b0 0x7f23580cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/2507835969 >> 172.21.15.70:0/2507835969 conn(0x7f235806b1f0 msgr2=0x7f235806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/2507835969 shutdown_connections 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/2507835969 wait complete. 2024-04-22T01:17:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 Processor -- start 2024-04-22T01:17:09.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- start start 2024-04-22T01:17:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f23581383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f23581409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581455b0 0x7f2358140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f235806e380 con 0x7f2358072060 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f235806e200 con 0x7f23581455b0 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f235806e500 con 0x7f23580cc150 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f23581383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f23581383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41922/0 (socket says 172.21.15.70:41922) 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 -- 172.21.15.70:0/3988580904 learned_addr learned my addr 172.21.15.70:0/3988580904 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f2356ffd640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f23581409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 -- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 msgr2=0x7f23581409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f2357fff640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581455b0 0x7f2358140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f23581409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 -- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581455b0 msgr2=0x7f2358140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581455b0 0x7f2358140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 -- 172.21.15.70:0/3988580904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23581425f0 con 0x7f2358072060 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f23577fe640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f23581383c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f235000ce30 tx=0x7f235000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235000f890 con 0x7f2358072060 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f235000fa30 con 0x7f2358072060 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2358138900 con 0x7f2358072060 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235000fe00 con 0x7f2358072060 2024-04-22T01:17:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.454+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2358141830 con 0x7f2358072060 2024-04-22T01:17:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.458+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2358068210 con 0x7f2358072060 2024-04-22T01:17:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.458+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2350018400 con 0x7f2358072060 2024-04-22T01:17:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.458+0000 7f233ffff640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23440678c0 0x7f2344069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.458+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5249+0+0 (secure 0 0 0) 0x7f2350086810 con 0x7f2358072060 2024-04-22T01:17:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.458+0000 7f2356ffd640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23440678c0 0x7f2344069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.458+0000 7f2356ffd640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23440678c0 0x7f2344069d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2358144530 tx=0x7f234c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.462+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2350014030 con 0x7f2358072060 2024-04-22T01:17:09.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.622+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f23580cbb00 con 0x7f2358072060 2024-04-22T01:17:09.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.622+0000 7f233ffff640 1 -- 172.21.15.70:0/3988580904 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v60) ==== 84+0+65 (secure 0 0 0) 0x7f235000ae20 con 0x7f2358072060 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:09.625 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:17:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23440678c0 msgr2=0x7f2344069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23440678c0 0x7f2344069d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2358144530 tx=0x7f234c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 msgr2=0x7f23581383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f23581383c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f235000ce30 tx=0x7f235000a650 comp rx=0 tx=0).stop 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 shutdown_connections 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23440678c0 0x7f2344069d80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23581455b0 0x7f2358140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580cc150 0x7f23581409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 --2- 172.21.15.70:0/3988580904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2358072060 0x7f23581383c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 >> 172.21.15.70:0/3988580904 conn(0x7f235806b1f0 msgr2=0x7f23580744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 shutdown_connections 2024-04-22T01:17:09.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.626+0000 7f235ddc6640 1 -- 172.21.15.70:0/3988580904 wait complete. 2024-04-22T01:17:09.643 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:17:09.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:17:09.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b946aa640 1 Processor -- start 2024-04-22T01:17:09.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b946aa640 1 -- start start 2024-04-22T01:17:09.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b946aa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b8c072280 0x7f1b8c0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b946aa640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b8c15c290 con 0x7f1b8c0cd050 2024-04-22T01:17:09.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b946aa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b8c15c570 con 0x7f1b8c0b4450 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b946aa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1b8c15c850 con 0x7f1b8c072280 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b91c45640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b8c072280 0x7f1b8c0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b92c47640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1b8c0cd050 0x7f1b8c0b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34830/0 (socket says 172.21.15.70:34830) 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b91c45640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b8c072280 0x7f1b8c0ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41934/0 (socket says 172.21.15.70:41934) 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b92c47640 1 -- 172.21.15.70:0/2658871897 learned_addr learned my addr 172.21.15.70:0/2658871897 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b91c45640 1 -- 172.21.15.70:0/2658871897 >> v1:172.21.15.98:6789/0 conn(0x7f1b8c0b4450 legacy=0x7f1b8c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b91c45640 1 -- 172.21.15.70:0/2658871897 >> v1:172.21.15.70:6789/0 conn(0x7f1b8c0cd050 legacy=0x7f1b8c0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b91c45640 1 -- 172.21.15.70:0/2658871897 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8c15cb30 con 0x7f1b8c072280 2024-04-22T01:17:09.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.766+0000 7f1b91444640 1 -- 172.21.15.70:0/2658871897 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133441482 0 0) 0x7f1b8c15c290 con 0x7f1b8c0cd050 2024-04-22T01:17:09.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b91c45640 1 --2- 172.21.15.70:0/2658871897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c072280 0x7f1b8c0ccb10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0097c0 tx=0x7f1b7c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f313668bb118d24b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b91444640 1 -- 172.21.15.70:0/2658871897 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c0b0070 con 0x7f1b8c072280 2024-04-22T01:17:09.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b91444640 1 -- 172.21.15.70:0/2658871897 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7c0ab600 con 0x7f1b8c072280 2024-04-22T01:17:09.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b91444640 1 -- 172.21.15.70:0/2658871897 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c0ab9d0 con 0x7f1b8c072280 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 -- 172.21.15.70:0/2658871897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c072280 msgr2=0x7f1b8c0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 --2- 172.21.15.70:0/2658871897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c072280 0x7f1b8c0ccb10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0097c0 tx=0x7f1b7c01c3d0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 -- 172.21.15.70:0/2658871897 shutdown_connections 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 --2- 172.21.15.70:0/2658871897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c072280 0x7f1b8c0ccb10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 -- 172.21.15.70:0/2658871897 >> 172.21.15.70:0/2658871897 conn(0x7f1b8c0b1d40 msgr2=0x7f1b8c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 -- 172.21.15.70:0/2658871897 shutdown_connections 2024-04-22T01:17:09.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 -- 172.21.15.70:0/2658871897 wait complete. 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 Processor -- start 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.770+0000 7f1b946aa640 1 -- start start 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b8c072280 0x7f1b8c152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 0x7f1b8c1371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b8c0cd050 0x7f1b8c156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c0cd980 con 0x7f1b8c0b4450 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c0cd800 con 0x7f1b8c072280 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1b8c169b20 con 0x7f1b8c0cd050 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 0x7f1b8c1371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 0x7f1b8c1371a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41948/0 (socket says 172.21.15.70:41948) 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 -- 172.21.15.70:0/1829550798 learned_addr learned my addr 172.21.15.70:0/1829550798 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:09.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b92c47640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b8c0cd050 0x7f1b8c156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b92446640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b8c072280 0x7f1b8c152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 -- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b8c0cd050 msgr2=0x7f1b8c156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b8c0cd050 0x7f1b8c156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 -- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b8c072280 msgr2=0x7f1b8c152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b8c072280 0x7f1b8c152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 -- 172.21.15.70:0/1829550798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8c135320 con 0x7f1b8c0b4450 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b91c45640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 0x7f1b8c1371a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0097c0 tx=0x7f1b7c01ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c0bd070 con 0x7f1b8c0b4450 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7c004230 con 0x7f1b8c0b4450 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8c135660 con 0x7f1b8c0b4450 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c0b0040 con 0x7f1b8c0b4450 2024-04-22T01:17:09.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8c135ea0 con 0x7f1b8c0b4450 2024-04-22T01:17:09.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b8c068210 con 0x7f1b8c0b4450 2024-04-22T01:17:09.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b7c0c0070 con 0x7f1b8c0b4450 2024-04-22T01:17:09.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b82ffd640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b700678c0 0x7f1b70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:09.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b92446640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b700678c0 0x7f1b70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:09.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5249+0+0 (secure 0 0 0) 0x7f1b7c11f670 con 0x7f1b8c0b4450 2024-04-22T01:17:09.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.774+0000 7f1b92446640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b700678c0 0x7f1b70069d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1b88004e30 tx=0x7f1b88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:09.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.778+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1b8c135660 con 0x7f1b8c0b4450 2024-04-22T01:17:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:09.942+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1b8c0cc650 con 0x7f1b8c0b4450 2024-04-22T01:17:10.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.342+0000 7f1b82ffd640 1 -- 172.21.15.70:0/1829550798 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v61) ==== 138+0+0 (secure 0 0 0) 0x7f1b7c0f6070 con 0x7f1b8c0b4450 2024-04-22T01:17:10.344 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:17:10.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b700678c0 msgr2=0x7f1b70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b700678c0 0x7f1b70069d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1b88004e30 tx=0x7f1b88009040 comp rx=0 tx=0).stop 2024-04-22T01:17:10.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 msgr2=0x7f1b8c1371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 0x7f1b8c1371a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0097c0 tx=0x7f1b7c01ac10 comp rx=0 tx=0).stop 2024-04-22T01:17:10.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 shutdown_connections 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b700678c0 0x7f1b70069d80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b8c0cd050 0x7f1b8c156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b8c0b4450 0x7f1b8c1371a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 --2- 172.21.15.70:0/1829550798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b8c072280 0x7f1b8c152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 >> 172.21.15.70:0/1829550798 conn(0x7f1b8c0b1d40 msgr2=0x7f1b8c14e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 shutdown_connections 2024-04-22T01:17:10.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.346+0000 7f1b946aa640 1 -- 172.21.15.70:0/1829550798 wait complete. 2024-04-22T01:17:10.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:17:10.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9c49f640 1 Processor -- start 2024-04-22T01:17:10.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9c49f640 1 -- start start 2024-04-22T01:17:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9c49f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 0x7f9c940b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9c49f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c9406d740 con 0x7f9c9415e1e0 2024-04-22T01:17:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9c49f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c9406c980 con 0x7f9c940b5da0 2024-04-22T01:17:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9c49f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9406d2b0 con 0x7f9c940b30e0 2024-04-22T01:17:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9c9415e1e0 0x7f9c940d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34836/0 (socket says 172.21.15.70:34836) 2024-04-22T01:17:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 -- 172.21.15.70:0/126699681 learned_addr learned my addr 172.21.15.70:0/126699681 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:10.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99a3a640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 0x7f9c940b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:10.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650291912 0 0) 0x7f9c9406d740 con 0x7f9c9415e1e0 2024-04-22T01:17:10.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c80003530 con 0x7f9c9415e1e0 2024-04-22T01:17:10.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2516180739 0 0) 0x7f9c80003530 con 0x7f9c9415e1e0 2024-04-22T01:17:10.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 msgr2=0x7f9c940b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:10.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 0x7f9c940b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 >> v1:172.21.15.98:6789/0 conn(0x7f9c940b5da0 legacy=0x7f9c940b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9406cc60 con 0x7f9c9415e1e0 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f9c90001680 con 0x7f9c9415e1e0 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 >> v1:172.21.15.70:6789/0 conn(0x7f9c9415e1e0 legacy=0x7f9c940d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c800039f0 0x7f9c80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 0x7f9c80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c80004d50 0x7f9c800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9406d2b0 con 0x7f9c800039f0 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8000ba80 con 0x7f9c80004d50 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c8000bc80 con 0x7f9c940b30e0 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c800039f0 0x7f9c80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 -- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 msgr2=0x7f9c80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940b30e0 0x7f9c80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 -- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c80004d50 msgr2=0x7f9c800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:10.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c80004d50 0x7f9c800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.498+0000 7f9c9a23b640 1 -- 172.21.15.70:0/126699681 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c80003530 con 0x7f9c800039f0 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c800039f0 0x7f9c80003e10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9c90001ec0 tx=0x7f9c9001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a1ac03e3c38a2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90007d40 con 0x7f9c800039f0 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c900a9070 con 0x7f9c800039f0 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c99239640 1 -- 172.21.15.70:0/126699681 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90005d40 con 0x7f9c800039f0 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 -- 172.21.15.70:0/126699681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c800039f0 msgr2=0x7f9c80003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/126699681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c800039f0 0x7f9c80003e10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9c90001ec0 tx=0x7f9c9001cc60 comp rx=0 tx=0).stop 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9aa3c640 1 -- 172.21.15.70:0/126699681 reap_dead start 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 -- 172.21.15.70:0/126699681 shutdown_connections 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 -- 172.21.15.70:0/126699681 >> 172.21.15.70:0/126699681 conn(0x7f9c94073110 msgr2=0x7f9c940d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 -- 172.21.15.70:0/126699681 shutdown_connections 2024-04-22T01:17:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 -- 172.21.15.70:0/126699681 wait complete. 2024-04-22T01:17:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 Processor -- start 2024-04-22T01:17:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.502+0000 7f9c9c49f640 1 -- start start 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c800039f0 0x7f9c94156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 0x7f9c940cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940cc990 0x7f9c9415d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c940b61b0 con 0x7f9c940caeb0 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c940b7f50 con 0x7f9c800039f0 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c94169ba0 con 0x7f9c940cc990 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 0x7f9c940cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 0x7f9c940cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41964/0 (socket says 172.21.15.70:41964) 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 -- 172.21.15.70:0/3282230349 learned_addr learned my addr 172.21.15.70:0/3282230349 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c800039f0 0x7f9c94156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 -- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940cc990 msgr2=0x7f9c9415d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9aa3c640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940cc990 0x7f9c9415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940cc990 0x7f9c9415d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 -- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c800039f0 msgr2=0x7f9c94156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c800039f0 0x7f9c94156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 -- 172.21.15.70:0/3282230349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c94156b40 con 0x7f9c940caeb0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c800039f0 0x7f9c94156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c99a3a640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 0x7f9c940cc450 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400e780 tx=0x7f9c8400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84017070 con 0x7f9c940caeb0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c8400ee90 con 0x7f9c940caeb0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c940cb3b0 con 0x7f9c940caeb0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8400cb80 con 0x7f9c940caeb0 2024-04-22T01:17:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c940cb970 con 0x7f9c940caeb0 2024-04-22T01:17:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c94068210 con 0x7f9c940caeb0 2024-04-22T01:17:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9c8401a070 con 0x7f9c940caeb0 2024-04-22T01:17:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c8affd640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c780678c0 0x7f9c78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5619+0+0 (secure 0 0 0) 0x7f9c84085ad0 con 0x7f9c940caeb0 2024-04-22T01:17:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.506+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c780678c0 0x7f9c78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.510+0000 7f9c9a23b640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c780678c0 0x7f9c78069d80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9c900ab000 tx=0x7f9c900a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:10.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.514+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9c84056660 con 0x7f9c940caeb0 2024-04-22T01:17:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:10.670+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9c9415ae40 con 0x7f9c940caeb0 2024-04-22T01:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.346+0000 7f9c8affd640 1 -- 172.21.15.70:0/3282230349 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v62) ==== 170+0+0 (secure 0 0 0) 0x7f9c84055f60 con 0x7f9c940caeb0 2024-04-22T01:17:11.349 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c780678c0 msgr2=0x7f9c78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c780678c0 0x7f9c78069d80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9c900ab000 tx=0x7f9c900a7070 comp rx=0 tx=0).stop 2024-04-22T01:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 msgr2=0x7f9c940cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 0x7f9c940cc450 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400e780 tx=0x7f9c8400ec40 comp rx=0 tx=0).stop 2024-04-22T01:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 shutdown_connections 2024-04-22T01:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c780678c0 0x7f9c78069d80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940cc990 0x7f9c9415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c940caeb0 0x7f9c940cc450 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 --2- 172.21.15.70:0/3282230349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c800039f0 0x7f9c94156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.350+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 >> 172.21.15.70:0/3282230349 conn(0x7f9c94073110 msgr2=0x7f9c941583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.354+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 shutdown_connections 2024-04-22T01:17:11.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.358+0000 7f9c9c49f640 1 -- 172.21.15.70:0/3282230349 wait complete. 2024-04-22T01:17:11.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:17:11.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.506+0000 7fc5bae88640 1 Processor -- start 2024-04-22T01:17:11.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.506+0000 7fc5bae88640 1 -- start start 2024-04-22T01:17:11.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5bae88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b40d0cf0 0x7fc5b40d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:11.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5bae88640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc5b40d27f0 0x7fc5b40d2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:11.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5bae88640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b415e520 con 0x7fc5b415e140 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5bae88640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc5b40d16b0 con 0x7fc5b40d27f0 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5bae88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b40b9330 con 0x7fc5b40d0cf0 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b8c24640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc5b415e140 0x7fc5b40cec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34838/0 (socket says 172.21.15.70:34838) 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b8c24640 1 -- 172.21.15.70:0/1182192392 learned_addr learned my addr 172.21.15.70:0/1182192392 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b9425640 1 --2- 172.21.15.70:0/1182192392 >> v2:172.21.15.70:3300/0 conn(0x7fc5b40d27f0 0x7fc5b40d2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b3fff640 1 --2- 172.21.15.70:0/1182192392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b40d0cf0 0x7fc5b40d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b37fe640 1 -- 172.21.15.70:0/1182192392 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194000813 0 0) 0x7fc5b415e520 con 0x7fc5b415e140 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b9425640 1 -- 172.21.15.70:0/1182192392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b40d0cf0 msgr2=0x7fc5b40d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b9425640 1 --2- 172.21.15.70:0/1182192392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b40d0cf0 0x7fc5b40d1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b9425640 1 -- 172.21.15.70:0/1182192392 >> v1:172.21.15.70:6789/0 conn(0x7fc5b415e140 legacy=0x7fc5b40cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:11.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b9425640 1 -- 172.21.15.70:0/1182192392 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b40b8d70 con 0x7fc5b40d27f0 2024-04-22T01:17:11.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b9425640 1 --2- 172.21.15.70:0/1182192392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b40d2c80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009d90 tx=0x7fc5ac01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d698c08332ae5d20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:11.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b37fe640 1 -- 172.21.15.70:0/1182192392 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0b2070 con 0x7fc5b40d27f0 2024-04-22T01:17:11.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b37fe640 1 -- 172.21.15.70:0/1182192392 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac0a5890 con 0x7fc5b40d27f0 2024-04-22T01:17:11.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.510+0000 7fc5b37fe640 1 -- 172.21.15.70:0/1182192392 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0a5c60 con 0x7fc5b40d27f0 2024-04-22T01:17:11.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/1182192392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 msgr2=0x7fc5b40d2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:11.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 --2- 172.21.15.70:0/1182192392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b40d2c80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009d90 tx=0x7fc5ac01c8e0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/1182192392 shutdown_connections 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 --2- 172.21.15.70:0/1182192392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b40d2c80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 --2- 172.21.15.70:0/1182192392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b40d0cf0 0x7fc5b40d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/1182192392 >> 172.21.15.70:0/1182192392 conn(0x7fc5b4072d20 msgr2=0x7fc5b4073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/1182192392 shutdown_connections 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/1182192392 wait complete. 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 Processor -- start 2024-04-22T01:17:11.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- start start 2024-04-22T01:17:11.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b40d0cf0 0x7fc5b41562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:11.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b4154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b415e140 0x7fc5b40d3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b40b8b10 con 0x7fc5b40d27f0 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b40b8990 con 0x7fc5b40d0cf0 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b416ac70 con 0x7fc5b415e140 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b4154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b4154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41980/0 (socket says 172.21.15.70:41980) 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 -- 172.21.15.70:0/2521575479 learned_addr learned my addr 172.21.15.70:0/2521575479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b8c24640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b40d0cf0 0x7fc5b41562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b9425640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b415e140 0x7fc5b40d3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 -- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b415e140 msgr2=0x7fc5b40d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b415e140 0x7fc5b40d3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 -- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b40d0cf0 msgr2=0x7fc5b41562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b40d0cf0 0x7fc5b41562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:11.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 -- 172.21.15.70:0/2521575479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b40cda00 con 0x7fc5b40d27f0 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b9425640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b415e140 0x7fc5b40d3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b8c24640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b40d0cf0 0x7fc5b41562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b3fff640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b4154560 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000d5f0 tx=0x7fc5a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a00125f0 con 0x7fc5b40d27f0 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a0019070 con 0x7fc5b40d27f0 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b40cd440 con 0x7fc5b40d27f0 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a00129b0 con 0x7fc5b40d27f0 2024-04-22T01:17:11.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.514+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b415f610 con 0x7fc5b40d27f0 2024-04-22T01:17:11.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.518+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b4068920 con 0x7fc5b40d27f0 2024-04-22T01:17:11.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.518+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5a0012b50 con 0x7fc5b40d27f0 2024-04-22T01:17:11.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.518+0000 7fc5b17fa640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc5980678c0 0x7fc598069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:11.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.518+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6001+0+0 (secure 0 0 0) 0x7fc5a0086c30 con 0x7fc5b40d27f0 2024-04-22T01:17:11.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.518+0000 7fc5b8c24640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc5980678c0 0x7fc598069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:11.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.518+0000 7fc5b8c24640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc5980678c0 0x7fc598069d80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8000f70 tx=0x7fc5a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:11.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.522+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc5a0057720 con 0x7fc5b40d27f0 2024-04-22T01:17:11.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:11.706+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc5b40b4e10 con 0x7fc5b40d27f0 2024-04-22T01:17:12.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.374+0000 7fc5b17fa640 1 -- 172.21.15.70:0/2521575479 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v29) ==== 173+0+0 (secure 0 0 0) 0x7fc5a0057020 con 0x7fc5b40d27f0 2024-04-22T01:17:12.374 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 9 and data pool 10 2024-04-22T01:17:12.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc5980678c0 msgr2=0x7fc598069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc5980678c0 0x7fc598069d80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8000f70 tx=0x7fc5a8009040 comp rx=0 tx=0).stop 2024-04-22T01:17:12.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 msgr2=0x7fc5b4154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b4154560 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000d5f0 tx=0x7fc5a000dab0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 shutdown_connections 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc5980678c0 0x7fc598069d80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b415e140 0x7fc5b40d3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b40d27f0 0x7fc5b4154560 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 --2- 172.21.15.70:0/2521575479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b40d0cf0 0x7fc5b41562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 >> 172.21.15.70:0/2521575479 conn(0x7fc5b4072d20 msgr2=0x7fc5b40b8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 shutdown_connections 2024-04-22T01:17:12.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.378+0000 7fc5bae88640 1 -- 172.21.15.70:0/2521575479 wait complete. 2024-04-22T01:17:12.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:12.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.538+0000 7f1dc17bf640 1 Processor -- start 2024-04-22T01:17:12.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.538+0000 7f1dc17bf640 1 -- start start 2024-04-22T01:17:12.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dc17bf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1dbc15aa30 0x7f1dbc0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dc17bf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1dbc0aee40 0x7f1dbc0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dc17bf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dbc06dbd0 con 0x7f1dbc15bb50 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dc17bf640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1dbc06de70 con 0x7f1dbc15aa30 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dc17bf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1dbc06e150 con 0x7f1dbc0aee40 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1dbc15aa30 0x7f1dbc0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1dbc15aa30 0x7f1dbc0b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41988/0 (socket says 172.21.15.70:41988) 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 -- 172.21.15.70:0/4028963830 learned_addr learned my addr 172.21.15.70:0/4028963830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbb7fe640 1 --2- 172.21.15.70:0/4028963830 >> v2:172.21.15.98:3300/0 conn(0x7f1dbc0aee40 0x7f1dbc0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 -- 172.21.15.70:0/4028963830 >> v2:172.21.15.98:3300/0 conn(0x7f1dbc0aee40 msgr2=0x7f1dbc0b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 --2- 172.21.15.70:0/4028963830 >> v2:172.21.15.98:3300/0 conn(0x7f1dbc0aee40 0x7f1dbc0b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 -- 172.21.15.70:0/4028963830 >> v1:172.21.15.98:6789/0 conn(0x7f1dbc15bb50 legacy=0x7f1dbc0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 -- 172.21.15.70:0/4028963830 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc06e430 con 0x7f1dbc15aa30 2024-04-22T01:17:12.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbb7fe640 1 --2- 172.21.15.70:0/4028963830 >> v2:172.21.15.98:3300/0 conn(0x7f1dbc0aee40 0x7f1dbc0b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:12.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dbaffd640 1 --2- 172.21.15.70:0/4028963830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc15aa30 0x7f1dbc0b4720 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc06ecd0 tx=0x7f1dac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a352c2248f6d332 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:12.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dba7fc640 1 -- 172.21.15.70:0/4028963830 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac0aa610 con 0x7f1dbc15aa30 2024-04-22T01:17:12.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dba7fc640 1 -- 172.21.15.70:0/4028963830 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dac0af070 con 0x7f1dbc15aa30 2024-04-22T01:17:12.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.542+0000 7f1dba7fc640 1 -- 172.21.15.70:0/4028963830 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac0aa9d0 con 0x7f1dbc15aa30 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/4028963830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc15aa30 msgr2=0x7f1dbc0b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/4028963830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc15aa30 0x7f1dbc0b4720 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc06ecd0 tx=0x7f1dac01c490 comp rx=0 tx=0).stop 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/4028963830 shutdown_connections 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/4028963830 >> v2:172.21.15.98:3300/0 conn(0x7f1dbc0aee40 0x7f1dbc0b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/4028963830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc15aa30 0x7f1dbc0b4720 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/4028963830 >> 172.21.15.70:0/4028963830 conn(0x7f1dbc06b1f0 msgr2=0x7f1dbc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/4028963830 shutdown_connections 2024-04-22T01:17:12.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/4028963830 wait complete. 2024-04-22T01:17:12.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 Processor -- start 2024-04-22T01:17:12.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- start start 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 0x7f1dbc14d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc15aa30 0x7f1dbc1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc15bb50 0x7f1dbc155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc06ead0 con 0x7f1dbc0aee40 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc06e950 con 0x7f1dbc15bb50 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1dbc167a90 con 0x7f1dbc15aa30 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 0x7f1dbc14d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 0x7f1dbc14d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41992/0 (socket says 172.21.15.70:41992) 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 -- 172.21.15.70:0/1267118064 learned_addr learned my addr 172.21.15.70:0/1267118064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1db27fc640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc15aa30 0x7f1dbc1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbb7fe640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc15bb50 0x7f1dbc155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 -- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc15aa30 msgr2=0x7f1dbc1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc15aa30 0x7f1dbc1552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 -- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc15bb50 msgr2=0x7f1dbc155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc15bb50 0x7f1dbc155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 -- 172.21.15.70:0/1267118064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc0b2b90 con 0x7f1dbc0aee40 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbb7fe640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc15bb50 0x7f1dbc155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dbaffd640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 0x7f1dbc14d610 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1dac01a0c0 tx=0x7f1dac01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac0b3800 con 0x7f1dbc0aee40 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dac0b39a0 con 0x7f1dbc0aee40 2024-04-22T01:17:12.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac0b3d90 con 0x7f1dbc0aee40 2024-04-22T01:17:12.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dbc0b2010 con 0x7f1dbc0aee40 2024-04-22T01:17:12.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.546+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dbc0b25d0 con 0x7f1dbc0aee40 2024-04-22T01:17:12.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.550+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dbc068210 con 0x7f1dbc0aee40 2024-04-22T01:17:12.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.550+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1dac0c6390 con 0x7f1dbc0aee40 2024-04-22T01:17:12.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.550+0000 7f1db3fff640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d9c0678c0 0x7f1d9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.550+0000 7f1db27fc640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d9c0678c0 0x7f1d9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.550+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6101+0+0 (secure 0 0 0) 0x7f1dac120500 con 0x7f1dbc0aee40 2024-04-22T01:17:12.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.550+0000 7f1db27fc640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d9c0678c0 0x7f1d9c069d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc1f6f60 tx=0x7f1da4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:12.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.554+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1dac0c3060 con 0x7f1dbc0aee40 2024-04-22T01:17:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.718+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1dbc0aec20 con 0x7f1dbc0aee40 2024-04-22T01:17:12.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1db3fff640 1 -- 172.21.15.70:0/1267118064 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+16861 (secure 0 0 0) 0x7f1dac0a7070 con 0x7f1dbc0aee40 2024-04-22T01:17:12.721 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:12.722 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":63,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:11.711898+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d9c0678c0 msgr2=0x7f1d9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d9c0678c0 0x7f1d9c069d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc1f6f60 tx=0x7f1da4009040 comp rx=0 tx=0).stop 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 msgr2=0x7f1dbc14d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 0x7f1dbc14d610 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1dac01a0c0 tx=0x7f1dac01acd0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 shutdown_connections 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d9c0678c0 0x7f1d9c069d80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc15bb50 0x7f1dbc155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc15aa30 0x7f1dbc1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 --2- 172.21.15.70:0/1267118064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dbc0aee40 0x7f1dbc14d610 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 >> 172.21.15.70:0/1267118064 conn(0x7f1dbc06b1f0 msgr2=0x7f1dbc0cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:12.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 shutdown_connections 2024-04-22T01:17:12.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.722+0000 7f1dc17bf640 1 -- 172.21.15.70:0/1267118064 wait complete. 2024-04-22T01:17:12.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:12.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:17:12.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:17:12.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 Processor -- start 2024-04-22T01:17:12.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 -- start start 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b2415acc0 0x7f1b2415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b2406bd70 0x7f1b2406c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b240af6e0 con 0x7f1b2406c710 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1b2415b5e0 con 0x7f1b2406bd70 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b2bfc5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1b2406cba0 con 0x7f1b2415acc0 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b2406bd70 0x7f1b2406c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b2406bd70 0x7f1b2406c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41996/0 (socket says 172.21.15.70:41996) 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 -- 172.21.15.70:0/1769211571 learned_addr learned my addr 172.21.15.70:0/1769211571 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29d61640 1 --2- 172.21.15.70:0/1769211571 >> v2:172.21.15.98:3300/0 conn(0x7f1b2415acc0 0x7f1b2415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 -- 172.21.15.70:0/1769211571 >> v2:172.21.15.98:3300/0 conn(0x7f1b2415acc0 msgr2=0x7f1b2415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:12.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 --2- 172.21.15.70:0/1769211571 >> v2:172.21.15.98:3300/0 conn(0x7f1b2415acc0 0x7f1b2415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 -- 172.21.15.70:0/1769211571 >> v1:172.21.15.98:6789/0 conn(0x7f1b2406c710 legacy=0x7f1b2406ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:12.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29560640 1 -- 172.21.15.70:0/1769211571 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b240af9c0 con 0x7f1b2406bd70 2024-04-22T01:17:12.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.862+0000 7f1b29d61640 1 --2- 172.21.15.70:0/1769211571 >> v2:172.21.15.98:3300/0 conn(0x7f1b2415acc0 0x7f1b2415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:12.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b29560640 1 --2- 172.21.15.70:0/1769211571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406bd70 0x7f1b2406c1d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1b240b5760 tx=0x7f1b2001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76ad1abae2e690d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:12.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b28d5f640 1 -- 172.21.15.70:0/1769211571 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200ab600 con 0x7f1b2406bd70 2024-04-22T01:17:12.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b28d5f640 1 -- 172.21.15.70:0/1769211571 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b200b0070 con 0x7f1b2406bd70 2024-04-22T01:17:12.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b28d5f640 1 -- 172.21.15.70:0/1769211571 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200ab9c0 con 0x7f1b2406bd70 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/1769211571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406bd70 msgr2=0x7f1b2406c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/1769211571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406bd70 0x7f1b2406c1d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1b240b5760 tx=0x7f1b2001c4f0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/1769211571 shutdown_connections 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/1769211571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406bd70 0x7f1b2406c1d0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/1769211571 >> v2:172.21.15.98:3300/0 conn(0x7f1b2415acc0 0x7f1b2415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/1769211571 >> 172.21.15.70:0/1769211571 conn(0x7f1b240b3d90 msgr2=0x7f1b240b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/1769211571 shutdown_connections 2024-04-22T01:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/1769211571 wait complete. 2024-04-22T01:17:12.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 Processor -- start 2024-04-22T01:17:12.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.866+0000 7f1b2bfc5640 1 -- start start 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b2406bd70 0x7f1b24144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 0x7f1b24149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b2415acc0 0x7f1b24148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24165910 con 0x7f1b2406c710 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24165790 con 0x7f1b2415acc0 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1b24165a90 con 0x7f1b2406bd70 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 0x7f1b24149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 0x7f1b24149e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42010/0 (socket says 172.21.15.70:42010) 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 -- 172.21.15.70:0/2965740598 learned_addr learned my addr 172.21.15.70:0/2965740598 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29d61640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b2406bd70 0x7f1b24144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2a562640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b2415acc0 0x7f1b24148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 -- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b2406bd70 msgr2=0x7f1b24144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b2406bd70 0x7f1b24144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 -- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b2415acc0 msgr2=0x7f1b24148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b2415acc0 0x7f1b24148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 -- 172.21.15.70:0/2965740598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b240b2010 con 0x7f1b2406c710 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2a562640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b2415acc0 0x7f1b24148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29560640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 0x7f1b24149e10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1b20019100 tx=0x7f1b2001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200b4800 con 0x7f1b2406c710 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b200b49c0 con 0x7f1b2406c710 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b240b22b0 con 0x7f1b2406c710 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200b4db0 con 0x7f1b2406c710 2024-04-22T01:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b240b2870 con 0x7f1b2406c710 2024-04-22T01:17:12.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b24068920 con 0x7f1b2406c710 2024-04-22T01:17:12.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b200aa070 con 0x7f1b2406c710 2024-04-22T01:17:12.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b1a7fc640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b080678c0 0x7f1b08069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:12.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6101+0+0 (secure 0 0 0) 0x7f1b201200c0 con 0x7f1b2406c710 2024-04-22T01:17:12.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29d61640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b080678c0 0x7f1b08069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:12.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.870+0000 7f1b29d61640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b080678c0 0x7f1b08069d80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1b14006fd0 tx=0x7f1b14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:12.874+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1b200c4060 con 0x7f1b2406c710 2024-04-22T01:17:13.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b2415b240 con 0x7f1b2406c710 2024-04-22T01:17:13.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b1a7fc640 1 -- 172.21.15.70:0/2965740598 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+16861 (secure 0 0 0) 0x7f1b20016110 con 0x7f1b2406c710 2024-04-22T01:17:13.027 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:13.027 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":63,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:11.711898+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b080678c0 msgr2=0x7f1b08069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b080678c0 0x7f1b08069d80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1b14006fd0 tx=0x7f1b14008040 comp rx=0 tx=0).stop 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 msgr2=0x7f1b24149e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 0x7f1b24149e10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1b20019100 tx=0x7f1b2001ad30 comp rx=0 tx=0).stop 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 shutdown_connections 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b080678c0 0x7f1b08069d80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b2415acc0 0x7f1b24148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b2406c710 0x7f1b24149e10 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 --2- 172.21.15.70:0/2965740598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b2406bd70 0x7f1b24144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 >> 172.21.15.70:0/2965740598 conn(0x7f1b240b3d90 msgr2=0x7f1b240d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 shutdown_connections 2024-04-22T01:17:13.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.026+0000 7f1b2bfc5640 1 -- 172.21.15.70:0/2965740598 wait complete. 2024-04-22T01:17:13.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:17:13.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.158+0000 7f796677d640 1 Processor -- start 2024-04-22T01:17:13.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.158+0000 7f796677d640 1 -- start start 2024-04-22T01:17:13.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7960072280 0x7f79600ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796015c290 con 0x7f79600b4450 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796015c570 con 0x7f79600cd050 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f796015c850 con 0x7f7960072280 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7960072280 0x7f79600ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f79600b4450 0x7f7960071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34852/0 (socket says 172.21.15.70:34852) 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795ffff640 1 -- 172.21.15.70:0/3690859677 learned_addr learned my addr 172.21.15.70:0/3690859677 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:13.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7960072280 0x7f79600ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42020/0 (socket says 172.21.15.70:42020) 2024-04-22T01:17:13.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795f7fe640 1 -- 172.21.15.70:0/3690859677 >> v1:172.21.15.98:6789/0 conn(0x7f79600cd050 legacy=0x7f79600b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795f7fe640 1 -- 172.21.15.70:0/3690859677 >> v1:172.21.15.70:6789/0 conn(0x7f79600b4450 legacy=0x7f7960071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795effd640 1 -- 172.21.15.70:0/3690859677 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272689068 0 0) 0x7f796015c290 con 0x7f79600b4450 2024-04-22T01:17:13.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795f7fe640 1 -- 172.21.15.70:0/3690859677 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796015cb30 con 0x7f7960072280 2024-04-22T01:17:13.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795f7fe640 1 --2- 172.21.15.70:0/3690859677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7960072280 0x7f79600ccb10 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f79540097c0 tx=0x7f795401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d3c01b1885a87e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:13.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795effd640 1 -- 172.21.15.70:0/3690859677 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540b0070 con 0x7f7960072280 2024-04-22T01:17:13.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795effd640 1 -- 172.21.15.70:0/3690859677 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79540ab600 con 0x7f7960072280 2024-04-22T01:17:13.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f795effd640 1 -- 172.21.15.70:0/3690859677 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540ab9d0 con 0x7f7960072280 2024-04-22T01:17:13.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 -- 172.21.15.70:0/3690859677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7960072280 msgr2=0x7f79600ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 --2- 172.21.15.70:0/3690859677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7960072280 0x7f79600ccb10 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f79540097c0 tx=0x7f795401c410 comp rx=0 tx=0).stop 2024-04-22T01:17:13.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 -- 172.21.15.70:0/3690859677 shutdown_connections 2024-04-22T01:17:13.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 --2- 172.21.15.70:0/3690859677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7960072280 0x7f79600ccb10 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.162+0000 7f796677d640 1 -- 172.21.15.70:0/3690859677 >> 172.21.15.70:0/3690859677 conn(0x7f79600b1d40 msgr2=0x7f79600cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:13.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- 172.21.15.70:0/3690859677 shutdown_connections 2024-04-22T01:17:13.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- 172.21.15.70:0/3690859677 wait complete. 2024-04-22T01:17:13.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 Processor -- start 2024-04-22T01:17:13.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- start start 2024-04-22T01:17:13.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7960072280 0x7f79601490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 0x7f79601509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79600cd050 0x7f796006f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79600cd980 con 0x7f79600b4450 2024-04-22T01:17:13.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79600cd800 con 0x7f79600cd050 2024-04-22T01:17:13.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7960169b20 con 0x7f7960072280 2024-04-22T01:17:13.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 0x7f79601509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 0x7f79601509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42032/0 (socket says 172.21.15.70:42032) 2024-04-22T01:17:13.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 -- 172.21.15.70:0/272739050 learned_addr learned my addr 172.21.15.70:0/272739050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:13.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795ffff640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7960072280 0x7f79601490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f7964d1a640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79600cd050 0x7f796006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 -- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7960072280 msgr2=0x7f79601490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7960072280 0x7f79601490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 -- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79600cd050 msgr2=0x7f796006f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79600cd050 0x7f796006f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 -- 172.21.15.70:0/272739050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796014f910 con 0x7f79600b4450 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f7964d1a640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79600cd050 0x7f796006f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795f7fe640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 0x7f79601509b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7954019020 tx=0x7f795401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:13.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540b0040 con 0x7f79600b4450 2024-04-22T01:17:13.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79540b9070 con 0x7f79600b4450 2024-04-22T01:17:13.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540b48a0 con 0x7f79600b4450 2024-04-22T01:17:13.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796014fbf0 con 0x7f79600b4450 2024-04-22T01:17:13.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79601501b0 con 0x7f79600b4450 2024-04-22T01:17:13.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.166+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7960068210 con 0x7f79600b4450 2024-04-22T01:17:13.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.170+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f79540ab4c0 con 0x7f79600b4450 2024-04-22T01:17:13.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.170+0000 7f795cff9640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f794c0678c0 0x7f794c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.170+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6101+0+0 (secure 0 0 0) 0x7f795411fc80 con 0x7f79600b4450 2024-04-22T01:17:13.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.170+0000 7f795ffff640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f794c0678c0 0x7f794c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.170+0000 7f795ffff640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f794c0678c0 0x7f794c069d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7950007fd0 tx=0x7f7950007ba0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:13.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.174+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7954016090 con 0x7f79600b4450 2024-04-22T01:17:13.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.358+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f79600d01e0 con 0x7f79600b4450 2024-04-22T01:17:13.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.402+0000 7f795cff9640 1 -- 172.21.15.70:0/272739050 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) ==== 118+0+0 (secure 0 0 0) 0x7f79540f06c0 con 0x7f79600b4450 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f794c0678c0 msgr2=0x7f794c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f794c0678c0 0x7f794c069d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7950007fd0 tx=0x7f7950007ba0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 msgr2=0x7f79601509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 0x7f79601509b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7954019020 tx=0x7f795401ac50 comp rx=0 tx=0).stop 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 shutdown_connections 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f794c0678c0 0x7f794c069d80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79600cd050 0x7f796006f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79600b4450 0x7f79601509b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 --2- 172.21.15.70:0/272739050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7960072280 0x7f79601490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 >> 172.21.15.70:0/272739050 conn(0x7f79600b1d40 msgr2=0x7f79600cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 shutdown_connections 2024-04-22T01:17:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.406+0000 7f796677d640 1 -- 172.21.15.70:0/272739050 wait complete. 2024-04-22T01:17:13.421 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-22T01:17:13.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:17:13.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff38defd640 1 Processor -- start 2024-04-22T01:17:13.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff38defd640 1 -- start start 2024-04-22T01:17:13.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff38defd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff38815de20 0x7ff3880b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff38defd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3880b75e0 con 0x7ff3880cea10 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff38defd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3880b78c0 con 0x7ff3880b6db0 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff38defd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff3880b7ba0 con 0x7ff38815de20 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff3877fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff38815de20 0x7ff3880b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff387fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff3880cea10 0x7ff3880cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34862/0 (socket says 172.21.15.70:34862) 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff387fff640 1 -- 172.21.15.70:0/4046494482 learned_addr learned my addr 172.21.15.70:0/4046494482 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff3877fe640 1 -- 172.21.15.70:0/4046494482 >> v1:172.21.15.98:6789/0 conn(0x7ff3880b6db0 legacy=0x7ff3880d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff3877fe640 1 -- 172.21.15.70:0/4046494482 >> v1:172.21.15.70:6789/0 conn(0x7ff3880cea10 legacy=0x7ff3880cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff3877fe640 1 -- 172.21.15.70:0/4046494482 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3880b7e80 con 0x7ff38815de20 2024-04-22T01:17:13.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.530+0000 7ff3867fc640 1 -- 172.21.15.70:0/4046494482 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701839436 0 0) 0x7ff3880b75e0 con 0x7ff3880cea10 2024-04-22T01:17:13.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff3877fe640 1 --2- 172.21.15.70:0/4046494482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff38815de20 0x7ff3880b98a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff3780097c0 tx=0x7ff37801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3dfb2a8b72af9d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:13.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff3867fc640 1 -- 172.21.15.70:0/4046494482 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3780b1070 con 0x7ff38815de20 2024-04-22T01:17:13.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff3867fc640 1 -- 172.21.15.70:0/4046494482 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3780ac620 con 0x7ff38815de20 2024-04-22T01:17:13.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff3867fc640 1 -- 172.21.15.70:0/4046494482 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3780ac9f0 con 0x7ff38815de20 2024-04-22T01:17:13.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 -- 172.21.15.70:0/4046494482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff38815de20 msgr2=0x7ff3880b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 --2- 172.21.15.70:0/4046494482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff38815de20 0x7ff3880b98a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff3780097c0 tx=0x7ff37801c430 comp rx=0 tx=0).stop 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 -- 172.21.15.70:0/4046494482 shutdown_connections 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 --2- 172.21.15.70:0/4046494482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff38815de20 0x7ff3880b98a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 -- 172.21.15.70:0/4046494482 >> 172.21.15.70:0/4046494482 conn(0x7ff388072d20 msgr2=0x7ff388073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 -- 172.21.15.70:0/4046494482 shutdown_connections 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 -- 172.21.15.70:0/4046494482 wait complete. 2024-04-22T01:17:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 Processor -- start 2024-04-22T01:17:13.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.534+0000 7ff38defd640 1 -- start start 2024-04-22T01:17:13.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3880b6db0 0x7ff3880cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 0x7ff388151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38815de20 0x7ff3880c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3880b9010 con 0x7ff3880cea10 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3880b8e90 con 0x7ff38815de20 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff3880b9190 con 0x7ff3880b6db0 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 0x7ff388151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 0x7ff388151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42064/0 (socket says 172.21.15.70:42064) 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 -- 172.21.15.70:0/3216832267 learned_addr learned my addr 172.21.15.70:0/3216832267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 -- 172.21.15.70:0/3216832267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3880b6db0 msgr2=0x7ff3880cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3880b6db0 0x7ff3880cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 -- 172.21.15.70:0/3216832267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38815de20 msgr2=0x7ff3880c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38815de20 0x7ff3880c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 -- 172.21.15.70:0/3216832267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3880cdc20 con 0x7ff3880cea10 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff386ffd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 0x7ff388151e60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00c490 tx=0x7ff37c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c017070 con 0x7ff3880cea10 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff37c003600 con 0x7ff3880cea10 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3880c8320 con 0x7ff3880cea10 2024-04-22T01:17:13.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c003a30 con 0x7ff3880cea10 2024-04-22T01:17:13.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3880c88e0 con 0x7ff3880cea10 2024-04-22T01:17:13.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff388068210 con 0x7ff3880cea10 2024-04-22T01:17:13.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff37c01a070 con 0x7ff3880cea10 2024-04-22T01:17:13.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff36ffff640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff3740678c0 0x7ff374069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:13.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7ff37c0852c0 con 0x7ff3880cea10 2024-04-22T01:17:13.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff3877fe640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff3740678c0 0x7ff374069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:13.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.538+0000 7ff3877fe640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff3740678c0 0x7ff374069d80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff3780ae000 tx=0x7ff378008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:13.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.542+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff37c055c70 con 0x7ff3880cea10 2024-04-22T01:17:13.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:13.722+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff3881591e0 con 0x7ff3880cea10 2024-04-22T01:17:13.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-22T01:17:14.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.402+0000 7ff36ffff640 1 -- 172.21.15.70:0/3216832267 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v31) ==== 122+0+0 (secure 0 0 0) 0x7ff37c055570 con 0x7ff3880cea10 2024-04-22T01:17:14.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff3740678c0 msgr2=0x7ff374069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff3740678c0 0x7ff374069d80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff3780ae000 tx=0x7ff378008040 comp rx=0 tx=0).stop 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 msgr2=0x7ff388151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 0x7ff388151e60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00c490 tx=0x7ff37c001e80 comp rx=0 tx=0).stop 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 shutdown_connections 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff3740678c0 0x7ff374069d80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38815de20 0x7ff3880c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3880cea10 0x7ff388151e60 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 --2- 172.21.15.70:0/3216832267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3880b6db0 0x7ff3880cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 >> 172.21.15.70:0/3216832267 conn(0x7ff388072d20 msgr2=0x7ff3880b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 shutdown_connections 2024-04-22T01:17:14.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.406+0000 7ff38defd640 1 -- 172.21.15.70:0/3216832267 wait complete. 2024-04-22T01:17:14.428 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:17:14.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:14.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 Processor -- start 2024-04-22T01:17:14.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 -- start start 2024-04-22T01:17:14.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 0x7fcb08071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb08072060 0x7fcb0806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb080cd7b0 con 0x7fcb081455b0 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcb080cc710 con 0x7fcb08072060 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb10333640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb080cc9f0 con 0x7fcb080cc150 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0e0cf640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcb081455b0 0x7fcb080cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34876/0 (socket says 172.21.15.70:34876) 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0e0cf640 1 -- 172.21.15.70:0/282064216 learned_addr learned my addr 172.21.15.70:0/282064216 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 0x7fcb08071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1804754 0 0) 0x7fcb080cd7b0 con 0x7fcb081455b0 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0e8d0640 1 --2- 172.21.15.70:0/282064216 >> v2:172.21.15.98:3300/0 conn(0x7fcb08072060 0x7fcb0806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaf4003530 con 0x7fcb081455b0 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1166681032 0 0) 0x7fcaf4003530 con 0x7fcb081455b0 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 msgr2=0x7fcb08071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 0x7fcb08071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 >> v2:172.21.15.98:3300/0 conn(0x7fcb08072060 msgr2=0x7fcb0806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 --2- 172.21.15.70:0/282064216 >> v2:172.21.15.98:3300/0 conn(0x7fcb08072060 0x7fcb0806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb080cccd0 con 0x7fcb081455b0 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fcb04001680 con 0x7fcb081455b0 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0e8d0640 1 --2- 172.21.15.70:0/282064216 >> v2:172.21.15.98:3300/0 conn(0x7fcb08072060 0x7fcb0806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.550+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 >> v1:172.21.15.70:6789/0 conn(0x7fcb081455b0 legacy=0x7fcb080cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40039c0 0x7fcaf4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcaf4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 0x7fcaf40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb080cc710 con 0x7fcaf4004810 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb080cc9f0 con 0x7fcaf40039c0 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf400bb30 con 0x7fcb080cc150 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcaf4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 -- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 msgr2=0x7fcaf40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0e8d0640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 0x7fcaf40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0e0cf640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40039c0 0x7fcaf4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080cc150 0x7fcaf40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 -- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40039c0 msgr2=0x7fcaf4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:14.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40039c0 0x7fcaf4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 -- 172.21.15.70:0/282064216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf4003530 con 0x7fcaf4004810 2024-04-22T01:17:14.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0e0cf640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40039c0 0x7fcaf4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcaf4004c90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009ac0 tx=0x7fcaf801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe99c81649f3cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8009690 con 0x7fcaf4004810 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf80a6430 con 0x7fcaf4004810 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0d0cd640 1 -- 172.21.15.70:0/282064216 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80a6800 con 0x7fcaf4004810 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb10333640 1 -- 172.21.15.70:0/282064216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 msgr2=0x7fcaf4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb10333640 1 --2- 172.21.15.70:0/282064216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcaf4004c90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009ac0 tx=0x7fcaf801ca50 comp rx=0 tx=0).stop 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb0e8d0640 1 -- 172.21.15.70:0/282064216 reap_dead start 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb10333640 1 -- 172.21.15.70:0/282064216 shutdown_connections 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb10333640 1 -- 172.21.15.70:0/282064216 >> 172.21.15.70:0/282064216 conn(0x7fcb0806b1f0 msgr2=0x7fcb0806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb10333640 1 -- 172.21.15.70:0/282064216 shutdown_connections 2024-04-22T01:17:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.554+0000 7fcb10333640 1 -- 172.21.15.70:0/282064216 wait complete. 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 Processor -- start 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- start start 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcb081f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080ae290 0x7fcb081fae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb080af6c0 0x7fcb08142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0806f2e0 con 0x7fcaf4004810 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0806f160 con 0x7fcb080af6c0 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb08072470 con 0x7fcb080ae290 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcb081f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcb081f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42092/0 (socket says 172.21.15.70:42092) 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 -- 172.21.15.70:0/546139943 learned_addr learned my addr 172.21.15.70:0/546139943 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080ae290 0x7fcb081fae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e8d0640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb080af6c0 0x7fcb08142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 -- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080ae290 msgr2=0x7fcb081fae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080ae290 0x7fcb081fae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 -- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb080af6c0 msgr2=0x7fcb08142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb080af6c0 0x7fcb08142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 -- 172.21.15.70:0/546139943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb081fa6c0 con 0x7fcaf4004810 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb0e0cf640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcb081f7050 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400d830 tx=0x7fcb0400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04019070 con 0x7fcaf4004810 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb040074e0 con 0x7fcaf4004810 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb081f78d0 con 0x7fcaf4004810 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04004a30 con 0x7fcaf4004810 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb081fa930 con 0x7fcaf4004810 2024-04-22T01:17:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb081fab30 con 0x7fcaf4004810 2024-04-22T01:17:14.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.558+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcb0401c070 con 0x7fcaf4004810 2024-04-22T01:17:14.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.562+0000 7fcafeffd640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcaec0678c0 0x7fcaec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.562+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fcb04085940 con 0x7fcaf4004810 2024-04-22T01:17:14.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.562+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcaec0678c0 0x7fcaec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.562+0000 7fcb0d8ce640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcaec0678c0 0x7fcaec069d80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80ad000 tx=0x7fcaf80b2040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.566+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcb081fab30 con 0x7fcaf4004810 2024-04-22T01:17:14.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.758+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb08068210 con 0x7fcaf4004810 2024-04-22T01:17:14.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcafeffd640 1 -- 172.21.15.70:0/546139943 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5179 (secure 0 0 0) 0x7fcb04056440 con 0x7fcaf4004810 2024-04-22T01:17:14.761 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:14.762 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:11.712206+0000","modified":"2024-04-22T01:17:13.727780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6330},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6330":{"gid":6330,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/4194165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4194165476},{"type":"v1","addr":"172.21.15.70:6837","nonce":4194165476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6330,"qdb_cluster":[6330]},"id":4}]} 2024-04-22T01:17:14.762 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcaec0678c0 msgr2=0x7fcaec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcaec0678c0 0x7fcaec069d80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80ad000 tx=0x7fcaf80b2040 comp rx=0 tx=0).stop 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 msgr2=0x7fcb081f7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcb081f7050 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400d830 tx=0x7fcb0400dcf0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 shutdown_connections 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcaec0678c0 0x7fcaec069d80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb080af6c0 0x7fcb08142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb080ae290 0x7fcb081fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 --2- 172.21.15.70:0/546139943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf4004810 0x7fcb081f7050 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 >> 172.21.15.70:0/546139943 conn(0x7fcb0806b1f0 msgr2=0x7fcb08161a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:14.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 shutdown_connections 2024-04-22T01:17:14.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.762+0000 7fcb10333640 1 -- 172.21.15.70:0/546139943 wait complete. 2024-04-22T01:17:14.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:14.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.894+0000 7f111ded1640 1 Processor -- start 2024-04-22T01:17:14.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.894+0000 7f111ded1640 1 -- start start 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f111ded1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11180aee40 0x7f11180b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f111ded1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111806dbd0 con 0x7f111815aa30 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f111ded1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111806de70 con 0x7f111815bb50 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f111ded1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f111806e150 con 0x7f11180aee40 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f1116ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11180aee40 0x7f11180b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f11177fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f111815aa30 0x7f11180ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34878/0 (socket says 172.21.15.70:34878) 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f11177fe640 1 -- 172.21.15.70:0/2776080701 learned_addr learned my addr 172.21.15.70:0/2776080701 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f1116ffd640 1 -- 172.21.15.70:0/2776080701 >> v1:172.21.15.98:6789/0 conn(0x7f111815bb50 legacy=0x7f11180b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f1116ffd640 1 -- 172.21.15.70:0/2776080701 >> v1:172.21.15.70:6789/0 conn(0x7f111815aa30 legacy=0x7f11180ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f1116ffd640 1 -- 172.21.15.70:0/2776080701 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111806e430 con 0x7f11180aee40 2024-04-22T01:17:14.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f1116ffd640 1 --2- 172.21.15.70:0/2776080701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180b4720 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f11000097c0 tx=0x7f110001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9037f00e53afc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:14.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f11167fc640 1 -- 172.21.15.70:0/2776080701 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000ab680 con 0x7f11180aee40 2024-04-22T01:17:14.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f11167fc640 1 -- 172.21.15.70:0/2776080701 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11000b0070 con 0x7f11180aee40 2024-04-22T01:17:14.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f11167fc640 1 -- 172.21.15.70:0/2776080701 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000aba40 con 0x7f11180aee40 2024-04-22T01:17:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f111ded1640 1 -- 172.21.15.70:0/2776080701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 msgr2=0x7f11180b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.898+0000 7f111ded1640 1 --2- 172.21.15.70:0/2776080701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180b4720 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f11000097c0 tx=0x7f110001c460 comp rx=0 tx=0).stop 2024-04-22T01:17:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2776080701 shutdown_connections 2024-04-22T01:17:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 --2- 172.21.15.70:0/2776080701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180b4720 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2776080701 >> 172.21.15.70:0/2776080701 conn(0x7f111806b1f0 msgr2=0x7f111806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:14.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2776080701 shutdown_connections 2024-04-22T01:17:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2776080701 wait complete. 2024-04-22T01:17:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 Processor -- start 2024-04-22T01:17:14.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- start start 2024-04-22T01:17:14.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111815aa30 0x7f11181505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f111815bb50 0x7f111815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f111806ead0 con 0x7f11180aee40 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f111806e950 con 0x7f111815aa30 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1118168aa0 con 0x7f111815bb50 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42110/0 (socket says 172.21.15.70:42110) 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 -- 172.21.15.70:0/2688011165 learned_addr learned my addr 172.21.15.70:0/2688011165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f1116ffd640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111815aa30 0x7f11181505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 -- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f111815bb50 msgr2=0x7f111815e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f1117fff640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f111815bb50 0x7f111815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f111815bb50 0x7f111815e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 -- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111815aa30 msgr2=0x7f11181505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111815aa30 0x7f11181505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 -- 172.21.15.70:0/2688011165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11180a9a00 con 0x7f11180aee40 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f1117fff640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f111815bb50 0x7f111815e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f1116ffd640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111815aa30 0x7f11181505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f11177fe640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180adf40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f110800cb10 tx=0x7f110800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1108017070 con 0x7f11180aee40 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f110800ac70 con 0x7f11180aee40 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11180a9720 con 0x7f11180aee40 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1108012620 con 0x7f11180aee40 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1118153460 con 0x7f11180aee40 2024-04-22T01:17:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.902+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1118068210 con 0x7f11180aee40 2024-04-22T01:17:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.906+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f110801a070 con 0x7f11180aee40 2024-04-22T01:17:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.906+0000 7f110ffff640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10f80678c0 0x7f10f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.906+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f1108085c10 con 0x7f11180aee40 2024-04-22T01:17:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.906+0000 7f1116ffd640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10f80678c0 0x7f10f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:14.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.906+0000 7f1116ffd640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10f80678c0 0x7f10f8069d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f11000af000 tx=0x7f1100008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:14.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:14.910+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f11080565d0 con 0x7f11180aee40 2024-04-22T01:17:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.058+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f111806f190 con 0x7f11180aee40 2024-04-22T01:17:15.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f110ffff640 1 -- 172.21.15.70:0/2688011165 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16848 (secure 0 0 0) 0x7f1108055ed0 con 0x7f11180aee40 2024-04-22T01:17:15.061 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:15.061 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:13.375919+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10f80678c0 msgr2=0x7f10f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10f80678c0 0x7f10f8069d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f11000af000 tx=0x7f1100008040 comp rx=0 tx=0).stop 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 msgr2=0x7f11180adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180adf40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f110800cb10 tx=0x7f110800a060 comp rx=0 tx=0).stop 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 shutdown_connections 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10f80678c0 0x7f10f8069d80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f111815bb50 0x7f111815e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111815aa30 0x7f11181505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 --2- 172.21.15.70:0/2688011165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11180aee40 0x7f11180adf40 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 >> 172.21.15.70:0/2688011165 conn(0x7f111806b1f0 msgr2=0x7f11180ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 shutdown_connections 2024-04-22T01:17:15.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.062+0000 7f111ded1640 1 -- 172.21.15.70:0/2688011165 wait complete. 2024-04-22T01:17:15.079 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:17:15.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:17:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.194+0000 7fdf97f62640 1 Processor -- start 2024-04-22T01:17:15.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf97f62640 1 -- start start 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf97f62640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf900b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf97f62640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf9015bb50 0x7fdf900ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf97f62640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9006dbd0 con 0x7fdf900aec70 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf97f62640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdf900af0a0 con 0x7fdf9015bb50 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf97f62640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9006de70 con 0x7fdf9015aa30 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf964ff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdf900aec70 0x7fdf9015e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34880/0 (socket says 172.21.15.70:34880) 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf964ff640 1 -- 172.21.15.70:0/2525757454 learned_addr learned my addr 172.21.15.70:0/2525757454 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 --2- 172.21.15.70:0/2525757454 >> v2:172.21.15.98:3300/0 conn(0x7fdf9015bb50 0x7fdf900ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf95cfe640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf900b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1260999644 0 0) 0x7fdf9006dbd0 con 0x7fdf900aec70 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf7c003530 con 0x7fdf900aec70 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4120687703 0 0) 0x7fdf7c003530 con 0x7fdf900aec70 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 msgr2=0x7fdf900b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf900b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 >> v2:172.21.15.98:3300/0 conn(0x7fdf9015bb50 msgr2=0x7fdf900ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 --2- 172.21.15.70:0/2525757454 >> v2:172.21.15.98:3300/0 conn(0x7fdf9015bb50 0x7fdf900ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9006e150 con 0x7fdf900aec70 2024-04-22T01:17:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fdf88001a00 con 0x7fdf900aec70 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 >> v1:172.21.15.70:6789/0 conn(0x7fdf900aec70 legacy=0x7fdf9015e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf7c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf7c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c004d50 0x7fdf7c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900af0a0 con 0x7fdf7c003f40 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9006de70 con 0x7fdf7c004d50 2024-04-22T01:17:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c00b9d0 con 0x7fdf9015aa30 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf7c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf95cfe640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf7c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf964ff640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c004d50 0x7fdf7c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 -- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 msgr2=0x7fdf7c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf7c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 -- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c004d50 msgr2=0x7fdf7c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf7c004d50 0x7fdf7c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf954fd640 1 -- 172.21.15.70:0/2525757454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c003530 con 0x7fdf7c003f40 2024-04-22T01:17:15.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.198+0000 7fdf95cfe640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf9015aa30 0x7fdf7c0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf954fd640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf7c004810 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009ae0 tx=0x7fdf8001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2af584940df942a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800b2070 con 0x7fdf7c003f40 2024-04-22T01:17:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf800a56a0 con 0x7fdf7c003f40 2024-04-22T01:17:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf94cfc640 1 -- 172.21.15.70:0/2525757454 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800096b0 con 0x7fdf7c003f40 2024-04-22T01:17:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- 172.21.15.70:0/2525757454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 msgr2=0x7fdf7c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 --2- 172.21.15.70:0/2525757454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf7c004810 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009ae0 tx=0x7fdf8001ca50 comp rx=0 tx=0).stop 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf964ff640 1 -- 172.21.15.70:0/2525757454 reap_dead start 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- 172.21.15.70:0/2525757454 shutdown_connections 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- 172.21.15.70:0/2525757454 >> 172.21.15.70:0/2525757454 conn(0x7fdf9006b1f0 msgr2=0x7fdf9006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- 172.21.15.70:0/2525757454 shutdown_connections 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- 172.21.15.70:0/2525757454 wait complete. 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 Processor -- start 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- start start 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf90159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf90144ee0 0x7fdf900addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf90071b20 0x7fdf90071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9015bfb0 con 0x7fdf7c003f40 2024-04-22T01:17:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900b4d00 con 0x7fdf90144ee0 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf97f62640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9015ae10 con 0x7fdf90071b20 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf95cfe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf90159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf95cfe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf90159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42130/0 (socket says 172.21.15.70:42130) 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf95cfe640 1 -- 172.21.15.70:0/883482126 learned_addr learned my addr 172.21.15.70:0/883482126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.202+0000 7fdf964ff640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf90071b20 0x7fdf90071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf954fd640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf90144ee0 0x7fdf900addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf95cfe640 1 -- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf90071b20 msgr2=0x7fdf90071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf95cfe640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf90071b20 0x7fdf90071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf95cfe640 1 -- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf90144ee0 msgr2=0x7fdf900addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf95cfe640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf90144ee0 0x7fdf900addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf95cfe640 1 -- 172.21.15.70:0/883482126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf900ae300 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf95cfe640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf90159430 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c010930 tx=0x7fdf8c010df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00f040 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf8c00ec20 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c004020 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf90142e50 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf90145730 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf90068210 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf8c0041c0 con 0x7fdf7c003f40 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf867fc640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf740678f0 0x7fdf74069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fdf8c085cf0 con 0x7fdf7c003f40 2024-04-22T01:17:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf954fd640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf740678f0 0x7fdf74069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.206+0000 7fdf954fd640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf740678f0 0x7fdf74069db0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdf800af000 tx=0x7fdf800046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.210+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdf8c08a070 con 0x7fdf7c003f40 2024-04-22T01:17:15.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.366+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf90152210 con 0x7fdf740678f0 2024-04-22T01:17:15.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.378+0000 7fdf867fc640 1 -- 172.21.15.70:0/883482126 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251983 (secure 0 0 0) 0x7fdf800fa860 con 0x7fdf740678f0 2024-04-22T01:17:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:17:15.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf740678f0 msgr2=0x7fdf74069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf740678f0 0x7fdf74069db0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdf800af000 tx=0x7fdf800046d0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 msgr2=0x7fdf90159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf90159430 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c010930 tx=0x7fdf8c010df0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 shutdown_connections 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdf740678f0 0x7fdf74069db0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf90071b20 0x7fdf90071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf90144ee0 0x7fdf900addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 --2- 172.21.15.70:0/883482126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf7c003f40 0x7fdf90159430 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 >> 172.21.15.70:0/883482126 conn(0x7fdf9006b1f0 msgr2=0x7fdf9015c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 shutdown_connections 2024-04-22T01:17:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.382+0000 7fdf97f62640 1 -- 172.21.15.70:0/883482126 wait complete. 2024-04-22T01:17:15.399 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-22T01:17:14.631706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252484,"kb_used_data":7176,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754722236,"statfs":{"total":773094113280,"available":772835569664,"internally_reserved":0,"allocated":7348224,"data_stored":3925860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.284716"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.425826+0000","last_change":"2024-04-22T01:17:12.425149+0000","last_active":"2024-04-22T01:17:12.425826+0000","last_peered":"2024-04-22T01:17:12.425826+0000","last_clean":"2024-04-22T01:17:12.425826+0000","last_became_active":"2024-04-22T01:17:12.424861+0000","last_became_peered":"2024-04-22T01:17:12.424861+0000","last_unstale":"2024-04-22T01:17:12.425826+0000","last_undegraded":"2024-04-22T01:17:12.425826+0000","last_fullsized":"2024-04-22T01:17:12.425826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:55.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.399729+0000","last_change":"2024-04-22T01:17:12.397295+0000","last_active":"2024-04-22T01:17:12.399729+0000","last_peered":"2024-04-22T01:17:12.399729+0000","last_clean":"2024-04-22T01:17:12.399729+0000","last_became_active":"2024-04-22T01:17:12.397063+0000","last_became_peered":"2024-04-22T01:17:12.397063+0000","last_unstale":"2024-04-22T01:17:12.399729+0000","last_undegraded":"2024-04-22T01:17:12.399729+0000","last_fullsized":"2024-04-22T01:17:12.399729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:23:03.665698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.455754+0000","last_change":"2024-04-22T01:17:12.451979+0000","last_active":"2024-04-22T01:17:12.455754+0000","last_peered":"2024-04-22T01:17:12.455754+0000","last_clean":"2024-04-22T01:17:12.455754+0000","last_became_active":"2024-04-22T01:17:12.448882+0000","last_became_peered":"2024-04-22T01:17:12.448882+0000","last_unstale":"2024-04-22T01:17:12.455754+0000","last_undegraded":"2024-04-22T01:17:12.455754+0000","last_fullsized":"2024-04-22T01:17:12.455754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:08.432611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384837+0000","last_change":"2024-04-22T01:17:12.431824+0000","last_active":"2024-04-22T01:17:13.384837+0000","last_peered":"2024-04-22T01:17:13.384837+0000","last_clean":"2024-04-22T01:17:13.384837+0000","last_became_active":"2024-04-22T01:17:12.431301+0000","last_became_peered":"2024-04-22T01:17:12.431301+0000","last_unstale":"2024-04-22T01:17:13.384837+0000","last_undegraded":"2024-04-22T01:17:13.384837+0000","last_fullsized":"2024-04-22T01:17:13.384837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:23:37.376966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.435813+0000","last_change":"2024-04-22T01:17:12.434550+0000","last_active":"2024-04-22T01:17:12.435813+0000","last_peered":"2024-04-22T01:17:12.435813+0000","last_clean":"2024-04-22T01:17:12.435813+0000","last_became_active":"2024-04-22T01:17:12.434141+0000","last_became_peered":"2024-04-22T01:17:12.434141+0000","last_unstale":"2024-04-22T01:17:12.435813+0000","last_undegraded":"2024-04-22T01:17:12.435813+0000","last_fullsized":"2024-04-22T01:17:12.435813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:37.713812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.447129+0000","last_change":"2024-04-22T01:17:12.445892+0000","last_active":"2024-04-22T01:17:12.447129+0000","last_peered":"2024-04-22T01:17:12.447129+0000","last_clean":"2024-04-22T01:17:12.447129+0000","last_became_active":"2024-04-22T01:17:12.445582+0000","last_became_peered":"2024-04-22T01:17:12.445582+0000","last_unstale":"2024-04-22T01:17:12.447129+0000","last_undegraded":"2024-04-22T01:17:12.447129+0000","last_fullsized":"2024-04-22T01:17:12.447129+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:11.255244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.447359+0000","last_change":"2024-04-22T01:17:12.446590+0000","last_active":"2024-04-22T01:17:12.447359+0000","last_peered":"2024-04-22T01:17:12.447359+0000","last_clean":"2024-04-22T01:17:12.447359+0000","last_became_active":"2024-04-22T01:17:12.446257+0000","last_became_peered":"2024-04-22T01:17:12.446257+0000","last_unstale":"2024-04-22T01:17:12.447359+0000","last_undegraded":"2024-04-22T01:17:12.447359+0000","last_fullsized":"2024-04-22T01:17:12.447359+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:47.783642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382504+0000","last_change":"2024-04-22T01:17:12.441706+0000","last_active":"2024-04-22T01:17:13.382504+0000","last_peered":"2024-04-22T01:17:13.382504+0000","last_clean":"2024-04-22T01:17:13.382504+0000","last_became_active":"2024-04-22T01:17:12.441059+0000","last_became_peered":"2024-04-22T01:17:12.441059+0000","last_unstale":"2024-04-22T01:17:13.382504+0000","last_undegraded":"2024-04-22T01:17:13.382504+0000","last_fullsized":"2024-04-22T01:17:13.382504+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:08.432611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.456666+0000","last_change":"2024-04-22T01:17:12.453778+0000","last_active":"2024-04-22T01:17:12.456666+0000","last_peered":"2024-04-22T01:17:12.456666+0000","last_clean":"2024-04-22T01:17:12.456666+0000","last_became_active":"2024-04-22T01:17:12.453362+0000","last_became_peered":"2024-04-22T01:17:12.453362+0000","last_unstale":"2024-04-22T01:17:12.456666+0000","last_undegraded":"2024-04-22T01:17:12.456666+0000","last_fullsized":"2024-04-22T01:17:12.456666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:04.198934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.454374+0000","last_change":"2024-04-22T01:17:12.445628+0000","last_active":"2024-04-22T01:17:12.454374+0000","last_peered":"2024-04-22T01:17:12.454374+0000","last_clean":"2024-04-22T01:17:12.454374+0000","last_became_active":"2024-04-22T01:17:12.444940+0000","last_became_peered":"2024-04-22T01:17:12.444940+0000","last_unstale":"2024-04-22T01:17:12.454374+0000","last_undegraded":"2024-04-22T01:17:12.454374+0000","last_fullsized":"2024-04-22T01:17:12.454374+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:45.176558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383044+0000","last_change":"2024-04-22T01:17:12.436645+0000","last_active":"2024-04-22T01:17:13.383044+0000","last_peered":"2024-04-22T01:17:13.383044+0000","last_clean":"2024-04-22T01:17:13.383044+0000","last_became_active":"2024-04-22T01:17:12.431232+0000","last_became_peered":"2024-04-22T01:17:12.431232+0000","last_unstale":"2024-04-22T01:17:13.383044+0000","last_undegraded":"2024-04-22T01:17:13.383044+0000","last_fullsized":"2024-04-22T01:17:13.383044+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:47.837694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382833+0000","last_change":"2024-04-22T01:17:12.447917+0000","last_active":"2024-04-22T01:17:13.382833+0000","last_peered":"2024-04-22T01:17:13.382833+0000","last_clean":"2024-04-22T01:17:13.382833+0000","last_became_active":"2024-04-22T01:17:12.447194+0000","last_became_peered":"2024-04-22T01:17:12.447194+0000","last_unstale":"2024-04-22T01:17:13.382833+0000","last_undegraded":"2024-04-22T01:17:13.382833+0000","last_fullsized":"2024-04-22T01:17:13.382833+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:16.497986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383324+0000","last_change":"2024-04-22T01:17:12.428338+0000","last_active":"2024-04-22T01:17:13.383324+0000","last_peered":"2024-04-22T01:17:13.383324+0000","last_clean":"2024-04-22T01:17:13.383324+0000","last_became_active":"2024-04-22T01:17:12.427607+0000","last_became_peered":"2024-04-22T01:17:12.427607+0000","last_unstale":"2024-04-22T01:17:13.383324+0000","last_undegraded":"2024-04-22T01:17:13.383324+0000","last_fullsized":"2024-04-22T01:17:13.383324+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.135714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.454467+0000","last_change":"2024-04-22T01:17:12.449050+0000","last_active":"2024-04-22T01:17:12.454467+0000","last_peered":"2024-04-22T01:17:12.454467+0000","last_clean":"2024-04-22T01:17:12.454467+0000","last_became_active":"2024-04-22T01:17:12.448491+0000","last_became_peered":"2024-04-22T01:17:12.448491+0000","last_unstale":"2024-04-22T01:17:12.454467+0000","last_undegraded":"2024-04-22T01:17:12.454467+0000","last_fullsized":"2024-04-22T01:17:12.454467+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:45.646916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.455714+0000","last_change":"2024-04-22T01:17:12.449689+0000","last_active":"2024-04-22T01:17:12.455714+0000","last_peered":"2024-04-22T01:17:12.455714+0000","last_clean":"2024-04-22T01:17:12.455714+0000","last_became_active":"2024-04-22T01:17:12.448438+0000","last_became_peered":"2024-04-22T01:17:12.448438+0000","last_unstale":"2024-04-22T01:17:12.455714+0000","last_undegraded":"2024-04-22T01:17:12.455714+0000","last_fullsized":"2024-04-22T01:17:12.455714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:23.233992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.411935+0000","last_change":"2024-04-22T01:17:12.410973+0000","last_active":"2024-04-22T01:17:12.411935+0000","last_peered":"2024-04-22T01:17:12.411935+0000","last_clean":"2024-04-22T01:17:12.411935+0000","last_became_active":"2024-04-22T01:17:12.410725+0000","last_became_peered":"2024-04-22T01:17:12.410725+0000","last_unstale":"2024-04-22T01:17:12.411935+0000","last_undegraded":"2024-04-22T01:17:12.411935+0000","last_fullsized":"2024-04-22T01:17:12.411935+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:06.528430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.437154+0000","last_change":"2024-04-22T01:17:12.436208+0000","last_active":"2024-04-22T01:17:12.437154+0000","last_peered":"2024-04-22T01:17:12.437154+0000","last_clean":"2024-04-22T01:17:12.437154+0000","last_became_active":"2024-04-22T01:17:12.435888+0000","last_became_peered":"2024-04-22T01:17:12.435888+0000","last_unstale":"2024-04-22T01:17:12.437154+0000","last_undegraded":"2024-04-22T01:17:12.437154+0000","last_fullsized":"2024-04-22T01:17:12.437154+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:05.219197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383698+0000","last_change":"2024-04-22T01:17:12.431695+0000","last_active":"2024-04-22T01:17:13.383698+0000","last_peered":"2024-04-22T01:17:13.383698+0000","last_clean":"2024-04-22T01:17:13.383698+0000","last_became_active":"2024-04-22T01:17:12.425976+0000","last_became_peered":"2024-04-22T01:17:12.425976+0000","last_unstale":"2024-04-22T01:17:13.383698+0000","last_undegraded":"2024-04-22T01:17:13.383698+0000","last_fullsized":"2024-04-22T01:17:13.383698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:39.139826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413459+0000","last_change":"2024-04-22T01:17:12.403596+0000","last_active":"2024-04-22T01:17:12.413459+0000","last_peered":"2024-04-22T01:17:12.413459+0000","last_clean":"2024-04-22T01:17:12.413459+0000","last_became_active":"2024-04-22T01:17:12.399140+0000","last_became_peered":"2024-04-22T01:17:12.399140+0000","last_unstale":"2024-04-22T01:17:12.413459+0000","last_undegraded":"2024-04-22T01:17:12.413459+0000","last_fullsized":"2024-04-22T01:17:12.413459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:04.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381618+0000","last_change":"2024-04-22T01:17:12.404309+0000","last_active":"2024-04-22T01:17:13.381618+0000","last_peered":"2024-04-22T01:17:13.381618+0000","last_clean":"2024-04-22T01:17:13.381618+0000","last_became_active":"2024-04-22T01:17:12.403970+0000","last_became_peered":"2024-04-22T01:17:12.403970+0000","last_unstale":"2024-04-22T01:17:13.381618+0000","last_undegraded":"2024-04-22T01:17:13.381618+0000","last_fullsized":"2024-04-22T01:17:13.381618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:40.008600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383754+0000","last_change":"2024-04-22T01:17:12.410404+0000","last_active":"2024-04-22T01:17:13.383754+0000","last_peered":"2024-04-22T01:17:13.383754+0000","last_clean":"2024-04-22T01:17:13.383754+0000","last_became_active":"2024-04-22T01:17:12.406685+0000","last_became_peered":"2024-04-22T01:17:12.406685+0000","last_unstale":"2024-04-22T01:17:13.383754+0000","last_undegraded":"2024-04-22T01:17:13.383754+0000","last_fullsized":"2024-04-22T01:17:13.383754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:45.508179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381193+0000","last_change":"2024-04-22T01:17:12.450898+0000","last_active":"2024-04-22T01:17:13.381193+0000","last_peered":"2024-04-22T01:17:13.381193+0000","last_clean":"2024-04-22T01:17:13.381193+0000","last_became_active":"2024-04-22T01:17:12.450630+0000","last_became_peered":"2024-04-22T01:17:12.450630+0000","last_unstale":"2024-04-22T01:17:13.381193+0000","last_undegraded":"2024-04-22T01:17:13.381193+0000","last_fullsized":"2024-04-22T01:17:13.381193+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:34:22.000068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381435+0000","last_change":"2024-04-22T01:17:12.446618+0000","last_active":"2024-04-22T01:17:13.381435+0000","last_peered":"2024-04-22T01:17:13.381435+0000","last_clean":"2024-04-22T01:17:13.381435+0000","last_became_active":"2024-04-22T01:17:12.446244+0000","last_became_peered":"2024-04-22T01:17:12.446244+0000","last_unstale":"2024-04-22T01:17:13.381435+0000","last_undegraded":"2024-04-22T01:17:13.381435+0000","last_fullsized":"2024-04-22T01:17:13.381435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:11.255244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.455781+0000","last_change":"2024-04-22T01:17:12.449970+0000","last_active":"2024-04-22T01:17:12.455781+0000","last_peered":"2024-04-22T01:17:12.455781+0000","last_clean":"2024-04-22T01:17:12.455781+0000","last_became_active":"2024-04-22T01:17:12.445906+0000","last_became_peered":"2024-04-22T01:17:12.445906+0000","last_unstale":"2024-04-22T01:17:12.455781+0000","last_undegraded":"2024-04-22T01:17:12.455781+0000","last_fullsized":"2024-04-22T01:17:12.455781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:35.589064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.385485+0000","last_change":"2024-04-22T01:17:12.409496+0000","last_active":"2024-04-22T01:17:13.385485+0000","last_peered":"2024-04-22T01:17:13.385485+0000","last_clean":"2024-04-22T01:17:13.385485+0000","last_became_active":"2024-04-22T01:17:12.408265+0000","last_became_peered":"2024-04-22T01:17:12.408265+0000","last_unstale":"2024-04-22T01:17:13.385485+0000","last_undegraded":"2024-04-22T01:17:13.385485+0000","last_fullsized":"2024-04-22T01:17:13.385485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:44.887237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383243+0000","last_change":"2024-04-22T01:17:12.419612+0000","last_active":"2024-04-22T01:17:13.383243+0000","last_peered":"2024-04-22T01:17:13.383243+0000","last_clean":"2024-04-22T01:17:13.383243+0000","last_became_active":"2024-04-22T01:17:12.418847+0000","last_became_peered":"2024-04-22T01:17:12.418847+0000","last_unstale":"2024-04-22T01:17:13.383243+0000","last_undegraded":"2024-04-22T01:17:13.383243+0000","last_fullsized":"2024-04-22T01:17:13.383243+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:04.198934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383448+0000","last_change":"2024-04-22T01:17:12.423974+0000","last_active":"2024-04-22T01:17:13.383448+0000","last_peered":"2024-04-22T01:17:13.383448+0000","last_clean":"2024-04-22T01:17:13.383448+0000","last_became_active":"2024-04-22T01:17:12.414842+0000","last_became_peered":"2024-04-22T01:17:12.414842+0000","last_unstale":"2024-04-22T01:17:13.383448+0000","last_undegraded":"2024-04-22T01:17:13.383448+0000","last_fullsized":"2024-04-22T01:17:13.383448+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:00.975780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.407582+0000","last_change":"2024-04-22T01:17:12.407116+0000","last_active":"2024-04-22T01:17:12.407582+0000","last_peered":"2024-04-22T01:17:12.407582+0000","last_clean":"2024-04-22T01:17:12.407582+0000","last_became_active":"2024-04-22T01:17:12.406838+0000","last_became_peered":"2024-04-22T01:17:12.406838+0000","last_unstale":"2024-04-22T01:17:12.407582+0000","last_undegraded":"2024-04-22T01:17:12.407582+0000","last_fullsized":"2024-04-22T01:17:12.407582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:42.942529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.454476+0000","last_change":"2024-04-22T01:17:12.449382+0000","last_active":"2024-04-22T01:17:12.454476+0000","last_peered":"2024-04-22T01:17:12.454476+0000","last_clean":"2024-04-22T01:17:12.454476+0000","last_became_active":"2024-04-22T01:17:12.448775+0000","last_became_peered":"2024-04-22T01:17:12.448775+0000","last_unstale":"2024-04-22T01:17:12.454476+0000","last_undegraded":"2024-04-22T01:17:12.454476+0000","last_fullsized":"2024-04-22T01:17:12.454476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:21.947282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384105+0000","last_change":"2024-04-22T01:17:12.441599+0000","last_active":"2024-04-22T01:17:13.384105+0000","last_peered":"2024-04-22T01:17:13.384105+0000","last_clean":"2024-04-22T01:17:13.384105+0000","last_became_active":"2024-04-22T01:17:12.441166+0000","last_became_peered":"2024-04-22T01:17:12.441166+0000","last_unstale":"2024-04-22T01:17:13.384105+0000","last_undegraded":"2024-04-22T01:17:13.384105+0000","last_fullsized":"2024-04-22T01:17:13.384105+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:11:58.058693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383485+0000","last_change":"2024-04-22T01:17:12.426093+0000","last_active":"2024-04-22T01:17:13.383485+0000","last_peered":"2024-04-22T01:17:13.383485+0000","last_clean":"2024-04-22T01:17:13.383485+0000","last_became_active":"2024-04-22T01:17:12.425411+0000","last_became_peered":"2024-04-22T01:17:12.425411+0000","last_unstale":"2024-04-22T01:17:13.383485+0000","last_undegraded":"2024-04-22T01:17:13.383485+0000","last_fullsized":"2024-04-22T01:17:13.383485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:37.713812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.403993+0000","last_change":"2024-04-22T01:17:12.401593+0000","last_active":"2024-04-22T01:17:12.403993+0000","last_peered":"2024-04-22T01:17:12.403993+0000","last_clean":"2024-04-22T01:17:12.403993+0000","last_became_active":"2024-04-22T01:17:12.401275+0000","last_became_peered":"2024-04-22T01:17:12.401275+0000","last_unstale":"2024-04-22T01:17:12.403993+0000","last_undegraded":"2024-04-22T01:17:12.403993+0000","last_fullsized":"2024-04-22T01:17:12.403993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:34:22.000068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.420224+0000","last_change":"2024-04-22T01:17:12.404597+0000","last_active":"2024-04-22T01:17:12.420224+0000","last_peered":"2024-04-22T01:17:12.420224+0000","last_clean":"2024-04-22T01:17:12.420224+0000","last_became_active":"2024-04-22T01:17:12.403776+0000","last_became_peered":"2024-04-22T01:17:12.403776+0000","last_unstale":"2024-04-22T01:17:12.420224+0000","last_undegraded":"2024-04-22T01:17:12.420224+0000","last_fullsized":"2024-04-22T01:17:12.420224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:07.112109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"63'4","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.453924+0000","last_change":"2024-04-22T01:17:11.398473+0000","last_active":"2024-04-22T01:17:12.453924+0000","last_peered":"2024-04-22T01:17:12.453924+0000","last_clean":"2024-04-22T01:17:12.453924+0000","last_became_active":"2024-04-22T01:17:11.398203+0000","last_became_peered":"2024-04-22T01:17:11.398203+0000","last_unstale":"2024-04-22T01:17:12.453924+0000","last_undegraded":"2024-04-22T01:17:12.453924+0000","last_fullsized":"2024-04-22T01:17:12.453924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:36.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383867+0000","last_change":"2024-04-22T01:17:12.417482+0000","last_active":"2024-04-22T01:17:13.383867+0000","last_peered":"2024-04-22T01:17:13.383867+0000","last_clean":"2024-04-22T01:17:13.383867+0000","last_became_active":"2024-04-22T01:17:12.416481+0000","last_became_peered":"2024-04-22T01:17:12.416481+0000","last_unstale":"2024-04-22T01:17:13.383867+0000","last_undegraded":"2024-04-22T01:17:13.383867+0000","last_fullsized":"2024-04-22T01:17:13.383867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:23:03.665698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.395347+0000","last_change":"2024-04-22T01:17:11.390095+0000","last_active":"2024-04-22T01:17:12.395347+0000","last_peered":"2024-04-22T01:17:12.395347+0000","last_clean":"2024-04-22T01:17:12.395347+0000","last_became_active":"2024-04-22T01:17:11.389816+0000","last_became_peered":"2024-04-22T01:17:11.389816+0000","last_unstale":"2024-04-22T01:17:12.395347+0000","last_undegraded":"2024-04-22T01:17:12.395347+0000","last_fullsized":"2024-04-22T01:17:12.395347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:20.132601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.402524+0000","last_change":"2024-04-22T01:17:12.395511+0000","last_active":"2024-04-22T01:17:12.402524+0000","last_peered":"2024-04-22T01:17:12.402524+0000","last_clean":"2024-04-22T01:17:12.402524+0000","last_became_active":"2024-04-22T01:17:12.395137+0000","last_became_peered":"2024-04-22T01:17:12.395137+0000","last_unstale":"2024-04-22T01:17:12.402524+0000","last_undegraded":"2024-04-22T01:17:12.402524+0000","last_fullsized":"2024-04-22T01:17:12.402524+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:44.887237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.420881+0000","last_change":"2024-04-22T01:17:11.388040+0000","last_active":"2024-04-22T01:17:12.420881+0000","last_peered":"2024-04-22T01:17:12.420881+0000","last_clean":"2024-04-22T01:17:12.420881+0000","last_became_active":"2024-04-22T01:17:11.387716+0000","last_became_peered":"2024-04-22T01:17:11.387716+0000","last_unstale":"2024-04-22T01:17:12.420881+0000","last_undegraded":"2024-04-22T01:17:12.420881+0000","last_fullsized":"2024-04-22T01:17:12.420881+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:44.505065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.456611+0000","last_change":"2024-04-22T01:17:12.452894+0000","last_active":"2024-04-22T01:17:12.456611+0000","last_peered":"2024-04-22T01:17:12.456611+0000","last_clean":"2024-04-22T01:17:12.456611+0000","last_became_active":"2024-04-22T01:17:12.452293+0000","last_became_peered":"2024-04-22T01:17:12.452293+0000","last_unstale":"2024-04-22T01:17:12.456611+0000","last_undegraded":"2024-04-22T01:17:12.456611+0000","last_fullsized":"2024-04-22T01:17:12.456611+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:16.497986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.401544+0000","last_change":"2024-04-22T01:17:11.367406+0000","last_active":"2024-04-22T01:17:12.401544+0000","last_peered":"2024-04-22T01:17:12.401544+0000","last_clean":"2024-04-22T01:17:12.401544+0000","last_became_active":"2024-04-22T01:17:11.367135+0000","last_became_peered":"2024-04-22T01:17:11.367135+0000","last_unstale":"2024-04-22T01:17:12.401544+0000","last_undegraded":"2024-04-22T01:17:12.401544+0000","last_fullsized":"2024-04-22T01:17:12.401544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:03.195821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.399008+0000","last_change":"2024-04-22T01:17:12.398045+0000","last_active":"2024-04-22T01:17:12.399008+0000","last_peered":"2024-04-22T01:17:12.399008+0000","last_clean":"2024-04-22T01:17:12.399008+0000","last_became_active":"2024-04-22T01:17:12.397746+0000","last_became_peered":"2024-04-22T01:17:12.397746+0000","last_unstale":"2024-04-22T01:17:12.399008+0000","last_undegraded":"2024-04-22T01:17:12.399008+0000","last_fullsized":"2024-04-22T01:17:12.399008+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:22.816872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381436+0000","last_change":"2024-04-22T01:17:11.363285+0000","last_active":"2024-04-22T01:17:13.381436+0000","last_peered":"2024-04-22T01:17:13.381436+0000","last_clean":"2024-04-22T01:17:13.381436+0000","last_became_active":"2024-04-22T01:17:11.362547+0000","last_became_peered":"2024-04-22T01:17:11.362547+0000","last_unstale":"2024-04-22T01:17:13.381436+0000","last_undegraded":"2024-04-22T01:17:13.381436+0000","last_fullsized":"2024-04-22T01:17:13.381436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:22.230878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.436904+0000","last_change":"2024-04-22T01:17:12.436067+0000","last_active":"2024-04-22T01:17:12.436904+0000","last_peered":"2024-04-22T01:17:12.436904+0000","last_clean":"2024-04-22T01:17:12.436904+0000","last_became_active":"2024-04-22T01:17:12.435799+0000","last_became_peered":"2024-04-22T01:17:12.435799+0000","last_unstale":"2024-04-22T01:17:12.436904+0000","last_undegraded":"2024-04-22T01:17:12.436904+0000","last_fullsized":"2024-04-22T01:17:12.436904+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:23:37.376966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381427+0000","last_change":"2024-04-22T01:17:11.368766+0000","last_active":"2024-04-22T01:17:13.381427+0000","last_peered":"2024-04-22T01:17:13.381427+0000","last_clean":"2024-04-22T01:17:13.381427+0000","last_became_active":"2024-04-22T01:17:11.368418+0000","last_became_peered":"2024-04-22T01:17:11.368418+0000","last_unstale":"2024-04-22T01:17:13.381427+0000","last_undegraded":"2024-04-22T01:17:13.381427+0000","last_fullsized":"2024-04-22T01:17:13.381427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:15.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423582+0000","last_change":"2024-04-22T01:17:12.415903+0000","last_active":"2024-04-22T01:17:12.423582+0000","last_peered":"2024-04-22T01:17:12.423582+0000","last_clean":"2024-04-22T01:17:12.423582+0000","last_became_active":"2024-04-22T01:17:12.415097+0000","last_became_peered":"2024-04-22T01:17:12.415097+0000","last_unstale":"2024-04-22T01:17:12.423582+0000","last_undegraded":"2024-04-22T01:17:12.423582+0000","last_fullsized":"2024-04-22T01:17:12.423582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:01.275955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381567+0000","last_change":"2024-04-22T01:17:11.377399+0000","last_active":"2024-04-22T01:17:13.381567+0000","last_peered":"2024-04-22T01:17:13.381567+0000","last_clean":"2024-04-22T01:17:13.381567+0000","last_became_active":"2024-04-22T01:17:11.377092+0000","last_became_peered":"2024-04-22T01:17:11.377092+0000","last_unstale":"2024-04-22T01:17:13.381567+0000","last_undegraded":"2024-04-22T01:17:13.381567+0000","last_fullsized":"2024-04-22T01:17:13.381567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:34.585950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381376+0000","last_change":"2024-04-22T01:17:12.427917+0000","last_active":"2024-04-22T01:17:13.381376+0000","last_peered":"2024-04-22T01:17:13.381376+0000","last_clean":"2024-04-22T01:17:13.381376+0000","last_became_active":"2024-04-22T01:17:12.427605+0000","last_became_peered":"2024-04-22T01:17:12.427605+0000","last_unstale":"2024-04-22T01:17:13.381376+0000","last_undegraded":"2024-04-22T01:17:13.381376+0000","last_fullsized":"2024-04-22T01:17:13.381376+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:47.688688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.392753+0000","last_change":"2024-04-22T01:17:11.398614+0000","last_active":"2024-04-22T01:17:12.392753+0000","last_peered":"2024-04-22T01:17:12.392753+0000","last_clean":"2024-04-22T01:17:12.392753+0000","last_became_active":"2024-04-22T01:17:11.397685+0000","last_became_peered":"2024-04-22T01:17:11.397685+0000","last_unstale":"2024-04-22T01:17:12.392753+0000","last_undegraded":"2024-04-22T01:17:12.392753+0000","last_fullsized":"2024-04-22T01:17:12.392753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:48.400105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.401676+0000","last_change":"2024-04-22T01:17:12.399917+0000","last_active":"2024-04-22T01:17:12.401676+0000","last_peered":"2024-04-22T01:17:12.401676+0000","last_clean":"2024-04-22T01:17:12.401676+0000","last_became_active":"2024-04-22T01:17:12.399619+0000","last_became_peered":"2024-04-22T01:17:12.399619+0000","last_unstale":"2024-04-22T01:17:12.401676+0000","last_undegraded":"2024-04-22T01:17:12.401676+0000","last_fullsized":"2024-04-22T01:17:12.401676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:02.844083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.439548+0000","last_change":"2024-04-22T01:17:11.406047+0000","last_active":"2024-04-22T01:17:12.439548+0000","last_peered":"2024-04-22T01:17:12.439548+0000","last_clean":"2024-04-22T01:17:12.439548+0000","last_became_active":"2024-04-22T01:17:11.402475+0000","last_became_peered":"2024-04-22T01:17:11.402475+0000","last_unstale":"2024-04-22T01:17:12.439548+0000","last_undegraded":"2024-04-22T01:17:12.439548+0000","last_fullsized":"2024-04-22T01:17:12.439548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:15:02.280889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381542+0000","last_change":"2024-04-22T01:17:12.411749+0000","last_active":"2024-04-22T01:17:13.381542+0000","last_peered":"2024-04-22T01:17:13.381542+0000","last_clean":"2024-04-22T01:17:13.381542+0000","last_became_active":"2024-04-22T01:17:12.411327+0000","last_became_peered":"2024-04-22T01:17:12.411327+0000","last_unstale":"2024-04-22T01:17:13.381542+0000","last_undegraded":"2024-04-22T01:17:13.381542+0000","last_fullsized":"2024-04-22T01:17:13.381542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:49.794109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.385269+0000","last_change":"2024-04-22T01:17:11.377486+0000","last_active":"2024-04-22T01:17:13.385269+0000","last_peered":"2024-04-22T01:17:13.385269+0000","last_clean":"2024-04-22T01:17:13.385269+0000","last_became_active":"2024-04-22T01:17:11.376647+0000","last_became_peered":"2024-04-22T01:17:11.376647+0000","last_unstale":"2024-04-22T01:17:13.385269+0000","last_undegraded":"2024-04-22T01:17:13.385269+0000","last_fullsized":"2024-04-22T01:17:13.385269+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:43.934103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423738+0000","last_change":"2024-04-22T01:17:12.421524+0000","last_active":"2024-04-22T01:17:12.423738+0000","last_peered":"2024-04-22T01:17:12.423738+0000","last_clean":"2024-04-22T01:17:12.423738+0000","last_became_active":"2024-04-22T01:17:12.420959+0000","last_became_peered":"2024-04-22T01:17:12.420959+0000","last_unstale":"2024-04-22T01:17:12.423738+0000","last_undegraded":"2024-04-22T01:17:12.423738+0000","last_fullsized":"2024-04-22T01:17:12.423738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:47.837694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"63'3","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.448937+0000","last_change":"2024-04-22T01:17:11.405992+0000","last_active":"2024-04-22T01:17:12.448937+0000","last_peered":"2024-04-22T01:17:12.448937+0000","last_clean":"2024-04-22T01:17:12.448937+0000","last_became_active":"2024-04-22T01:17:11.402522+0000","last_became_peered":"2024-04-22T01:17:11.402522+0000","last_unstale":"2024-04-22T01:17:12.448937+0000","last_undegraded":"2024-04-22T01:17:12.448937+0000","last_fullsized":"2024-04-22T01:17:12.448937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:20.718044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384730+0000","last_change":"2024-04-22T01:17:12.432231+0000","last_active":"2024-04-22T01:17:13.384730+0000","last_peered":"2024-04-22T01:17:13.384730+0000","last_clean":"2024-04-22T01:17:13.384730+0000","last_became_active":"2024-04-22T01:17:12.431580+0000","last_became_peered":"2024-04-22T01:17:12.431580+0000","last_unstale":"2024-04-22T01:17:13.384730+0000","last_undegraded":"2024-04-22T01:17:13.384730+0000","last_fullsized":"2024-04-22T01:17:13.384730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:55.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"63'6","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.444036+0000","last_change":"2024-04-22T01:17:11.378192+0000","last_active":"2024-04-22T01:17:12.444036+0000","last_peered":"2024-04-22T01:17:12.444036+0000","last_clean":"2024-04-22T01:17:12.444036+0000","last_became_active":"2024-04-22T01:17:11.377840+0000","last_became_peered":"2024-04-22T01:17:11.377840+0000","last_unstale":"2024-04-22T01:17:12.444036+0000","last_undegraded":"2024-04-22T01:17:12.444036+0000","last_fullsized":"2024-04-22T01:17:12.444036+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:46.834581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382551+0000","last_change":"2024-04-22T01:17:12.452029+0000","last_active":"2024-04-22T01:17:13.382551+0000","last_peered":"2024-04-22T01:17:13.382551+0000","last_clean":"2024-04-22T01:17:13.382551+0000","last_became_active":"2024-04-22T01:17:12.451117+0000","last_became_peered":"2024-04-22T01:17:12.451117+0000","last_unstale":"2024-04-22T01:17:13.382551+0000","last_undegraded":"2024-04-22T01:17:13.382551+0000","last_fullsized":"2024-04-22T01:17:13.382551+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:01.275955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.449842+0000","last_change":"2024-04-22T01:17:11.402025+0000","last_active":"2024-04-22T01:17:12.449842+0000","last_peered":"2024-04-22T01:17:12.449842+0000","last_clean":"2024-04-22T01:17:12.449842+0000","last_became_active":"2024-04-22T01:17:11.401072+0000","last_became_peered":"2024-04-22T01:17:11.401072+0000","last_unstale":"2024-04-22T01:17:12.449842+0000","last_undegraded":"2024-04-22T01:17:12.449842+0000","last_fullsized":"2024-04-22T01:17:12.449842+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:10.582013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382313+0000","last_change":"2024-04-22T01:17:12.408786+0000","last_active":"2024-04-22T01:17:13.382313+0000","last_peered":"2024-04-22T01:17:13.382313+0000","last_clean":"2024-04-22T01:17:13.382313+0000","last_became_active":"2024-04-22T01:17:12.407988+0000","last_became_peered":"2024-04-22T01:17:12.407988+0000","last_unstale":"2024-04-22T01:17:13.382313+0000","last_undegraded":"2024-04-22T01:17:13.382313+0000","last_fullsized":"2024-04-22T01:17:13.382313+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:23.233992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384341+0000","last_change":"2024-04-22T01:17:11.387269+0000","last_active":"2024-04-22T01:17:13.384341+0000","last_peered":"2024-04-22T01:17:13.384341+0000","last_clean":"2024-04-22T01:17:13.384341+0000","last_became_active":"2024-04-22T01:17:11.386551+0000","last_became_peered":"2024-04-22T01:17:11.386551+0000","last_unstale":"2024-04-22T01:17:13.384341+0000","last_undegraded":"2024-04-22T01:17:13.384341+0000","last_fullsized":"2024-04-22T01:17:13.384341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:41.939415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423475+0000","last_change":"2024-04-22T01:17:12.413713+0000","last_active":"2024-04-22T01:17:12.423475+0000","last_peered":"2024-04-22T01:17:12.423475+0000","last_clean":"2024-04-22T01:17:12.423475+0000","last_became_active":"2024-04-22T01:17:12.405307+0000","last_became_peered":"2024-04-22T01:17:12.405307+0000","last_unstale":"2024-04-22T01:17:12.423475+0000","last_undegraded":"2024-04-22T01:17:12.423475+0000","last_fullsized":"2024-04-22T01:17:12.423475+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:49.804868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.432859+0000","last_change":"2024-04-22T01:17:11.402310+0000","last_active":"2024-04-22T01:17:12.432859+0000","last_peered":"2024-04-22T01:17:12.432859+0000","last_clean":"2024-04-22T01:17:12.432859+0000","last_became_active":"2024-04-22T01:17:11.401522+0000","last_became_peered":"2024-04-22T01:17:11.401522+0000","last_unstale":"2024-04-22T01:17:12.432859+0000","last_undegraded":"2024-04-22T01:17:12.432859+0000","last_fullsized":"2024-04-22T01:17:12.432859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:32:22.458266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.442417+0000","last_change":"2024-04-22T01:17:12.442078+0000","last_active":"2024-04-22T01:17:12.442417+0000","last_peered":"2024-04-22T01:17:12.442417+0000","last_clean":"2024-04-22T01:17:12.442417+0000","last_became_active":"2024-04-22T01:17:12.441886+0000","last_became_peered":"2024-04-22T01:17:12.441886+0000","last_unstale":"2024-04-22T01:17:12.442417+0000","last_undegraded":"2024-04-22T01:17:12.442417+0000","last_fullsized":"2024-04-22T01:17:12.442417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:21.721158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"63'8","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382830+0000","last_change":"2024-04-22T01:17:11.406770+0000","last_active":"2024-04-22T01:17:13.382830+0000","last_peered":"2024-04-22T01:17:13.382830+0000","last_clean":"2024-04-22T01:17:13.382830+0000","last_became_active":"2024-04-22T01:17:11.406165+0000","last_became_peered":"2024-04-22T01:17:11.406165+0000","last_unstale":"2024-04-22T01:17:13.382830+0000","last_undegraded":"2024-04-22T01:17:13.382830+0000","last_fullsized":"2024-04-22T01:17:13.382830+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:21:52.705606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.453295+0000","last_change":"2024-04-22T01:17:12.445044+0000","last_active":"2024-04-22T01:17:12.453295+0000","last_peered":"2024-04-22T01:17:12.453295+0000","last_clean":"2024-04-22T01:17:12.453295+0000","last_became_active":"2024-04-22T01:17:12.444794+0000","last_became_peered":"2024-04-22T01:17:12.444794+0000","last_unstale":"2024-04-22T01:17:12.453295+0000","last_undegraded":"2024-04-22T01:17:12.453295+0000","last_fullsized":"2024-04-22T01:17:12.453295+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:23.997157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384859+0000","last_change":"2024-04-22T01:17:11.381706+0000","last_active":"2024-04-22T01:17:13.384859+0000","last_peered":"2024-04-22T01:17:13.384859+0000","last_clean":"2024-04-22T01:17:13.384859+0000","last_became_active":"2024-04-22T01:17:11.380892+0000","last_became_peered":"2024-04-22T01:17:11.380892+0000","last_unstale":"2024-04-22T01:17:13.384859+0000","last_undegraded":"2024-04-22T01:17:13.384859+0000","last_fullsized":"2024-04-22T01:17:13.384859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:23:36.373853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.435769+0000","last_change":"2024-04-22T01:17:12.434071+0000","last_active":"2024-04-22T01:17:12.435769+0000","last_peered":"2024-04-22T01:17:12.435769+0000","last_clean":"2024-04-22T01:17:12.435769+0000","last_became_active":"2024-04-22T01:17:12.432929+0000","last_became_peered":"2024-04-22T01:17:12.432929+0000","last_unstale":"2024-04-22T01:17:12.435769+0000","last_undegraded":"2024-04-22T01:17:12.435769+0000","last_fullsized":"2024-04-22T01:17:12.435769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:00.975780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.448297+0000","last_change":"2024-04-22T01:17:11.405174+0000","last_active":"2024-04-22T01:17:12.448297+0000","last_peered":"2024-04-22T01:17:12.448297+0000","last_clean":"2024-04-22T01:17:12.448297+0000","last_became_active":"2024-04-22T01:17:11.402072+0000","last_became_peered":"2024-04-22T01:17:11.402072+0000","last_unstale":"2024-04-22T01:17:12.448297+0000","last_undegraded":"2024-04-22T01:17:12.448297+0000","last_fullsized":"2024-04-22T01:17:12.448297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:38:53.540328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384076+0000","last_change":"2024-04-22T01:17:12.436550+0000","last_active":"2024-04-22T01:17:13.384076+0000","last_peered":"2024-04-22T01:17:13.384076+0000","last_clean":"2024-04-22T01:17:13.384076+0000","last_became_active":"2024-04-22T01:17:12.426633+0000","last_became_peered":"2024-04-22T01:17:12.426633+0000","last_unstale":"2024-04-22T01:17:13.384076+0000","last_undegraded":"2024-04-22T01:17:13.384076+0000","last_fullsized":"2024-04-22T01:17:13.384076+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:42.942529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.452978+0000","last_change":"2024-04-22T01:17:11.401688+0000","last_active":"2024-04-22T01:17:12.452978+0000","last_peered":"2024-04-22T01:17:12.452978+0000","last_clean":"2024-04-22T01:17:12.452978+0000","last_became_active":"2024-04-22T01:17:11.398306+0000","last_became_peered":"2024-04-22T01:17:11.398306+0000","last_unstale":"2024-04-22T01:17:12.452978+0000","last_undegraded":"2024-04-22T01:17:12.452978+0000","last_fullsized":"2024-04-22T01:17:12.452978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:18.085687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413571+0000","last_change":"2024-04-22T01:17:12.403576+0000","last_active":"2024-04-22T01:17:12.413571+0000","last_peered":"2024-04-22T01:17:12.413571+0000","last_clean":"2024-04-22T01:17:12.413571+0000","last_became_active":"2024-04-22T01:17:12.402846+0000","last_became_peered":"2024-04-22T01:17:12.402846+0000","last_unstale":"2024-04-22T01:17:12.413571+0000","last_undegraded":"2024-04-22T01:17:12.413571+0000","last_fullsized":"2024-04-22T01:17:12.413571+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:24.987567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381665+0000","last_change":"2024-04-22T01:17:11.383746+0000","last_active":"2024-04-22T01:17:13.381665+0000","last_peered":"2024-04-22T01:17:13.381665+0000","last_clean":"2024-04-22T01:17:13.381665+0000","last_became_active":"2024-04-22T01:17:11.383508+0000","last_became_peered":"2024-04-22T01:17:11.383508+0000","last_unstale":"2024-04-22T01:17:13.381665+0000","last_undegraded":"2024-04-22T01:17:13.381665+0000","last_fullsized":"2024-04-22T01:17:13.381665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:20.944168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.393113+0000","last_change":"2024-04-22T01:17:12.391586+0000","last_active":"2024-04-22T01:17:12.393113+0000","last_peered":"2024-04-22T01:17:12.393113+0000","last_clean":"2024-04-22T01:17:12.393113+0000","last_became_active":"2024-04-22T01:17:12.391308+0000","last_became_peered":"2024-04-22T01:17:12.391308+0000","last_unstale":"2024-04-22T01:17:12.393113+0000","last_undegraded":"2024-04-22T01:17:12.393113+0000","last_fullsized":"2024-04-22T01:17:12.393113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:11:58.058693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.415003+0000","last_change":"2024-04-22T01:17:11.381596+0000","last_active":"2024-04-22T01:17:12.415003+0000","last_peered":"2024-04-22T01:17:12.415003+0000","last_clean":"2024-04-22T01:17:12.415003+0000","last_became_active":"2024-04-22T01:17:11.381302+0000","last_became_peered":"2024-04-22T01:17:11.381302+0000","last_unstale":"2024-04-22T01:17:12.415003+0000","last_undegraded":"2024-04-22T01:17:12.415003+0000","last_fullsized":"2024-04-22T01:17:12.415003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:15.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382708+0000","last_change":"2024-04-22T01:17:12.407065+0000","last_active":"2024-04-22T01:17:13.382708+0000","last_peered":"2024-04-22T01:17:13.382708+0000","last_clean":"2024-04-22T01:17:13.382708+0000","last_became_active":"2024-04-22T01:17:12.406287+0000","last_became_peered":"2024-04-22T01:17:12.406287+0000","last_unstale":"2024-04-22T01:17:13.382708+0000","last_undegraded":"2024-04-22T01:17:13.382708+0000","last_fullsized":"2024-04-22T01:17:13.382708+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:35.589064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384547+0000","last_change":"2024-04-22T01:17:11.382007+0000","last_active":"2024-04-22T01:17:13.384547+0000","last_peered":"2024-04-22T01:17:13.384547+0000","last_clean":"2024-04-22T01:17:13.384547+0000","last_became_active":"2024-04-22T01:17:11.379830+0000","last_became_peered":"2024-04-22T01:17:11.379830+0000","last_unstale":"2024-04-22T01:17:13.384547+0000","last_undegraded":"2024-04-22T01:17:13.384547+0000","last_fullsized":"2024-04-22T01:17:13.384547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:54.789380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423536+0000","last_change":"2024-04-22T01:17:12.414281+0000","last_active":"2024-04-22T01:17:12.423536+0000","last_peered":"2024-04-22T01:17:12.423536+0000","last_clean":"2024-04-22T01:17:12.423536+0000","last_became_active":"2024-04-22T01:17:12.399829+0000","last_became_peered":"2024-04-22T01:17:12.399829+0000","last_unstale":"2024-04-22T01:17:12.423536+0000","last_undegraded":"2024-04-22T01:17:12.423536+0000","last_fullsized":"2024-04-22T01:17:12.423536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:52.655050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.386977+0000","last_change":"2024-04-22T01:17:11.390057+0000","last_active":"2024-04-22T01:17:12.386977+0000","last_peered":"2024-04-22T01:17:12.386977+0000","last_clean":"2024-04-22T01:17:12.386977+0000","last_became_active":"2024-04-22T01:17:11.389754+0000","last_became_peered":"2024-04-22T01:17:11.389754+0000","last_unstale":"2024-04-22T01:17:12.386977+0000","last_undegraded":"2024-04-22T01:17:12.386977+0000","last_fullsized":"2024-04-22T01:17:12.386977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:34.585950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.433817+0000","last_change":"2024-04-22T01:17:12.433308+0000","last_active":"2024-04-22T01:17:12.433817+0000","last_peered":"2024-04-22T01:17:12.433817+0000","last_clean":"2024-04-22T01:17:12.433817+0000","last_became_active":"2024-04-22T01:17:12.433031+0000","last_became_peered":"2024-04-22T01:17:12.433031+0000","last_unstale":"2024-04-22T01:17:12.433817+0000","last_undegraded":"2024-04-22T01:17:12.433817+0000","last_fullsized":"2024-04-22T01:17:12.433817+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:47.688688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.385290+0000","last_change":"2024-04-22T01:17:11.391173+0000","last_active":"2024-04-22T01:17:13.385290+0000","last_peered":"2024-04-22T01:17:13.385290+0000","last_clean":"2024-04-22T01:17:13.385290+0000","last_became_active":"2024-04-22T01:17:11.390471+0000","last_became_peered":"2024-04-22T01:17:11.390471+0000","last_unstale":"2024-04-22T01:17:13.385290+0000","last_undegraded":"2024-04-22T01:17:13.385290+0000","last_fullsized":"2024-04-22T01:17:13.385290+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:38:53.540328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.398448+0000","last_change":"2024-04-22T01:17:12.394789+0000","last_active":"2024-04-22T01:17:12.398448+0000","last_peered":"2024-04-22T01:17:12.398448+0000","last_clean":"2024-04-22T01:17:12.398448+0000","last_became_active":"2024-04-22T01:17:12.393902+0000","last_became_peered":"2024-04-22T01:17:12.393902+0000","last_unstale":"2024-04-22T01:17:12.398448+0000","last_undegraded":"2024-04-22T01:17:12.398448+0000","last_fullsized":"2024-04-22T01:17:12.398448+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:02.722073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384043+0000","last_change":"2024-04-22T01:17:11.388136+0000","last_active":"2024-04-22T01:17:13.384043+0000","last_peered":"2024-04-22T01:17:13.384043+0000","last_clean":"2024-04-22T01:17:13.384043+0000","last_became_active":"2024-04-22T01:17:11.387341+0000","last_became_peered":"2024-04-22T01:17:11.387341+0000","last_unstale":"2024-04-22T01:17:13.384043+0000","last_undegraded":"2024-04-22T01:17:13.384043+0000","last_fullsized":"2024-04-22T01:17:13.384043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:04.216084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383634+0000","last_change":"2024-04-22T01:17:12.409872+0000","last_active":"2024-04-22T01:17:13.383634+0000","last_peered":"2024-04-22T01:17:13.383634+0000","last_clean":"2024-04-22T01:17:13.383634+0000","last_became_active":"2024-04-22T01:17:12.408721+0000","last_became_peered":"2024-04-22T01:17:12.408721+0000","last_unstale":"2024-04-22T01:17:13.383634+0000","last_undegraded":"2024-04-22T01:17:13.383634+0000","last_fullsized":"2024-04-22T01:17:13.383634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:54.957068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.388913+0000","last_change":"2024-04-22T01:17:11.387959+0000","last_active":"2024-04-22T01:17:12.388913+0000","last_peered":"2024-04-22T01:17:12.388913+0000","last_clean":"2024-04-22T01:17:12.388913+0000","last_became_active":"2024-04-22T01:17:11.387690+0000","last_became_peered":"2024-04-22T01:17:11.387690+0000","last_unstale":"2024-04-22T01:17:12.388913+0000","last_undegraded":"2024-04-22T01:17:12.388913+0000","last_fullsized":"2024-04-22T01:17:12.388913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:53.953955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413504+0000","last_change":"2024-04-22T01:17:12.403223+0000","last_active":"2024-04-22T01:17:12.413504+0000","last_peered":"2024-04-22T01:17:12.413504+0000","last_clean":"2024-04-22T01:17:12.413504+0000","last_became_active":"2024-04-22T01:17:12.399642+0000","last_became_peered":"2024-04-22T01:17:12.399642+0000","last_unstale":"2024-04-22T01:17:12.413504+0000","last_undegraded":"2024-04-22T01:17:12.413504+0000","last_fullsized":"2024-04-22T01:17:12.413504+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:34.131480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.392545+0000","last_change":"2024-04-22T01:17:11.378152+0000","last_active":"2024-04-22T01:17:12.392545+0000","last_peered":"2024-04-22T01:17:12.392545+0000","last_clean":"2024-04-22T01:17:12.392545+0000","last_became_active":"2024-04-22T01:17:11.377873+0000","last_became_peered":"2024-04-22T01:17:11.377873+0000","last_unstale":"2024-04-22T01:17:12.392545+0000","last_undegraded":"2024-04-22T01:17:12.392545+0000","last_fullsized":"2024-04-22T01:17:12.392545+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:25:59.972666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423887+0000","last_change":"2024-04-22T01:17:12.414769+0000","last_active":"2024-04-22T01:17:12.423887+0000","last_peered":"2024-04-22T01:17:12.423887+0000","last_clean":"2024-04-22T01:17:12.423887+0000","last_became_active":"2024-04-22T01:17:12.412987+0000","last_became_peered":"2024-04-22T01:17:12.412987+0000","last_unstale":"2024-04-22T01:17:12.423887+0000","last_undegraded":"2024-04-22T01:17:12.423887+0000","last_fullsized":"2024-04-22T01:17:12.423887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:48.268764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.434371+0000","last_change":"2024-04-22T01:17:11.401689+0000","last_active":"2024-04-22T01:17:12.434371+0000","last_peered":"2024-04-22T01:17:12.434371+0000","last_clean":"2024-04-22T01:17:12.434371+0000","last_became_active":"2024-04-22T01:17:11.398314+0000","last_became_peered":"2024-04-22T01:17:11.398314+0000","last_unstale":"2024-04-22T01:17:12.434371+0000","last_undegraded":"2024-04-22T01:17:12.434371+0000","last_fullsized":"2024-04-22T01:17:12.434371+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:15:19.727063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383909+0000","last_change":"2024-04-22T01:17:12.453300+0000","last_active":"2024-04-22T01:17:13.383909+0000","last_peered":"2024-04-22T01:17:13.383909+0000","last_clean":"2024-04-22T01:17:13.383909+0000","last_became_active":"2024-04-22T01:17:12.452978+0000","last_became_peered":"2024-04-22T01:17:12.452978+0000","last_unstale":"2024-04-22T01:17:13.383909+0000","last_undegraded":"2024-04-22T01:17:13.383909+0000","last_fullsized":"2024-04-22T01:17:13.383909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:02.844083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381400+0000","last_change":"2024-04-22T01:17:11.364206+0000","last_active":"2024-04-22T01:17:13.381400+0000","last_peered":"2024-04-22T01:17:13.381400+0000","last_clean":"2024-04-22T01:17:13.381400+0000","last_became_active":"2024-04-22T01:17:11.363927+0000","last_became_peered":"2024-04-22T01:17:11.363927+0000","last_unstale":"2024-04-22T01:17:13.381400+0000","last_undegraded":"2024-04-22T01:17:13.381400+0000","last_fullsized":"2024-04-22T01:17:13.381400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:03.195821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.437783+0000","last_change":"2024-04-22T01:17:12.437352+0000","last_active":"2024-04-22T01:17:12.437783+0000","last_peered":"2024-04-22T01:17:12.437783+0000","last_clean":"2024-04-22T01:17:12.437783+0000","last_became_active":"2024-04-22T01:17:12.437117+0000","last_became_peered":"2024-04-22T01:17:12.437117+0000","last_unstale":"2024-04-22T01:17:12.437783+0000","last_undegraded":"2024-04-22T01:17:12.437783+0000","last_fullsized":"2024-04-22T01:17:12.437783+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:38:54.543441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381567+0000","last_change":"2024-04-22T01:17:11.379804+0000","last_active":"2024-04-22T01:17:13.381567+0000","last_peered":"2024-04-22T01:17:13.381567+0000","last_clean":"2024-04-22T01:17:13.381567+0000","last_became_active":"2024-04-22T01:17:11.379535+0000","last_became_peered":"2024-04-22T01:17:11.379535+0000","last_unstale":"2024-04-22T01:17:13.381567+0000","last_undegraded":"2024-04-22T01:17:13.381567+0000","last_fullsized":"2024-04-22T01:17:13.381567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:07.429497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413049+0000","last_change":"2024-04-22T01:17:12.412556+0000","last_active":"2024-04-22T01:17:12.413049+0000","last_peered":"2024-04-22T01:17:12.413049+0000","last_clean":"2024-04-22T01:17:12.413049+0000","last_became_active":"2024-04-22T01:17:12.412350+0000","last_became_peered":"2024-04-22T01:17:12.412350+0000","last_unstale":"2024-04-22T01:17:12.413049+0000","last_undegraded":"2024-04-22T01:17:12.413049+0000","last_fullsized":"2024-04-22T01:17:12.413049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:24.052822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.393744+0000","last_change":"2024-04-22T01:17:11.373985+0000","last_active":"2024-04-22T01:17:12.393744+0000","last_peered":"2024-04-22T01:17:12.393744+0000","last_clean":"2024-04-22T01:17:12.393744+0000","last_became_active":"2024-04-22T01:17:11.373409+0000","last_became_peered":"2024-04-22T01:17:11.373409+0000","last_unstale":"2024-04-22T01:17:12.393744+0000","last_undegraded":"2024-04-22T01:17:12.393744+0000","last_fullsized":"2024-04-22T01:17:12.393744+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:38.136712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423730+0000","last_change":"2024-04-22T01:17:12.413141+0000","last_active":"2024-04-22T01:17:12.423730+0000","last_peered":"2024-04-22T01:17:12.423730+0000","last_clean":"2024-04-22T01:17:12.423730+0000","last_became_active":"2024-04-22T01:17:12.404399+0000","last_became_peered":"2024-04-22T01:17:12.404399+0000","last_unstale":"2024-04-22T01:17:12.423730+0000","last_undegraded":"2024-04-22T01:17:12.423730+0000","last_fullsized":"2024-04-22T01:17:12.423730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:56.892456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381029+0000","last_change":"2024-04-22T01:17:11.363253+0000","last_active":"2024-04-22T01:17:13.381029+0000","last_peered":"2024-04-22T01:17:13.381029+0000","last_clean":"2024-04-22T01:17:13.381029+0000","last_became_active":"2024-04-22T01:17:11.362658+0000","last_became_peered":"2024-04-22T01:17:11.362658+0000","last_unstale":"2024-04-22T01:17:13.381029+0000","last_undegraded":"2024-04-22T01:17:13.381029+0000","last_fullsized":"2024-04-22T01:17:13.381029+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:44.643803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":245,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.421135+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:17:12.421135+0000","last_peered":"2024-04-22T01:17:12.421135+0000","last_clean":"2024-04-22T01:17:12.421135+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:17:12.421135+0000","last_undegraded":"2024-04-22T01:17:12.421135+0000","last_fullsized":"2024-04-22T01:17:12.421135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1372160,"data_stored":933682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":342572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":765952,"data_stored":342560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":344150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":342538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36088,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335752,"statfs":{"total":96636764160,"available":96599810048,"internally_reserved":0,"allocated":761856,"data_stored":342538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":344914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":932906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:17:15.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:17:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.554+0000 7f126d2c0640 1 Processor -- start 2024-04-22T01:17:15.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.554+0000 7f126d2c0640 1 -- start start 2024-04-22T01:17:15.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f126d2c0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f12681455b0 0x7f12680cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f126d2c0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 0x7f1268071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f126d2c0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12680d0250 con 0x7f1268072060 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f126d2c0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f12680724d0 con 0x7f12681455b0 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f126d2c0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f12680cd7b0 con 0x7f12680cc150 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f12677fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1268072060 0x7f126815f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34896/0 (socket says 172.21.15.70:34896) 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f12677fe640 1 -- 172.21.15.70:0/1556476362 learned_addr learned my addr 172.21.15.70:0/1556476362 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f12667fc640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 0x7f1268071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 --2- 172.21.15.70:0/1556476362 >> v2:172.21.15.98:3300/0 conn(0x7f12681455b0 0x7f12680cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944712713 0 0) 0x7f12680d0250 con 0x7f1268072060 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1254003530 con 0x7f1268072060 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3426633591 0 0) 0x7f1254003530 con 0x7f1268072060 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 msgr2=0x7f1268071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 0x7f1268071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 >> v2:172.21.15.98:3300/0 conn(0x7f12681455b0 msgr2=0x7f12680cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 --2- 172.21.15.70:0/1556476362 >> v2:172.21.15.98:3300/0 conn(0x7f12681455b0 0x7f12680cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12680cc710 con 0x7f1268072060 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1258001a00 con 0x7f1268072060 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 >> v1:172.21.15.70:6789/0 conn(0x7f1268072060 legacy=0x7f126815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12540039c0 0x7f1254003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254004810 0x7f1254004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 0x7f12540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12680724d0 con 0x7f12540039c0 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12680cd7b0 con 0x7f1254004810 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f125400ba80 con 0x7f12680cc150 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12540039c0 0x7f1254003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f12677fe640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 0x7f12540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 -- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 msgr2=0x7f12540051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12680cc150 0x7f12540051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 -- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254004810 msgr2=0x7f1254004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1254004810 0x7f1254004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.558+0000 7f1266ffd640 1 -- 172.21.15.70:0/1556476362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1254003530 con 0x7f12540039c0 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f1266ffd640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12540039c0 0x7f1254003de0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1260009ae0 tx=0x7f126001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25cc69c6876f76b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12600096b0 con 0x7f12540039c0 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12600a6490 con 0x7f12540039c0 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f1265ffb640 1 -- 172.21.15.70:0/1556476362 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12600a6880 con 0x7f12540039c0 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/1556476362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12540039c0 msgr2=0x7f1254003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 --2- 172.21.15.70:0/1556476362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12540039c0 0x7f1254003de0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1260009ae0 tx=0x7f126001cab0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 -- 172.21.15.70:0/1556476362 reap_dead start 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/1556476362 shutdown_connections 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/1556476362 >> 172.21.15.70:0/1556476362 conn(0x7f126806b1f0 msgr2=0x7f126806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/1556476362 shutdown_connections 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/1556476362 wait complete. 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 Processor -- start 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- start start 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12540039c0 0x7f12680aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f126815b410 0x7f12680aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 0x7f1268165ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1268166b30 con 0x7f1268165b70 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f126806f210 con 0x7f12540039c0 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1268166cb0 con 0x7f126815b410 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 0x7f1268165ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 0x7f1268165ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42154/0 (socket says 172.21.15.70:42154) 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 -- 172.21.15.70:0/3621311628 learned_addr learned my addr 172.21.15.70:0/3621311628 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12667fc640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f126815b410 0x7f12680aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f1266ffd640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12540039c0 0x7f12680aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 -- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f126815b410 msgr2=0x7f12680aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f126815b410 0x7f12680aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 -- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12540039c0 msgr2=0x7f12680aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12540039c0 0x7f12680aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 -- 172.21.15.70:0/3621311628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12681f5fe0 con 0x7f1268165b70 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12667fc640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f126815b410 0x7f12680aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f12677fe640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 0x7f1268165ff0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1268070030 tx=0x7f125800e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125800eb70 con 0x7f1268165b70 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1258018070 con 0x7f1268165b70 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12680aa8f0 con 0x7f1268165b70 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258013420 con 0x7f1268165b70 2024-04-22T01:17:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.562+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12681f5740 con 0x7f1268165b70 2024-04-22T01:17:15.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.566+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1268166530 con 0x7f1268165b70 2024-04-22T01:17:15.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.566+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f12580135c0 con 0x7f1268165b70 2024-04-22T01:17:15.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.566+0000 7f125f7fe640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12480678f0 0x7f1248069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.566+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f1258086100 con 0x7f1268165b70 2024-04-22T01:17:15.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.566+0000 7f1266ffd640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12480678f0 0x7f1248069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.566+0000 7f1266ffd640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12480678f0 0x7f1248069db0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f12600ad000 tx=0x7f12600b2040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.570+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1258056ab0 con 0x7f1268165b70 2024-04-22T01:17:15.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.726+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1268068210 con 0x7f12480678f0 2024-04-22T01:17:15.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.742+0000 7f125f7fe640 1 -- 172.21.15.70:0/3621311628 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251983 (secure 0 0 0) 0x7f1268068210 con 0x7f12480678f0 2024-04-22T01:17:15.745 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:15.746 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:17:15.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12480678f0 msgr2=0x7f1248069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12480678f0 0x7f1248069db0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f12600ad000 tx=0x7f12600b2040 comp rx=0 tx=0).stop 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 msgr2=0x7f1268165ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 0x7f1268165ff0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1268070030 tx=0x7f125800e160 comp rx=0 tx=0).stop 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 shutdown_connections 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12480678f0 0x7f1248069db0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1268165b70 0x7f1268165ff0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f126815b410 0x7f12680aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 --2- 172.21.15.70:0/3621311628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12540039c0 0x7f12680aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 >> 172.21.15.70:0/3621311628 conn(0x7f126806b1f0 msgr2=0x7f12680af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 shutdown_connections 2024-04-22T01:17:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.746+0000 7f126d2c0640 1 -- 172.21.15.70:0/3621311628 wait complete. 2024-04-22T01:17:15.763 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-22T01:17:14.631706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252484,"kb_used_data":7176,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754722236,"statfs":{"total":773094113280,"available":772835569664,"internally_reserved":0,"allocated":7348224,"data_stored":3925860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.284716"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.425826+0000","last_change":"2024-04-22T01:17:12.425149+0000","last_active":"2024-04-22T01:17:12.425826+0000","last_peered":"2024-04-22T01:17:12.425826+0000","last_clean":"2024-04-22T01:17:12.425826+0000","last_became_active":"2024-04-22T01:17:12.424861+0000","last_became_peered":"2024-04-22T01:17:12.424861+0000","last_unstale":"2024-04-22T01:17:12.425826+0000","last_undegraded":"2024-04-22T01:17:12.425826+0000","last_fullsized":"2024-04-22T01:17:12.425826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:55.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.399729+0000","last_change":"2024-04-22T01:17:12.397295+0000","last_active":"2024-04-22T01:17:12.399729+0000","last_peered":"2024-04-22T01:17:12.399729+0000","last_clean":"2024-04-22T01:17:12.399729+0000","last_became_active":"2024-04-22T01:17:12.397063+0000","last_became_peered":"2024-04-22T01:17:12.397063+0000","last_unstale":"2024-04-22T01:17:12.399729+0000","last_undegraded":"2024-04-22T01:17:12.399729+0000","last_fullsized":"2024-04-22T01:17:12.399729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:23:03.665698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.455754+0000","last_change":"2024-04-22T01:17:12.451979+0000","last_active":"2024-04-22T01:17:12.455754+0000","last_peered":"2024-04-22T01:17:12.455754+0000","last_clean":"2024-04-22T01:17:12.455754+0000","last_became_active":"2024-04-22T01:17:12.448882+0000","last_became_peered":"2024-04-22T01:17:12.448882+0000","last_unstale":"2024-04-22T01:17:12.455754+0000","last_undegraded":"2024-04-22T01:17:12.455754+0000","last_fullsized":"2024-04-22T01:17:12.455754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:08.432611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384837+0000","last_change":"2024-04-22T01:17:12.431824+0000","last_active":"2024-04-22T01:17:13.384837+0000","last_peered":"2024-04-22T01:17:13.384837+0000","last_clean":"2024-04-22T01:17:13.384837+0000","last_became_active":"2024-04-22T01:17:12.431301+0000","last_became_peered":"2024-04-22T01:17:12.431301+0000","last_unstale":"2024-04-22T01:17:13.384837+0000","last_undegraded":"2024-04-22T01:17:13.384837+0000","last_fullsized":"2024-04-22T01:17:13.384837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:23:37.376966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.435813+0000","last_change":"2024-04-22T01:17:12.434550+0000","last_active":"2024-04-22T01:17:12.435813+0000","last_peered":"2024-04-22T01:17:12.435813+0000","last_clean":"2024-04-22T01:17:12.435813+0000","last_became_active":"2024-04-22T01:17:12.434141+0000","last_became_peered":"2024-04-22T01:17:12.434141+0000","last_unstale":"2024-04-22T01:17:12.435813+0000","last_undegraded":"2024-04-22T01:17:12.435813+0000","last_fullsized":"2024-04-22T01:17:12.435813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:37.713812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.447129+0000","last_change":"2024-04-22T01:17:12.445892+0000","last_active":"2024-04-22T01:17:12.447129+0000","last_peered":"2024-04-22T01:17:12.447129+0000","last_clean":"2024-04-22T01:17:12.447129+0000","last_became_active":"2024-04-22T01:17:12.445582+0000","last_became_peered":"2024-04-22T01:17:12.445582+0000","last_unstale":"2024-04-22T01:17:12.447129+0000","last_undegraded":"2024-04-22T01:17:12.447129+0000","last_fullsized":"2024-04-22T01:17:12.447129+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:11.255244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.447359+0000","last_change":"2024-04-22T01:17:12.446590+0000","last_active":"2024-04-22T01:17:12.447359+0000","last_peered":"2024-04-22T01:17:12.447359+0000","last_clean":"2024-04-22T01:17:12.447359+0000","last_became_active":"2024-04-22T01:17:12.446257+0000","last_became_peered":"2024-04-22T01:17:12.446257+0000","last_unstale":"2024-04-22T01:17:12.447359+0000","last_undegraded":"2024-04-22T01:17:12.447359+0000","last_fullsized":"2024-04-22T01:17:12.447359+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:47.783642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382504+0000","last_change":"2024-04-22T01:17:12.441706+0000","last_active":"2024-04-22T01:17:13.382504+0000","last_peered":"2024-04-22T01:17:13.382504+0000","last_clean":"2024-04-22T01:17:13.382504+0000","last_became_active":"2024-04-22T01:17:12.441059+0000","last_became_peered":"2024-04-22T01:17:12.441059+0000","last_unstale":"2024-04-22T01:17:13.382504+0000","last_undegraded":"2024-04-22T01:17:13.382504+0000","last_fullsized":"2024-04-22T01:17:13.382504+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:08.432611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.456666+0000","last_change":"2024-04-22T01:17:12.453778+0000","last_active":"2024-04-22T01:17:12.456666+0000","last_peered":"2024-04-22T01:17:12.456666+0000","last_clean":"2024-04-22T01:17:12.456666+0000","last_became_active":"2024-04-22T01:17:12.453362+0000","last_became_peered":"2024-04-22T01:17:12.453362+0000","last_unstale":"2024-04-22T01:17:12.456666+0000","last_undegraded":"2024-04-22T01:17:12.456666+0000","last_fullsized":"2024-04-22T01:17:12.456666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:04.198934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.454374+0000","last_change":"2024-04-22T01:17:12.445628+0000","last_active":"2024-04-22T01:17:12.454374+0000","last_peered":"2024-04-22T01:17:12.454374+0000","last_clean":"2024-04-22T01:17:12.454374+0000","last_became_active":"2024-04-22T01:17:12.444940+0000","last_became_peered":"2024-04-22T01:17:12.444940+0000","last_unstale":"2024-04-22T01:17:12.454374+0000","last_undegraded":"2024-04-22T01:17:12.454374+0000","last_fullsized":"2024-04-22T01:17:12.454374+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:45.176558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383044+0000","last_change":"2024-04-22T01:17:12.436645+0000","last_active":"2024-04-22T01:17:13.383044+0000","last_peered":"2024-04-22T01:17:13.383044+0000","last_clean":"2024-04-22T01:17:13.383044+0000","last_became_active":"2024-04-22T01:17:12.431232+0000","last_became_peered":"2024-04-22T01:17:12.431232+0000","last_unstale":"2024-04-22T01:17:13.383044+0000","last_undegraded":"2024-04-22T01:17:13.383044+0000","last_fullsized":"2024-04-22T01:17:13.383044+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:47.837694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382833+0000","last_change":"2024-04-22T01:17:12.447917+0000","last_active":"2024-04-22T01:17:13.382833+0000","last_peered":"2024-04-22T01:17:13.382833+0000","last_clean":"2024-04-22T01:17:13.382833+0000","last_became_active":"2024-04-22T01:17:12.447194+0000","last_became_peered":"2024-04-22T01:17:12.447194+0000","last_unstale":"2024-04-22T01:17:13.382833+0000","last_undegraded":"2024-04-22T01:17:13.382833+0000","last_fullsized":"2024-04-22T01:17:13.382833+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:16.497986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383324+0000","last_change":"2024-04-22T01:17:12.428338+0000","last_active":"2024-04-22T01:17:13.383324+0000","last_peered":"2024-04-22T01:17:13.383324+0000","last_clean":"2024-04-22T01:17:13.383324+0000","last_became_active":"2024-04-22T01:17:12.427607+0000","last_became_peered":"2024-04-22T01:17:12.427607+0000","last_unstale":"2024-04-22T01:17:13.383324+0000","last_undegraded":"2024-04-22T01:17:13.383324+0000","last_fullsized":"2024-04-22T01:17:13.383324+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.135714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.454467+0000","last_change":"2024-04-22T01:17:12.449050+0000","last_active":"2024-04-22T01:17:12.454467+0000","last_peered":"2024-04-22T01:17:12.454467+0000","last_clean":"2024-04-22T01:17:12.454467+0000","last_became_active":"2024-04-22T01:17:12.448491+0000","last_became_peered":"2024-04-22T01:17:12.448491+0000","last_unstale":"2024-04-22T01:17:12.454467+0000","last_undegraded":"2024-04-22T01:17:12.454467+0000","last_fullsized":"2024-04-22T01:17:12.454467+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:45.646916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.455714+0000","last_change":"2024-04-22T01:17:12.449689+0000","last_active":"2024-04-22T01:17:12.455714+0000","last_peered":"2024-04-22T01:17:12.455714+0000","last_clean":"2024-04-22T01:17:12.455714+0000","last_became_active":"2024-04-22T01:17:12.448438+0000","last_became_peered":"2024-04-22T01:17:12.448438+0000","last_unstale":"2024-04-22T01:17:12.455714+0000","last_undegraded":"2024-04-22T01:17:12.455714+0000","last_fullsized":"2024-04-22T01:17:12.455714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:23.233992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.411935+0000","last_change":"2024-04-22T01:17:12.410973+0000","last_active":"2024-04-22T01:17:12.411935+0000","last_peered":"2024-04-22T01:17:12.411935+0000","last_clean":"2024-04-22T01:17:12.411935+0000","last_became_active":"2024-04-22T01:17:12.410725+0000","last_became_peered":"2024-04-22T01:17:12.410725+0000","last_unstale":"2024-04-22T01:17:12.411935+0000","last_undegraded":"2024-04-22T01:17:12.411935+0000","last_fullsized":"2024-04-22T01:17:12.411935+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:06.528430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.437154+0000","last_change":"2024-04-22T01:17:12.436208+0000","last_active":"2024-04-22T01:17:12.437154+0000","last_peered":"2024-04-22T01:17:12.437154+0000","last_clean":"2024-04-22T01:17:12.437154+0000","last_became_active":"2024-04-22T01:17:12.435888+0000","last_became_peered":"2024-04-22T01:17:12.435888+0000","last_unstale":"2024-04-22T01:17:12.437154+0000","last_undegraded":"2024-04-22T01:17:12.437154+0000","last_fullsized":"2024-04-22T01:17:12.437154+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:05.219197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383698+0000","last_change":"2024-04-22T01:17:12.431695+0000","last_active":"2024-04-22T01:17:13.383698+0000","last_peered":"2024-04-22T01:17:13.383698+0000","last_clean":"2024-04-22T01:17:13.383698+0000","last_became_active":"2024-04-22T01:17:12.425976+0000","last_became_peered":"2024-04-22T01:17:12.425976+0000","last_unstale":"2024-04-22T01:17:13.383698+0000","last_undegraded":"2024-04-22T01:17:13.383698+0000","last_fullsized":"2024-04-22T01:17:13.383698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:39.139826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413459+0000","last_change":"2024-04-22T01:17:12.403596+0000","last_active":"2024-04-22T01:17:12.413459+0000","last_peered":"2024-04-22T01:17:12.413459+0000","last_clean":"2024-04-22T01:17:12.413459+0000","last_became_active":"2024-04-22T01:17:12.399140+0000","last_became_peered":"2024-04-22T01:17:12.399140+0000","last_unstale":"2024-04-22T01:17:12.413459+0000","last_undegraded":"2024-04-22T01:17:12.413459+0000","last_fullsized":"2024-04-22T01:17:12.413459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:04.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381618+0000","last_change":"2024-04-22T01:17:12.404309+0000","last_active":"2024-04-22T01:17:13.381618+0000","last_peered":"2024-04-22T01:17:13.381618+0000","last_clean":"2024-04-22T01:17:13.381618+0000","last_became_active":"2024-04-22T01:17:12.403970+0000","last_became_peered":"2024-04-22T01:17:12.403970+0000","last_unstale":"2024-04-22T01:17:13.381618+0000","last_undegraded":"2024-04-22T01:17:13.381618+0000","last_fullsized":"2024-04-22T01:17:13.381618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:40.008600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383754+0000","last_change":"2024-04-22T01:17:12.410404+0000","last_active":"2024-04-22T01:17:13.383754+0000","last_peered":"2024-04-22T01:17:13.383754+0000","last_clean":"2024-04-22T01:17:13.383754+0000","last_became_active":"2024-04-22T01:17:12.406685+0000","last_became_peered":"2024-04-22T01:17:12.406685+0000","last_unstale":"2024-04-22T01:17:13.383754+0000","last_undegraded":"2024-04-22T01:17:13.383754+0000","last_fullsized":"2024-04-22T01:17:13.383754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:45.508179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381193+0000","last_change":"2024-04-22T01:17:12.450898+0000","last_active":"2024-04-22T01:17:13.381193+0000","last_peered":"2024-04-22T01:17:13.381193+0000","last_clean":"2024-04-22T01:17:13.381193+0000","last_became_active":"2024-04-22T01:17:12.450630+0000","last_became_peered":"2024-04-22T01:17:12.450630+0000","last_unstale":"2024-04-22T01:17:13.381193+0000","last_undegraded":"2024-04-22T01:17:13.381193+0000","last_fullsized":"2024-04-22T01:17:13.381193+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:34:22.000068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381435+0000","last_change":"2024-04-22T01:17:12.446618+0000","last_active":"2024-04-22T01:17:13.381435+0000","last_peered":"2024-04-22T01:17:13.381435+0000","last_clean":"2024-04-22T01:17:13.381435+0000","last_became_active":"2024-04-22T01:17:12.446244+0000","last_became_peered":"2024-04-22T01:17:12.446244+0000","last_unstale":"2024-04-22T01:17:13.381435+0000","last_undegraded":"2024-04-22T01:17:13.381435+0000","last_fullsized":"2024-04-22T01:17:13.381435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:11.255244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.455781+0000","last_change":"2024-04-22T01:17:12.449970+0000","last_active":"2024-04-22T01:17:12.455781+0000","last_peered":"2024-04-22T01:17:12.455781+0000","last_clean":"2024-04-22T01:17:12.455781+0000","last_became_active":"2024-04-22T01:17:12.445906+0000","last_became_peered":"2024-04-22T01:17:12.445906+0000","last_unstale":"2024-04-22T01:17:12.455781+0000","last_undegraded":"2024-04-22T01:17:12.455781+0000","last_fullsized":"2024-04-22T01:17:12.455781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:35.589064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.385485+0000","last_change":"2024-04-22T01:17:12.409496+0000","last_active":"2024-04-22T01:17:13.385485+0000","last_peered":"2024-04-22T01:17:13.385485+0000","last_clean":"2024-04-22T01:17:13.385485+0000","last_became_active":"2024-04-22T01:17:12.408265+0000","last_became_peered":"2024-04-22T01:17:12.408265+0000","last_unstale":"2024-04-22T01:17:13.385485+0000","last_undegraded":"2024-04-22T01:17:13.385485+0000","last_fullsized":"2024-04-22T01:17:13.385485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:44.887237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383243+0000","last_change":"2024-04-22T01:17:12.419612+0000","last_active":"2024-04-22T01:17:13.383243+0000","last_peered":"2024-04-22T01:17:13.383243+0000","last_clean":"2024-04-22T01:17:13.383243+0000","last_became_active":"2024-04-22T01:17:12.418847+0000","last_became_peered":"2024-04-22T01:17:12.418847+0000","last_unstale":"2024-04-22T01:17:13.383243+0000","last_undegraded":"2024-04-22T01:17:13.383243+0000","last_fullsized":"2024-04-22T01:17:13.383243+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:04.198934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383448+0000","last_change":"2024-04-22T01:17:12.423974+0000","last_active":"2024-04-22T01:17:13.383448+0000","last_peered":"2024-04-22T01:17:13.383448+0000","last_clean":"2024-04-22T01:17:13.383448+0000","last_became_active":"2024-04-22T01:17:12.414842+0000","last_became_peered":"2024-04-22T01:17:12.414842+0000","last_unstale":"2024-04-22T01:17:13.383448+0000","last_undegraded":"2024-04-22T01:17:13.383448+0000","last_fullsized":"2024-04-22T01:17:13.383448+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:00.975780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.407582+0000","last_change":"2024-04-22T01:17:12.407116+0000","last_active":"2024-04-22T01:17:12.407582+0000","last_peered":"2024-04-22T01:17:12.407582+0000","last_clean":"2024-04-22T01:17:12.407582+0000","last_became_active":"2024-04-22T01:17:12.406838+0000","last_became_peered":"2024-04-22T01:17:12.406838+0000","last_unstale":"2024-04-22T01:17:12.407582+0000","last_undegraded":"2024-04-22T01:17:12.407582+0000","last_fullsized":"2024-04-22T01:17:12.407582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:42.942529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.454476+0000","last_change":"2024-04-22T01:17:12.449382+0000","last_active":"2024-04-22T01:17:12.454476+0000","last_peered":"2024-04-22T01:17:12.454476+0000","last_clean":"2024-04-22T01:17:12.454476+0000","last_became_active":"2024-04-22T01:17:12.448775+0000","last_became_peered":"2024-04-22T01:17:12.448775+0000","last_unstale":"2024-04-22T01:17:12.454476+0000","last_undegraded":"2024-04-22T01:17:12.454476+0000","last_fullsized":"2024-04-22T01:17:12.454476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:21.947282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384105+0000","last_change":"2024-04-22T01:17:12.441599+0000","last_active":"2024-04-22T01:17:13.384105+0000","last_peered":"2024-04-22T01:17:13.384105+0000","last_clean":"2024-04-22T01:17:13.384105+0000","last_became_active":"2024-04-22T01:17:12.441166+0000","last_became_peered":"2024-04-22T01:17:12.441166+0000","last_unstale":"2024-04-22T01:17:13.384105+0000","last_undegraded":"2024-04-22T01:17:13.384105+0000","last_fullsized":"2024-04-22T01:17:13.384105+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:11:58.058693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383485+0000","last_change":"2024-04-22T01:17:12.426093+0000","last_active":"2024-04-22T01:17:13.383485+0000","last_peered":"2024-04-22T01:17:13.383485+0000","last_clean":"2024-04-22T01:17:13.383485+0000","last_became_active":"2024-04-22T01:17:12.425411+0000","last_became_peered":"2024-04-22T01:17:12.425411+0000","last_unstale":"2024-04-22T01:17:13.383485+0000","last_undegraded":"2024-04-22T01:17:13.383485+0000","last_fullsized":"2024-04-22T01:17:13.383485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:37.713812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.403993+0000","last_change":"2024-04-22T01:17:12.401593+0000","last_active":"2024-04-22T01:17:12.403993+0000","last_peered":"2024-04-22T01:17:12.403993+0000","last_clean":"2024-04-22T01:17:12.403993+0000","last_became_active":"2024-04-22T01:17:12.401275+0000","last_became_peered":"2024-04-22T01:17:12.401275+0000","last_unstale":"2024-04-22T01:17:12.403993+0000","last_undegraded":"2024-04-22T01:17:12.403993+0000","last_fullsized":"2024-04-22T01:17:12.403993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:34:22.000068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.420224+0000","last_change":"2024-04-22T01:17:12.404597+0000","last_active":"2024-04-22T01:17:12.420224+0000","last_peered":"2024-04-22T01:17:12.420224+0000","last_clean":"2024-04-22T01:17:12.420224+0000","last_became_active":"2024-04-22T01:17:12.403776+0000","last_became_peered":"2024-04-22T01:17:12.403776+0000","last_unstale":"2024-04-22T01:17:12.420224+0000","last_undegraded":"2024-04-22T01:17:12.420224+0000","last_fullsized":"2024-04-22T01:17:12.420224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:07.112109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"63'4","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.453924+0000","last_change":"2024-04-22T01:17:11.398473+0000","last_active":"2024-04-22T01:17:12.453924+0000","last_peered":"2024-04-22T01:17:12.453924+0000","last_clean":"2024-04-22T01:17:12.453924+0000","last_became_active":"2024-04-22T01:17:11.398203+0000","last_became_peered":"2024-04-22T01:17:11.398203+0000","last_unstale":"2024-04-22T01:17:12.453924+0000","last_undegraded":"2024-04-22T01:17:12.453924+0000","last_fullsized":"2024-04-22T01:17:12.453924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:36.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383867+0000","last_change":"2024-04-22T01:17:12.417482+0000","last_active":"2024-04-22T01:17:13.383867+0000","last_peered":"2024-04-22T01:17:13.383867+0000","last_clean":"2024-04-22T01:17:13.383867+0000","last_became_active":"2024-04-22T01:17:12.416481+0000","last_became_peered":"2024-04-22T01:17:12.416481+0000","last_unstale":"2024-04-22T01:17:13.383867+0000","last_undegraded":"2024-04-22T01:17:13.383867+0000","last_fullsized":"2024-04-22T01:17:13.383867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:23:03.665698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.395347+0000","last_change":"2024-04-22T01:17:11.390095+0000","last_active":"2024-04-22T01:17:12.395347+0000","last_peered":"2024-04-22T01:17:12.395347+0000","last_clean":"2024-04-22T01:17:12.395347+0000","last_became_active":"2024-04-22T01:17:11.389816+0000","last_became_peered":"2024-04-22T01:17:11.389816+0000","last_unstale":"2024-04-22T01:17:12.395347+0000","last_undegraded":"2024-04-22T01:17:12.395347+0000","last_fullsized":"2024-04-22T01:17:12.395347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:20.132601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.402524+0000","last_change":"2024-04-22T01:17:12.395511+0000","last_active":"2024-04-22T01:17:12.402524+0000","last_peered":"2024-04-22T01:17:12.402524+0000","last_clean":"2024-04-22T01:17:12.402524+0000","last_became_active":"2024-04-22T01:17:12.395137+0000","last_became_peered":"2024-04-22T01:17:12.395137+0000","last_unstale":"2024-04-22T01:17:12.402524+0000","last_undegraded":"2024-04-22T01:17:12.402524+0000","last_fullsized":"2024-04-22T01:17:12.402524+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:44.887237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.420881+0000","last_change":"2024-04-22T01:17:11.388040+0000","last_active":"2024-04-22T01:17:12.420881+0000","last_peered":"2024-04-22T01:17:12.420881+0000","last_clean":"2024-04-22T01:17:12.420881+0000","last_became_active":"2024-04-22T01:17:11.387716+0000","last_became_peered":"2024-04-22T01:17:11.387716+0000","last_unstale":"2024-04-22T01:17:12.420881+0000","last_undegraded":"2024-04-22T01:17:12.420881+0000","last_fullsized":"2024-04-22T01:17:12.420881+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:44.505065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.456611+0000","last_change":"2024-04-22T01:17:12.452894+0000","last_active":"2024-04-22T01:17:12.456611+0000","last_peered":"2024-04-22T01:17:12.456611+0000","last_clean":"2024-04-22T01:17:12.456611+0000","last_became_active":"2024-04-22T01:17:12.452293+0000","last_became_peered":"2024-04-22T01:17:12.452293+0000","last_unstale":"2024-04-22T01:17:12.456611+0000","last_undegraded":"2024-04-22T01:17:12.456611+0000","last_fullsized":"2024-04-22T01:17:12.456611+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:16.497986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.401544+0000","last_change":"2024-04-22T01:17:11.367406+0000","last_active":"2024-04-22T01:17:12.401544+0000","last_peered":"2024-04-22T01:17:12.401544+0000","last_clean":"2024-04-22T01:17:12.401544+0000","last_became_active":"2024-04-22T01:17:11.367135+0000","last_became_peered":"2024-04-22T01:17:11.367135+0000","last_unstale":"2024-04-22T01:17:12.401544+0000","last_undegraded":"2024-04-22T01:17:12.401544+0000","last_fullsized":"2024-04-22T01:17:12.401544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:03.195821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.399008+0000","last_change":"2024-04-22T01:17:12.398045+0000","last_active":"2024-04-22T01:17:12.399008+0000","last_peered":"2024-04-22T01:17:12.399008+0000","last_clean":"2024-04-22T01:17:12.399008+0000","last_became_active":"2024-04-22T01:17:12.397746+0000","last_became_peered":"2024-04-22T01:17:12.397746+0000","last_unstale":"2024-04-22T01:17:12.399008+0000","last_undegraded":"2024-04-22T01:17:12.399008+0000","last_fullsized":"2024-04-22T01:17:12.399008+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:22.816872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381436+0000","last_change":"2024-04-22T01:17:11.363285+0000","last_active":"2024-04-22T01:17:13.381436+0000","last_peered":"2024-04-22T01:17:13.381436+0000","last_clean":"2024-04-22T01:17:13.381436+0000","last_became_active":"2024-04-22T01:17:11.362547+0000","last_became_peered":"2024-04-22T01:17:11.362547+0000","last_unstale":"2024-04-22T01:17:13.381436+0000","last_undegraded":"2024-04-22T01:17:13.381436+0000","last_fullsized":"2024-04-22T01:17:13.381436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:22.230878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.436904+0000","last_change":"2024-04-22T01:17:12.436067+0000","last_active":"2024-04-22T01:17:12.436904+0000","last_peered":"2024-04-22T01:17:12.436904+0000","last_clean":"2024-04-22T01:17:12.436904+0000","last_became_active":"2024-04-22T01:17:12.435799+0000","last_became_peered":"2024-04-22T01:17:12.435799+0000","last_unstale":"2024-04-22T01:17:12.436904+0000","last_undegraded":"2024-04-22T01:17:12.436904+0000","last_fullsized":"2024-04-22T01:17:12.436904+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:23:37.376966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381427+0000","last_change":"2024-04-22T01:17:11.368766+0000","last_active":"2024-04-22T01:17:13.381427+0000","last_peered":"2024-04-22T01:17:13.381427+0000","last_clean":"2024-04-22T01:17:13.381427+0000","last_became_active":"2024-04-22T01:17:11.368418+0000","last_became_peered":"2024-04-22T01:17:11.368418+0000","last_unstale":"2024-04-22T01:17:13.381427+0000","last_undegraded":"2024-04-22T01:17:13.381427+0000","last_fullsized":"2024-04-22T01:17:13.381427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:15.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423582+0000","last_change":"2024-04-22T01:17:12.415903+0000","last_active":"2024-04-22T01:17:12.423582+0000","last_peered":"2024-04-22T01:17:12.423582+0000","last_clean":"2024-04-22T01:17:12.423582+0000","last_became_active":"2024-04-22T01:17:12.415097+0000","last_became_peered":"2024-04-22T01:17:12.415097+0000","last_unstale":"2024-04-22T01:17:12.423582+0000","last_undegraded":"2024-04-22T01:17:12.423582+0000","last_fullsized":"2024-04-22T01:17:12.423582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:01.275955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381567+0000","last_change":"2024-04-22T01:17:11.377399+0000","last_active":"2024-04-22T01:17:13.381567+0000","last_peered":"2024-04-22T01:17:13.381567+0000","last_clean":"2024-04-22T01:17:13.381567+0000","last_became_active":"2024-04-22T01:17:11.377092+0000","last_became_peered":"2024-04-22T01:17:11.377092+0000","last_unstale":"2024-04-22T01:17:13.381567+0000","last_undegraded":"2024-04-22T01:17:13.381567+0000","last_fullsized":"2024-04-22T01:17:13.381567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:34.585950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381376+0000","last_change":"2024-04-22T01:17:12.427917+0000","last_active":"2024-04-22T01:17:13.381376+0000","last_peered":"2024-04-22T01:17:13.381376+0000","last_clean":"2024-04-22T01:17:13.381376+0000","last_became_active":"2024-04-22T01:17:12.427605+0000","last_became_peered":"2024-04-22T01:17:12.427605+0000","last_unstale":"2024-04-22T01:17:13.381376+0000","last_undegraded":"2024-04-22T01:17:13.381376+0000","last_fullsized":"2024-04-22T01:17:13.381376+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:47.688688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.392753+0000","last_change":"2024-04-22T01:17:11.398614+0000","last_active":"2024-04-22T01:17:12.392753+0000","last_peered":"2024-04-22T01:17:12.392753+0000","last_clean":"2024-04-22T01:17:12.392753+0000","last_became_active":"2024-04-22T01:17:11.397685+0000","last_became_peered":"2024-04-22T01:17:11.397685+0000","last_unstale":"2024-04-22T01:17:12.392753+0000","last_undegraded":"2024-04-22T01:17:12.392753+0000","last_fullsized":"2024-04-22T01:17:12.392753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:48.400105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.401676+0000","last_change":"2024-04-22T01:17:12.399917+0000","last_active":"2024-04-22T01:17:12.401676+0000","last_peered":"2024-04-22T01:17:12.401676+0000","last_clean":"2024-04-22T01:17:12.401676+0000","last_became_active":"2024-04-22T01:17:12.399619+0000","last_became_peered":"2024-04-22T01:17:12.399619+0000","last_unstale":"2024-04-22T01:17:12.401676+0000","last_undegraded":"2024-04-22T01:17:12.401676+0000","last_fullsized":"2024-04-22T01:17:12.401676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:02.844083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.439548+0000","last_change":"2024-04-22T01:17:11.406047+0000","last_active":"2024-04-22T01:17:12.439548+0000","last_peered":"2024-04-22T01:17:12.439548+0000","last_clean":"2024-04-22T01:17:12.439548+0000","last_became_active":"2024-04-22T01:17:11.402475+0000","last_became_peered":"2024-04-22T01:17:11.402475+0000","last_unstale":"2024-04-22T01:17:12.439548+0000","last_undegraded":"2024-04-22T01:17:12.439548+0000","last_fullsized":"2024-04-22T01:17:12.439548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:15:02.280889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381542+0000","last_change":"2024-04-22T01:17:12.411749+0000","last_active":"2024-04-22T01:17:13.381542+0000","last_peered":"2024-04-22T01:17:13.381542+0000","last_clean":"2024-04-22T01:17:13.381542+0000","last_became_active":"2024-04-22T01:17:12.411327+0000","last_became_peered":"2024-04-22T01:17:12.411327+0000","last_unstale":"2024-04-22T01:17:13.381542+0000","last_undegraded":"2024-04-22T01:17:13.381542+0000","last_fullsized":"2024-04-22T01:17:13.381542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:49.794109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.385269+0000","last_change":"2024-04-22T01:17:11.377486+0000","last_active":"2024-04-22T01:17:13.385269+0000","last_peered":"2024-04-22T01:17:13.385269+0000","last_clean":"2024-04-22T01:17:13.385269+0000","last_became_active":"2024-04-22T01:17:11.376647+0000","last_became_peered":"2024-04-22T01:17:11.376647+0000","last_unstale":"2024-04-22T01:17:13.385269+0000","last_undegraded":"2024-04-22T01:17:13.385269+0000","last_fullsized":"2024-04-22T01:17:13.385269+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:43.934103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423738+0000","last_change":"2024-04-22T01:17:12.421524+0000","last_active":"2024-04-22T01:17:12.423738+0000","last_peered":"2024-04-22T01:17:12.423738+0000","last_clean":"2024-04-22T01:17:12.423738+0000","last_became_active":"2024-04-22T01:17:12.420959+0000","last_became_peered":"2024-04-22T01:17:12.420959+0000","last_unstale":"2024-04-22T01:17:12.423738+0000","last_undegraded":"2024-04-22T01:17:12.423738+0000","last_fullsized":"2024-04-22T01:17:12.423738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:47.837694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"63'3","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.448937+0000","last_change":"2024-04-22T01:17:11.405992+0000","last_active":"2024-04-22T01:17:12.448937+0000","last_peered":"2024-04-22T01:17:12.448937+0000","last_clean":"2024-04-22T01:17:12.448937+0000","last_became_active":"2024-04-22T01:17:11.402522+0000","last_became_peered":"2024-04-22T01:17:11.402522+0000","last_unstale":"2024-04-22T01:17:12.448937+0000","last_undegraded":"2024-04-22T01:17:12.448937+0000","last_fullsized":"2024-04-22T01:17:12.448937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:20.718044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384730+0000","last_change":"2024-04-22T01:17:12.432231+0000","last_active":"2024-04-22T01:17:13.384730+0000","last_peered":"2024-04-22T01:17:13.384730+0000","last_clean":"2024-04-22T01:17:13.384730+0000","last_became_active":"2024-04-22T01:17:12.431580+0000","last_became_peered":"2024-04-22T01:17:12.431580+0000","last_unstale":"2024-04-22T01:17:13.384730+0000","last_undegraded":"2024-04-22T01:17:13.384730+0000","last_fullsized":"2024-04-22T01:17:13.384730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:55.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"63'6","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.444036+0000","last_change":"2024-04-22T01:17:11.378192+0000","last_active":"2024-04-22T01:17:12.444036+0000","last_peered":"2024-04-22T01:17:12.444036+0000","last_clean":"2024-04-22T01:17:12.444036+0000","last_became_active":"2024-04-22T01:17:11.377840+0000","last_became_peered":"2024-04-22T01:17:11.377840+0000","last_unstale":"2024-04-22T01:17:12.444036+0000","last_undegraded":"2024-04-22T01:17:12.444036+0000","last_fullsized":"2024-04-22T01:17:12.444036+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:46.834581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382551+0000","last_change":"2024-04-22T01:17:12.452029+0000","last_active":"2024-04-22T01:17:13.382551+0000","last_peered":"2024-04-22T01:17:13.382551+0000","last_clean":"2024-04-22T01:17:13.382551+0000","last_became_active":"2024-04-22T01:17:12.451117+0000","last_became_peered":"2024-04-22T01:17:12.451117+0000","last_unstale":"2024-04-22T01:17:13.382551+0000","last_undegraded":"2024-04-22T01:17:13.382551+0000","last_fullsized":"2024-04-22T01:17:13.382551+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:01.275955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.449842+0000","last_change":"2024-04-22T01:17:11.402025+0000","last_active":"2024-04-22T01:17:12.449842+0000","last_peered":"2024-04-22T01:17:12.449842+0000","last_clean":"2024-04-22T01:17:12.449842+0000","last_became_active":"2024-04-22T01:17:11.401072+0000","last_became_peered":"2024-04-22T01:17:11.401072+0000","last_unstale":"2024-04-22T01:17:12.449842+0000","last_undegraded":"2024-04-22T01:17:12.449842+0000","last_fullsized":"2024-04-22T01:17:12.449842+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:10.582013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382313+0000","last_change":"2024-04-22T01:17:12.408786+0000","last_active":"2024-04-22T01:17:13.382313+0000","last_peered":"2024-04-22T01:17:13.382313+0000","last_clean":"2024-04-22T01:17:13.382313+0000","last_became_active":"2024-04-22T01:17:12.407988+0000","last_became_peered":"2024-04-22T01:17:12.407988+0000","last_unstale":"2024-04-22T01:17:13.382313+0000","last_undegraded":"2024-04-22T01:17:13.382313+0000","last_fullsized":"2024-04-22T01:17:13.382313+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:23.233992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384341+0000","last_change":"2024-04-22T01:17:11.387269+0000","last_active":"2024-04-22T01:17:13.384341+0000","last_peered":"2024-04-22T01:17:13.384341+0000","last_clean":"2024-04-22T01:17:13.384341+0000","last_became_active":"2024-04-22T01:17:11.386551+0000","last_became_peered":"2024-04-22T01:17:11.386551+0000","last_unstale":"2024-04-22T01:17:13.384341+0000","last_undegraded":"2024-04-22T01:17:13.384341+0000","last_fullsized":"2024-04-22T01:17:13.384341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:41.939415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423475+0000","last_change":"2024-04-22T01:17:12.413713+0000","last_active":"2024-04-22T01:17:12.423475+0000","last_peered":"2024-04-22T01:17:12.423475+0000","last_clean":"2024-04-22T01:17:12.423475+0000","last_became_active":"2024-04-22T01:17:12.405307+0000","last_became_peered":"2024-04-22T01:17:12.405307+0000","last_unstale":"2024-04-22T01:17:12.423475+0000","last_undegraded":"2024-04-22T01:17:12.423475+0000","last_fullsized":"2024-04-22T01:17:12.423475+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:49.804868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.432859+0000","last_change":"2024-04-22T01:17:11.402310+0000","last_active":"2024-04-22T01:17:12.432859+0000","last_peered":"2024-04-22T01:17:12.432859+0000","last_clean":"2024-04-22T01:17:12.432859+0000","last_became_active":"2024-04-22T01:17:11.401522+0000","last_became_peered":"2024-04-22T01:17:11.401522+0000","last_unstale":"2024-04-22T01:17:12.432859+0000","last_undegraded":"2024-04-22T01:17:12.432859+0000","last_fullsized":"2024-04-22T01:17:12.432859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:32:22.458266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.442417+0000","last_change":"2024-04-22T01:17:12.442078+0000","last_active":"2024-04-22T01:17:12.442417+0000","last_peered":"2024-04-22T01:17:12.442417+0000","last_clean":"2024-04-22T01:17:12.442417+0000","last_became_active":"2024-04-22T01:17:12.441886+0000","last_became_peered":"2024-04-22T01:17:12.441886+0000","last_unstale":"2024-04-22T01:17:12.442417+0000","last_undegraded":"2024-04-22T01:17:12.442417+0000","last_fullsized":"2024-04-22T01:17:12.442417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:21.721158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"63'8","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382830+0000","last_change":"2024-04-22T01:17:11.406770+0000","last_active":"2024-04-22T01:17:13.382830+0000","last_peered":"2024-04-22T01:17:13.382830+0000","last_clean":"2024-04-22T01:17:13.382830+0000","last_became_active":"2024-04-22T01:17:11.406165+0000","last_became_peered":"2024-04-22T01:17:11.406165+0000","last_unstale":"2024-04-22T01:17:13.382830+0000","last_undegraded":"2024-04-22T01:17:13.382830+0000","last_fullsized":"2024-04-22T01:17:13.382830+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:21:52.705606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.453295+0000","last_change":"2024-04-22T01:17:12.445044+0000","last_active":"2024-04-22T01:17:12.453295+0000","last_peered":"2024-04-22T01:17:12.453295+0000","last_clean":"2024-04-22T01:17:12.453295+0000","last_became_active":"2024-04-22T01:17:12.444794+0000","last_became_peered":"2024-04-22T01:17:12.444794+0000","last_unstale":"2024-04-22T01:17:12.453295+0000","last_undegraded":"2024-04-22T01:17:12.453295+0000","last_fullsized":"2024-04-22T01:17:12.453295+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:23.997157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384859+0000","last_change":"2024-04-22T01:17:11.381706+0000","last_active":"2024-04-22T01:17:13.384859+0000","last_peered":"2024-04-22T01:17:13.384859+0000","last_clean":"2024-04-22T01:17:13.384859+0000","last_became_active":"2024-04-22T01:17:11.380892+0000","last_became_peered":"2024-04-22T01:17:11.380892+0000","last_unstale":"2024-04-22T01:17:13.384859+0000","last_undegraded":"2024-04-22T01:17:13.384859+0000","last_fullsized":"2024-04-22T01:17:13.384859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:23:36.373853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.435769+0000","last_change":"2024-04-22T01:17:12.434071+0000","last_active":"2024-04-22T01:17:12.435769+0000","last_peered":"2024-04-22T01:17:12.435769+0000","last_clean":"2024-04-22T01:17:12.435769+0000","last_became_active":"2024-04-22T01:17:12.432929+0000","last_became_peered":"2024-04-22T01:17:12.432929+0000","last_unstale":"2024-04-22T01:17:12.435769+0000","last_undegraded":"2024-04-22T01:17:12.435769+0000","last_fullsized":"2024-04-22T01:17:12.435769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:00.975780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.448297+0000","last_change":"2024-04-22T01:17:11.405174+0000","last_active":"2024-04-22T01:17:12.448297+0000","last_peered":"2024-04-22T01:17:12.448297+0000","last_clean":"2024-04-22T01:17:12.448297+0000","last_became_active":"2024-04-22T01:17:11.402072+0000","last_became_peered":"2024-04-22T01:17:11.402072+0000","last_unstale":"2024-04-22T01:17:12.448297+0000","last_undegraded":"2024-04-22T01:17:12.448297+0000","last_fullsized":"2024-04-22T01:17:12.448297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:38:53.540328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384076+0000","last_change":"2024-04-22T01:17:12.436550+0000","last_active":"2024-04-22T01:17:13.384076+0000","last_peered":"2024-04-22T01:17:13.384076+0000","last_clean":"2024-04-22T01:17:13.384076+0000","last_became_active":"2024-04-22T01:17:12.426633+0000","last_became_peered":"2024-04-22T01:17:12.426633+0000","last_unstale":"2024-04-22T01:17:13.384076+0000","last_undegraded":"2024-04-22T01:17:13.384076+0000","last_fullsized":"2024-04-22T01:17:13.384076+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:42.942529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.452978+0000","last_change":"2024-04-22T01:17:11.401688+0000","last_active":"2024-04-22T01:17:12.452978+0000","last_peered":"2024-04-22T01:17:12.452978+0000","last_clean":"2024-04-22T01:17:12.452978+0000","last_became_active":"2024-04-22T01:17:11.398306+0000","last_became_peered":"2024-04-22T01:17:11.398306+0000","last_unstale":"2024-04-22T01:17:12.452978+0000","last_undegraded":"2024-04-22T01:17:12.452978+0000","last_fullsized":"2024-04-22T01:17:12.452978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:18.085687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413571+0000","last_change":"2024-04-22T01:17:12.403576+0000","last_active":"2024-04-22T01:17:12.413571+0000","last_peered":"2024-04-22T01:17:12.413571+0000","last_clean":"2024-04-22T01:17:12.413571+0000","last_became_active":"2024-04-22T01:17:12.402846+0000","last_became_peered":"2024-04-22T01:17:12.402846+0000","last_unstale":"2024-04-22T01:17:12.413571+0000","last_undegraded":"2024-04-22T01:17:12.413571+0000","last_fullsized":"2024-04-22T01:17:12.413571+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:24.987567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381665+0000","last_change":"2024-04-22T01:17:11.383746+0000","last_active":"2024-04-22T01:17:13.381665+0000","last_peered":"2024-04-22T01:17:13.381665+0000","last_clean":"2024-04-22T01:17:13.381665+0000","last_became_active":"2024-04-22T01:17:11.383508+0000","last_became_peered":"2024-04-22T01:17:11.383508+0000","last_unstale":"2024-04-22T01:17:13.381665+0000","last_undegraded":"2024-04-22T01:17:13.381665+0000","last_fullsized":"2024-04-22T01:17:13.381665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:20.944168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.393113+0000","last_change":"2024-04-22T01:17:12.391586+0000","last_active":"2024-04-22T01:17:12.393113+0000","last_peered":"2024-04-22T01:17:12.393113+0000","last_clean":"2024-04-22T01:17:12.393113+0000","last_became_active":"2024-04-22T01:17:12.391308+0000","last_became_peered":"2024-04-22T01:17:12.391308+0000","last_unstale":"2024-04-22T01:17:12.393113+0000","last_undegraded":"2024-04-22T01:17:12.393113+0000","last_fullsized":"2024-04-22T01:17:12.393113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:11:58.058693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.415003+0000","last_change":"2024-04-22T01:17:11.381596+0000","last_active":"2024-04-22T01:17:12.415003+0000","last_peered":"2024-04-22T01:17:12.415003+0000","last_clean":"2024-04-22T01:17:12.415003+0000","last_became_active":"2024-04-22T01:17:11.381302+0000","last_became_peered":"2024-04-22T01:17:11.381302+0000","last_unstale":"2024-04-22T01:17:12.415003+0000","last_undegraded":"2024-04-22T01:17:12.415003+0000","last_fullsized":"2024-04-22T01:17:12.415003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:15.494872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.382708+0000","last_change":"2024-04-22T01:17:12.407065+0000","last_active":"2024-04-22T01:17:13.382708+0000","last_peered":"2024-04-22T01:17:13.382708+0000","last_clean":"2024-04-22T01:17:13.382708+0000","last_became_active":"2024-04-22T01:17:12.406287+0000","last_became_peered":"2024-04-22T01:17:12.406287+0000","last_unstale":"2024-04-22T01:17:13.382708+0000","last_undegraded":"2024-04-22T01:17:13.382708+0000","last_fullsized":"2024-04-22T01:17:13.382708+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:35.589064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384547+0000","last_change":"2024-04-22T01:17:11.382007+0000","last_active":"2024-04-22T01:17:13.384547+0000","last_peered":"2024-04-22T01:17:13.384547+0000","last_clean":"2024-04-22T01:17:13.384547+0000","last_became_active":"2024-04-22T01:17:11.379830+0000","last_became_peered":"2024-04-22T01:17:11.379830+0000","last_unstale":"2024-04-22T01:17:13.384547+0000","last_undegraded":"2024-04-22T01:17:13.384547+0000","last_fullsized":"2024-04-22T01:17:13.384547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:54.789380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423536+0000","last_change":"2024-04-22T01:17:12.414281+0000","last_active":"2024-04-22T01:17:12.423536+0000","last_peered":"2024-04-22T01:17:12.423536+0000","last_clean":"2024-04-22T01:17:12.423536+0000","last_became_active":"2024-04-22T01:17:12.399829+0000","last_became_peered":"2024-04-22T01:17:12.399829+0000","last_unstale":"2024-04-22T01:17:12.423536+0000","last_undegraded":"2024-04-22T01:17:12.423536+0000","last_fullsized":"2024-04-22T01:17:12.423536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:52.655050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.386977+0000","last_change":"2024-04-22T01:17:11.390057+0000","last_active":"2024-04-22T01:17:12.386977+0000","last_peered":"2024-04-22T01:17:12.386977+0000","last_clean":"2024-04-22T01:17:12.386977+0000","last_became_active":"2024-04-22T01:17:11.389754+0000","last_became_peered":"2024-04-22T01:17:11.389754+0000","last_unstale":"2024-04-22T01:17:12.386977+0000","last_undegraded":"2024-04-22T01:17:12.386977+0000","last_fullsized":"2024-04-22T01:17:12.386977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:34.585950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.433817+0000","last_change":"2024-04-22T01:17:12.433308+0000","last_active":"2024-04-22T01:17:12.433817+0000","last_peered":"2024-04-22T01:17:12.433817+0000","last_clean":"2024-04-22T01:17:12.433817+0000","last_became_active":"2024-04-22T01:17:12.433031+0000","last_became_peered":"2024-04-22T01:17:12.433031+0000","last_unstale":"2024-04-22T01:17:12.433817+0000","last_undegraded":"2024-04-22T01:17:12.433817+0000","last_fullsized":"2024-04-22T01:17:12.433817+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:47.688688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.385290+0000","last_change":"2024-04-22T01:17:11.391173+0000","last_active":"2024-04-22T01:17:13.385290+0000","last_peered":"2024-04-22T01:17:13.385290+0000","last_clean":"2024-04-22T01:17:13.385290+0000","last_became_active":"2024-04-22T01:17:11.390471+0000","last_became_peered":"2024-04-22T01:17:11.390471+0000","last_unstale":"2024-04-22T01:17:13.385290+0000","last_undegraded":"2024-04-22T01:17:13.385290+0000","last_fullsized":"2024-04-22T01:17:13.385290+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:38:53.540328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.398448+0000","last_change":"2024-04-22T01:17:12.394789+0000","last_active":"2024-04-22T01:17:12.398448+0000","last_peered":"2024-04-22T01:17:12.398448+0000","last_clean":"2024-04-22T01:17:12.398448+0000","last_became_active":"2024-04-22T01:17:12.393902+0000","last_became_peered":"2024-04-22T01:17:12.393902+0000","last_unstale":"2024-04-22T01:17:12.398448+0000","last_undegraded":"2024-04-22T01:17:12.398448+0000","last_fullsized":"2024-04-22T01:17:12.398448+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:02.722073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.384043+0000","last_change":"2024-04-22T01:17:11.388136+0000","last_active":"2024-04-22T01:17:13.384043+0000","last_peered":"2024-04-22T01:17:13.384043+0000","last_clean":"2024-04-22T01:17:13.384043+0000","last_became_active":"2024-04-22T01:17:11.387341+0000","last_became_peered":"2024-04-22T01:17:11.387341+0000","last_unstale":"2024-04-22T01:17:13.384043+0000","last_undegraded":"2024-04-22T01:17:13.384043+0000","last_fullsized":"2024-04-22T01:17:13.384043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:04.216084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383634+0000","last_change":"2024-04-22T01:17:12.409872+0000","last_active":"2024-04-22T01:17:13.383634+0000","last_peered":"2024-04-22T01:17:13.383634+0000","last_clean":"2024-04-22T01:17:13.383634+0000","last_became_active":"2024-04-22T01:17:12.408721+0000","last_became_peered":"2024-04-22T01:17:12.408721+0000","last_unstale":"2024-04-22T01:17:13.383634+0000","last_undegraded":"2024-04-22T01:17:13.383634+0000","last_fullsized":"2024-04-22T01:17:13.383634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:54.957068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.388913+0000","last_change":"2024-04-22T01:17:11.387959+0000","last_active":"2024-04-22T01:17:12.388913+0000","last_peered":"2024-04-22T01:17:12.388913+0000","last_clean":"2024-04-22T01:17:12.388913+0000","last_became_active":"2024-04-22T01:17:11.387690+0000","last_became_peered":"2024-04-22T01:17:11.387690+0000","last_unstale":"2024-04-22T01:17:12.388913+0000","last_undegraded":"2024-04-22T01:17:12.388913+0000","last_fullsized":"2024-04-22T01:17:12.388913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:53.953955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413504+0000","last_change":"2024-04-22T01:17:12.403223+0000","last_active":"2024-04-22T01:17:12.413504+0000","last_peered":"2024-04-22T01:17:12.413504+0000","last_clean":"2024-04-22T01:17:12.413504+0000","last_became_active":"2024-04-22T01:17:12.399642+0000","last_became_peered":"2024-04-22T01:17:12.399642+0000","last_unstale":"2024-04-22T01:17:12.413504+0000","last_undegraded":"2024-04-22T01:17:12.413504+0000","last_fullsized":"2024-04-22T01:17:12.413504+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:34.131480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.392545+0000","last_change":"2024-04-22T01:17:11.378152+0000","last_active":"2024-04-22T01:17:12.392545+0000","last_peered":"2024-04-22T01:17:12.392545+0000","last_clean":"2024-04-22T01:17:12.392545+0000","last_became_active":"2024-04-22T01:17:11.377873+0000","last_became_peered":"2024-04-22T01:17:11.377873+0000","last_unstale":"2024-04-22T01:17:12.392545+0000","last_undegraded":"2024-04-22T01:17:12.392545+0000","last_fullsized":"2024-04-22T01:17:12.392545+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:25:59.972666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423887+0000","last_change":"2024-04-22T01:17:12.414769+0000","last_active":"2024-04-22T01:17:12.423887+0000","last_peered":"2024-04-22T01:17:12.423887+0000","last_clean":"2024-04-22T01:17:12.423887+0000","last_became_active":"2024-04-22T01:17:12.412987+0000","last_became_peered":"2024-04-22T01:17:12.412987+0000","last_unstale":"2024-04-22T01:17:12.423887+0000","last_undegraded":"2024-04-22T01:17:12.423887+0000","last_fullsized":"2024-04-22T01:17:12.423887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:48.268764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.434371+0000","last_change":"2024-04-22T01:17:11.401689+0000","last_active":"2024-04-22T01:17:12.434371+0000","last_peered":"2024-04-22T01:17:12.434371+0000","last_clean":"2024-04-22T01:17:12.434371+0000","last_became_active":"2024-04-22T01:17:11.398314+0000","last_became_peered":"2024-04-22T01:17:11.398314+0000","last_unstale":"2024-04-22T01:17:12.434371+0000","last_undegraded":"2024-04-22T01:17:12.434371+0000","last_fullsized":"2024-04-22T01:17:12.434371+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:15:19.727063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.383909+0000","last_change":"2024-04-22T01:17:12.453300+0000","last_active":"2024-04-22T01:17:13.383909+0000","last_peered":"2024-04-22T01:17:13.383909+0000","last_clean":"2024-04-22T01:17:13.383909+0000","last_became_active":"2024-04-22T01:17:12.452978+0000","last_became_peered":"2024-04-22T01:17:12.452978+0000","last_unstale":"2024-04-22T01:17:13.383909+0000","last_undegraded":"2024-04-22T01:17:13.383909+0000","last_fullsized":"2024-04-22T01:17:13.383909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:02.844083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381400+0000","last_change":"2024-04-22T01:17:11.364206+0000","last_active":"2024-04-22T01:17:13.381400+0000","last_peered":"2024-04-22T01:17:13.381400+0000","last_clean":"2024-04-22T01:17:13.381400+0000","last_became_active":"2024-04-22T01:17:11.363927+0000","last_became_peered":"2024-04-22T01:17:11.363927+0000","last_unstale":"2024-04-22T01:17:13.381400+0000","last_undegraded":"2024-04-22T01:17:13.381400+0000","last_fullsized":"2024-04-22T01:17:13.381400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:03.195821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.437783+0000","last_change":"2024-04-22T01:17:12.437352+0000","last_active":"2024-04-22T01:17:12.437783+0000","last_peered":"2024-04-22T01:17:12.437783+0000","last_clean":"2024-04-22T01:17:12.437783+0000","last_became_active":"2024-04-22T01:17:12.437117+0000","last_became_peered":"2024-04-22T01:17:12.437117+0000","last_unstale":"2024-04-22T01:17:12.437783+0000","last_undegraded":"2024-04-22T01:17:12.437783+0000","last_fullsized":"2024-04-22T01:17:12.437783+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:38:54.543441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381567+0000","last_change":"2024-04-22T01:17:11.379804+0000","last_active":"2024-04-22T01:17:13.381567+0000","last_peered":"2024-04-22T01:17:13.381567+0000","last_clean":"2024-04-22T01:17:13.381567+0000","last_became_active":"2024-04-22T01:17:11.379535+0000","last_became_peered":"2024-04-22T01:17:11.379535+0000","last_unstale":"2024-04-22T01:17:13.381567+0000","last_undegraded":"2024-04-22T01:17:13.381567+0000","last_fullsized":"2024-04-22T01:17:13.381567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:07.429497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.413049+0000","last_change":"2024-04-22T01:17:12.412556+0000","last_active":"2024-04-22T01:17:12.413049+0000","last_peered":"2024-04-22T01:17:12.413049+0000","last_clean":"2024-04-22T01:17:12.413049+0000","last_became_active":"2024-04-22T01:17:12.412350+0000","last_became_peered":"2024-04-22T01:17:12.412350+0000","last_unstale":"2024-04-22T01:17:12.413049+0000","last_undegraded":"2024-04-22T01:17:12.413049+0000","last_fullsized":"2024-04-22T01:17:12.413049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:24.052822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.393744+0000","last_change":"2024-04-22T01:17:11.373985+0000","last_active":"2024-04-22T01:17:12.393744+0000","last_peered":"2024-04-22T01:17:12.393744+0000","last_clean":"2024-04-22T01:17:12.393744+0000","last_became_active":"2024-04-22T01:17:11.373409+0000","last_became_peered":"2024-04-22T01:17:11.373409+0000","last_unstale":"2024-04-22T01:17:12.393744+0000","last_undegraded":"2024-04-22T01:17:12.393744+0000","last_fullsized":"2024-04-22T01:17:12.393744+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:38.136712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.423730+0000","last_change":"2024-04-22T01:17:12.413141+0000","last_active":"2024-04-22T01:17:12.423730+0000","last_peered":"2024-04-22T01:17:12.423730+0000","last_clean":"2024-04-22T01:17:12.423730+0000","last_became_active":"2024-04-22T01:17:12.404399+0000","last_became_peered":"2024-04-22T01:17:12.404399+0000","last_unstale":"2024-04-22T01:17:12.423730+0000","last_undegraded":"2024-04-22T01:17:12.423730+0000","last_fullsized":"2024-04-22T01:17:12.423730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:11.345785+0000","last_clean_scrub_stamp":"2024-04-22T01:17:11.345785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:56.892456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:17:13.381029+0000","last_change":"2024-04-22T01:17:11.363253+0000","last_active":"2024-04-22T01:17:13.381029+0000","last_peered":"2024-04-22T01:17:13.381029+0000","last_clean":"2024-04-22T01:17:13.381029+0000","last_became_active":"2024-04-22T01:17:11.362658+0000","last_became_peered":"2024-04-22T01:17:11.362658+0000","last_unstale":"2024-04-22T01:17:13.381029+0000","last_undegraded":"2024-04-22T01:17:13.381029+0000","last_fullsized":"2024-04-22T01:17:13.381029+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:10.342672+0000","last_clean_scrub_stamp":"2024-04-22T01:17:10.342672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:44.643803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":245,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:17:12.421135+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:17:12.421135+0000","last_peered":"2024-04-22T01:17:12.421135+0000","last_clean":"2024-04-22T01:17:12.421135+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:17:12.421135+0000","last_undegraded":"2024-04-22T01:17:12.421135+0000","last_fullsized":"2024-04-22T01:17:12.421135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1372160,"data_stored":933682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":342572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":765952,"data_stored":342560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":344150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":342538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36088,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335752,"statfs":{"total":96636764160,"available":96599810048,"internally_reserved":0,"allocated":761856,"data_stored":342538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":344914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":932906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:17:15.766 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:17:15.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-22T01:17:15.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 Processor -- start 2024-04-22T01:17:15.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 -- start start 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c099d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6f7c0cd280 0x7f6f7c09a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6f7c0b43e0 0x7f6f7c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6f7c0c5720 con 0x7f6f7c0cd280 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6f7c0cd710 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f81afb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7c0cd9f0 con 0x7f6f7c15f3e0 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6f7c0cd280 0x7f6f7c09a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6f7c0cd280 0x7f6f7c09a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42170/0 (socket says 172.21.15.70:42170) 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 -- 172.21.15.70:0/2997215324 learned_addr learned my addr 172.21.15.70:0/2997215324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7b7fe640 1 --2- 172.21.15.70:0/2997215324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c099d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7bfff640 1 --2- 172.21.15.70:0/2997215324 >> v2:172.21.15.98:3300/0 conn(0x7f6f7c0b43e0 0x7f6f7c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 -- 172.21.15.70:0/2997215324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 msgr2=0x7f6f7c099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 --2- 172.21.15.70:0/2997215324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c099d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 -- 172.21.15.70:0/2997215324 >> v2:172.21.15.98:3300/0 conn(0x7f6f7c0b43e0 msgr2=0x7f6f7c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 --2- 172.21.15.70:0/2997215324 >> v2:172.21.15.98:3300/0 conn(0x7f6f7c0b43e0 0x7f6f7c0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7affd640 1 -- 172.21.15.70:0/2997215324 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c0cdcd0 con 0x7f6f7c0cd280 2024-04-22T01:17:15.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.938+0000 7f6f7bfff640 1 --2- 172.21.15.70:0/2997215324 >> v2:172.21.15.98:3300/0 conn(0x7f6f7c0b43e0 0x7f6f7c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:15.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f7affd640 1 --2- 172.21.15.70:0/2997215324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0cd280 0x7f6f7c09a270 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0cbdf0 tx=0x7f6f6401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839cf8635e7cc829 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f7a7fc640 1 -- 172.21.15.70:0/2997215324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640aa430 con 0x7f6f7c0cd280 2024-04-22T01:17:15.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f7a7fc640 1 -- 172.21.15.70:0/2997215324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f640af070 con 0x7f6f7c0cd280 2024-04-22T01:17:15.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f7a7fc640 1 -- 172.21.15.70:0/2997215324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640aa7f0 con 0x7f6f7c0cd280 2024-04-22T01:17:15.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 -- 172.21.15.70:0/2997215324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0cd280 msgr2=0x7f6f7c09a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 --2- 172.21.15.70:0/2997215324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0cd280 0x7f6f7c09a270 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0cbdf0 tx=0x7f6f6401c4f0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 -- 172.21.15.70:0/2997215324 shutdown_connections 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 --2- 172.21.15.70:0/2997215324 >> v2:172.21.15.98:3300/0 conn(0x7f6f7c0b43e0 0x7f6f7c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 --2- 172.21.15.70:0/2997215324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0cd280 0x7f6f7c09a270 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 --2- 172.21.15.70:0/2997215324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c099d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 -- 172.21.15.70:0/2997215324 >> 172.21.15.70:0/2997215324 conn(0x7f6f7c0b36c0 msgr2=0x7f6f7c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 -- 172.21.15.70:0/2997215324 shutdown_connections 2024-04-22T01:17:15.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 -- 172.21.15.70:0/2997215324 wait complete. 2024-04-22T01:17:15.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 Processor -- start 2024-04-22T01:17:15.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.942+0000 7f6f81afb640 1 -- start start 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 0x7f6f7c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f7c154ca0 0x7f6f7c0c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c0c6d00 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c0c6b80 con 0x7f6f7c154ca0 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7c0c6e80 con 0x7f6f7c15f3e0 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 0x7f6f7c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 0x7f6f7c0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42172/0 (socket says 172.21.15.70:42172) 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 -- 172.21.15.70:0/1792637314 learned_addr learned my addr 172.21.15.70:0/1792637314 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7affd640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7bfff640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f7c154ca0 0x7f6f7c0c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 -- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 msgr2=0x7f6f7c0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c0b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 -- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f7c154ca0 msgr2=0x7f6f7c0c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f7c154ca0 0x7f6f7c0c4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 -- 172.21.15.70:0/1792637314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c159e80 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7b7fe640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 0x7f6f7c0c8dc0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6f74007e40 tx=0x7f6f7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74014620 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f74019070 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74014a00 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7c15a160 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7c15a720 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7c068920 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f74014ba0 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f73fff640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6f5c0678c0 0x7f6f5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:15.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f7affd640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6f5c0678c0 0x7f6f5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:15.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.946+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f6f740866e0 con 0x7f6f7c0b43e0 2024-04-22T01:17:15.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.950+0000 7f6f7affd640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6f5c0678c0 0x7f6f5c069d80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6f640ae000 tx=0x7f6f640a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:15.950+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6f74057090 con 0x7f6f7c0b43e0 2024-04-22T01:17:16.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.154+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f6f7c09b400 con 0x7f6f7c0b43e0 2024-04-22T01:17:16.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f73fff640 1 -- 172.21.15.70:0/1792637314 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f6f74056990 con 0x7f6f7c0b43e0 2024-04-22T01:17:16.158 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:17:16.158 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:17:16.158 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-22T01:17:16.158 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-22T01:17:16.158 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-22T01:17:16.159 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-22T01:17:16.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6f5c0678c0 msgr2=0x7f6f5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6f5c0678c0 0x7f6f5c069d80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6f640ae000 tx=0x7f6f640a7000 comp rx=0 tx=0).stop 2024-04-22T01:17:16.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 msgr2=0x7f6f7c0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 0x7f6f7c0c8dc0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6f74007e40 tx=0x7f6f7400e4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 shutdown_connections 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6f5c0678c0 0x7f6f5c069d80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f7c154ca0 0x7f6f7c0c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f7c15f3e0 0x7f6f7c0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 --2- 172.21.15.70:0/1792637314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f7c0b43e0 0x7f6f7c0c8dc0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 >> 172.21.15.70:0/1792637314 conn(0x7f6f7c0b36c0 msgr2=0x7f6f7c0c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 shutdown_connections 2024-04-22T01:17:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.158+0000 7f6f81afb640 1 -- 172.21.15.70:0/1792637314 wait complete. 2024-04-22T01:17:16.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:16.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.290+0000 7fa027831640 1 Processor -- start 2024-04-22T01:17:16.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.290+0000 7fa027831640 1 -- start start 2024-04-22T01:17:16.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa027831640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa02015dd90 0x7fa020099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:16.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa027831640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa02006d790 0x7fa02009a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:16.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa027831640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0200d03a0 con 0x7fa0200cfe50 2024-04-22T01:17:16.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa027831640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa0200d0fb0 con 0x7fa02015dd90 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa027831640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa0200d0680 con 0x7fa02006d790 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa02015dd90 0x7fa020099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa02015dd90 0x7fa020099cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60244/0 (socket says 172.21.15.70:60244) 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 -- 172.21.15.70:0/600303932 learned_addr learned my addr 172.21.15.70:0/600303932 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa024dcc640 1 --2- 172.21.15.70:0/600303932 >> v2:172.21.15.98:3300/0 conn(0x7fa02006d790 0x7fa02009a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 -- 172.21.15.70:0/600303932 >> v2:172.21.15.98:3300/0 conn(0x7fa02006d790 msgr2=0x7fa02009a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 --2- 172.21.15.70:0/600303932 >> v2:172.21.15.98:3300/0 conn(0x7fa02006d790 0x7fa02009a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 -- 172.21.15.70:0/600303932 >> v1:172.21.15.98:6789/0 conn(0x7fa0200cfe50 legacy=0x7fa020161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:16.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.294+0000 7fa0255cd640 1 -- 172.21.15.70:0/600303932 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0200d0960 con 0x7fa02015dd90 2024-04-22T01:17:16.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa0255cd640 1 --2- 172.21.15.70:0/600303932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa02015dd90 0x7fa020099cc0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa0200b7b70 tx=0x7fa01801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd071c75e7d5cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:16.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa01ffff640 1 -- 172.21.15.70:0/600303932 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0180aa430 con 0x7fa02015dd90 2024-04-22T01:17:16.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa01ffff640 1 -- 172.21.15.70:0/600303932 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0180af070 con 0x7fa02015dd90 2024-04-22T01:17:16.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa01ffff640 1 -- 172.21.15.70:0/600303932 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0180aa7f0 con 0x7fa02015dd90 2024-04-22T01:17:16.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- 172.21.15.70:0/600303932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa02015dd90 msgr2=0x7fa020099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 --2- 172.21.15.70:0/600303932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa02015dd90 0x7fa020099cc0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa0200b7b70 tx=0x7fa01801c4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- 172.21.15.70:0/600303932 shutdown_connections 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 --2- 172.21.15.70:0/600303932 >> v2:172.21.15.98:3300/0 conn(0x7fa02006d790 0x7fa02009a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 --2- 172.21.15.70:0/600303932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa02015dd90 0x7fa020099cc0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- 172.21.15.70:0/600303932 >> 172.21.15.70:0/600303932 conn(0x7fa0200b35d0 msgr2=0x7fa0200b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- 172.21.15.70:0/600303932 shutdown_connections 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- 172.21.15.70:0/600303932 wait complete. 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 Processor -- start 2024-04-22T01:17:16.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- start start 2024-04-22T01:17:16.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa02006d790 0x7fa020157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:16.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 0x7fa020154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02015dd90 0x7fa0200cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa02016b020 con 0x7fa0200cfe50 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa02016aea0 con 0x7fa02006d790 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa027831640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa02016b1a0 con 0x7fa02015dd90 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa024dcc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 0x7fa020154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa024dcc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 0x7fa020154cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60260/0 (socket says 172.21.15.70:60260) 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.298+0000 7fa024dcc640 1 -- 172.21.15.70:0/3049799122 learned_addr learned my addr 172.21.15.70:0/3049799122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa025dce640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02015dd90 0x7fa0200cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa0255cd640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa02006d790 0x7fa020157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa024dcc640 1 -- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02015dd90 msgr2=0x7fa0200cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa024dcc640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02015dd90 0x7fa0200cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa024dcc640 1 -- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa02006d790 msgr2=0x7fa020157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa024dcc640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa02006d790 0x7fa020157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa024dcc640 1 -- 172.21.15.70:0/3049799122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0200caba0 con 0x7fa0200cfe50 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa024dcc640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 0x7fa020154cd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa01400d620 tx=0x7fa01400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01400b620 con 0x7fa0200cfe50 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa014019070 con 0x7fa0200cfe50 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01400ba20 con 0x7fa0200cfe50 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0200cae80 con 0x7fa0200cfe50 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0200cb440 con 0x7fa0200cfe50 2024-04-22T01:17:16.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa020068920 con 0x7fa0200cfe50 2024-04-22T01:17:16.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa01400bbc0 con 0x7fa0200cfe50 2024-04-22T01:17:16.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa01dffb640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa00c0678c0 0x7fa00c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:16.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fa014086330 con 0x7fa0200cfe50 2024-04-22T01:17:16.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa0255cd640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa00c0678c0 0x7fa00c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:16.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.302+0000 7fa0255cd640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa00c0678c0 0x7fa00c069d80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa0180ae000 tx=0x7fa018008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:16.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.306+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa014056ce0 con 0x7fa0200cfe50 2024-04-22T01:17:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.490+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0200b2b00 con 0x7fa0200cfe50 2024-04-22T01:17:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.490+0000 7fa01dffb640 1 -- 172.21.15.70:0/3049799122 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5179 (secure 0 0 0) 0x7fa0140565e0 con 0x7fa0200cfe50 2024-04-22T01:17:16.492 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:16.492 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:11.712206+0000","modified":"2024-04-22T01:17:13.727780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6330},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6330":{"gid":6330,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/4194165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4194165476},{"type":"v1","addr":"172.21.15.70:6837","nonce":4194165476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6330,"qdb_cluster":[6330]},"id":4}]} 2024-04-22T01:17:16.492 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-22T01:17:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa00c0678c0 msgr2=0x7fa00c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa00c0678c0 0x7fa00c069d80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa0180ae000 tx=0x7fa018008040 comp rx=0 tx=0).stop 2024-04-22T01:17:16.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 msgr2=0x7fa020154cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 0x7fa020154cd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa01400d620 tx=0x7fa01400dae0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 shutdown_connections 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa00c0678c0 0x7fa00c069d80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02015dd90 0x7fa0200cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0200cfe50 0x7fa020154cd0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 --2- 172.21.15.70:0/3049799122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa02006d790 0x7fa020157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 >> 172.21.15.70:0/3049799122 conn(0x7fa0200b35d0 msgr2=0x7fa020070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 shutdown_connections 2024-04-22T01:17:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:16.494+0000 7fa027831640 1 -- 172.21.15.70:0/3049799122 wait complete. 2024-04-22T01:17:16.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:17:11.712206+0000', 'modified': '2024-04-22T01:17:13.727780+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6330}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6330': {'gid': 6330, 'name': 'c', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/4194165476', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 4194165476}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 4194165476}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6330, 'qdb_cluster': [6330]} 2024-04-22T01:17:16.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:17:16.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T01:17:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 Processor -- start 2024-04-22T01:17:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 -- start start 2024-04-22T01:17:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ca01445b0 0x7f6ca0072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 0x7f6ca00ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca0144990 con 0x7f6ca00b4360 2024-04-22T01:17:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6ca00b3bd0 con 0x7f6ca01445b0 2024-04-22T01:17:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca73fc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca00b2b50 con 0x7f6ca00b47f0 2024-04-22T01:17:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca4997640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6ca00b4360 0x7f6ca00cdce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49188/0 (socket says 172.21.15.70:49188) 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca4997640 1 -- 172.21.15.70:0/461952814 learned_addr learned my addr 172.21.15.70:0/461952814 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca5999640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 0x7f6ca00ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca5198640 1 --2- 172.21.15.70:0/461952814 >> v2:172.21.15.98:3300/0 conn(0x7f6ca01445b0 0x7f6ca0072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909264339 0 0) 0x7f6c90001120 con 0x7f6ca00b4360 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c8c003530 con 0x7f6ca00b4360 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2328228240 0 0) 0x7f6c8c003530 con 0x7f6ca00b4360 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 msgr2=0x7f6ca00ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 0x7f6ca00ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 >> v2:172.21.15.98:3300/0 conn(0x7f6ca01445b0 msgr2=0x7f6ca0072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 --2- 172.21.15.70:0/461952814 >> v2:172.21.15.98:3300/0 conn(0x7f6ca01445b0 0x7f6ca0072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca00b4c60 con 0x7f6ca00b4360 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6c90001850 con 0x7f6ca00b4360 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 >> v1:172.21.15.70:6789/0 conn(0x7f6ca00b4360 legacy=0x7f6ca00cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c003a30 0x7f6c8c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 0x7f6c8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c004d50 0x7f6c8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca00b3bd0 con 0x7f6c8c003a30 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca00b2b50 con 0x7f6c8c004d50 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c00b9d0 con 0x7f6ca00b47f0 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca5198640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c003a30 0x7f6c8c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca5999640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c004d50 0x7f6c8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.042+0000 7f6ca4997640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 0x7f6c8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5198640 1 -- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 msgr2=0x7f6c8c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5198640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ca00b47f0 0x7f6c8c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5198640 1 -- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c004d50 msgr2=0x7f6c8c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5198640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8c004d50 0x7f6c8c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5198640 1 -- 172.21.15.70:0/461952814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c003530 con 0x7f6c8c003a30 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5198640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c003a30 0x7f6c8c003e50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009ae0 tx=0x7f6c9c01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f88dbb4a60a55c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0b2070 con 0x7f6c8c003a30 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9c0a5650 con 0x7f6c8c003a30 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6c97fff640 1 -- 172.21.15.70:0/461952814 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0096b0 con 0x7f6c8c003a30 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca73fc640 1 -- 172.21.15.70:0/461952814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c003a30 msgr2=0x7f6c8c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/461952814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c8c003a30 0x7f6c8c003e50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009ae0 tx=0x7f6c9c01ca80 comp rx=0 tx=0).stop 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca5999640 1 -- 172.21.15.70:0/461952814 reap_dead start 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca73fc640 1 -- 172.21.15.70:0/461952814 shutdown_connections 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca73fc640 1 -- 172.21.15.70:0/461952814 >> 172.21.15.70:0/461952814 conn(0x7f6ca00cf9e0 msgr2=0x7f6ca00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca73fc640 1 -- 172.21.15.70:0/461952814 shutdown_connections 2024-04-22T01:17:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.046+0000 7f6ca73fc640 1 -- 172.21.15.70:0/461952814 wait complete. 2024-04-22T01:17:17.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 Processor -- start 2024-04-22T01:17:17.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 -- start start 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c003a30 0x7f6ca0136d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca006b250 0x7f6ca006b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 0x7f6ca01f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0072a40 con 0x7f6ca01f83e0 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca00b36b0 con 0x7f6ca006b250 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca006edb0 con 0x7f6c8c003a30 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 0x7f6ca01f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 0x7f6ca01f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60282/0 (socket says 172.21.15.70:60282) 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 -- 172.21.15.70:0/2851300642 learned_addr learned my addr 172.21.15.70:0/2851300642 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:17.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5198640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c003a30 0x7f6ca0136d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca4997640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca006b250 0x7f6ca006b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 -- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c003a30 msgr2=0x7f6ca0136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c003a30 0x7f6ca0136d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 -- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca006b250 msgr2=0x7f6ca006b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca006b250 0x7f6ca006b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 -- 172.21.15.70:0/2851300642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca015c690 con 0x7f6ca01f83e0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5999640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 0x7f6ca01f8860 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6c98007d10 tx=0x7f6c980115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6c95ffb640 1 -- 172.21.15.70:0/2851300642 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98011e40 con 0x7f6ca01f83e0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6c95ffb640 1 -- 172.21.15.70:0/2851300642 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9801b070 con 0x7f6ca01f83e0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca01592c0 con 0x7f6ca01f83e0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6c95ffb640 1 -- 172.21.15.70:0/2851300642 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c980039a0 con 0x7f6ca01f83e0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca0159880 con 0x7f6ca01f83e0 2024-04-22T01:17:17.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6c95ffb640 1 -- 172.21.15.70:0/2851300642 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c98003b40 con 0x7f6ca01f83e0 2024-04-22T01:17:17.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6c95ffb640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c840678f0 0x7f6c84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6c95ffb640 1 -- 172.21.15.70:0/2851300642 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f6c98086450 con 0x7f6ca01f83e0 2024-04-22T01:17:17.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.050+0000 7f6ca5198640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c840678f0 0x7f6c84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca5198640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c840678f0 0x7f6c84069db0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0af000 tx=0x7f6c9c0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 Processor -- start 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- start start 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 0x56386329dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638630eb5e0 con 0x5638632e7810 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563863072c50 con 0x56386329dbc0 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5638630e75a0 con 0x5638632e6ef0 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60298/0 (socket says 172.21.15.70:60298) 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 -- 172.21.15.70:0/664650275 learned_addr learned my addr 172.21.15.70:0/664650275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca5198640 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 -- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 msgr2=0x5638632e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632e72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 -- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 msgr2=0x56386329dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 0x56386329dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 -- 172.21.15.70:0/664650275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563863070d10 con 0x5638632e7810 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca5198640 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632e72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6ca4997640 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329d680 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6c90001b00 tx=0x7f6c90005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b740090bbd268aa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6c8affd640 1 -- 172.21.15.70:0/664650275 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90019070 con 0x5638632e7810 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6c8affd640 1 -- 172.21.15.70:0/664650275 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9000f9f0 con 0x5638632e7810 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6c8affd640 1 -- 172.21.15.70:0/664650275 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90010040 con 0x5638632e7810 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- 172.21.15.70:0/664650275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 msgr2=0x56386329d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329d680 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6c90001b00 tx=0x7f6c90005e50 comp rx=0 tx=0).stop 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- 172.21.15.70:0/664650275 shutdown_connections 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 0x56386329dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329d680 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 --2- 172.21.15.70:0/664650275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632e72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- 172.21.15.70:0/664650275 >> 172.21.15.70:0/664650275 conn(0x563863299b70 msgr2=0x56386329bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- 172.21.15.70:0/664650275 shutdown_connections 2024-04-22T01:17:17.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- 172.21.15.70:0/664650275 wait complete. 2024-04-22T01:17:17.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 -- start start 2024-04-22T01:17:17.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.054+0000 7f6caa575000 1 Processor -- start 2024-04-22T01:17:17.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 0x5638632cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e7810 0x5638632cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638630a5520 con 0x56386329dbc0 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638630b8340 con 0x5638632e7810 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5638630af670 con 0x5638632e6ef0 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 10 client.0 _ll_register_callbacks cb 0x5638632885c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 0x5638632cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 0x5638632cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60300/0 (socket says 172.21.15.70:60300) 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 -- 172.21.15.70:0/1406087484 learned_addr learned my addr 172.21.15.70:0/1406087484 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca4997640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5999640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e7810 0x5638632cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 -- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 msgr2=0x5638632cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 -- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e7810 msgr2=0x5638632cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e7810 0x5638632cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 -- 172.21.15.70:0/1406087484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638630837b0 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5999640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e7810 0x5638632cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca5198640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 0x5638632cbf20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c01cac0 tx=0x7f6c9c019820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1d8326f8a73c9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 1 -- 172.21.15.70:0/1406087484 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0c7060 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 1 -- 172.21.15.70:0/1406087484 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f6c9c0a5b20 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c9000f9f0 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 1 -- 172.21.15.70:0/1406087484 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c9c0cd050 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 10 client.6474 fetch_fsmap learned FSMap version 31 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) -- 0x563863072ea0 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 1 -- 172.21.15.70:0/1406087484 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9000f9f0 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 1 -- 172.21.15.70:0/1406087484 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 3494+0+0 (secure 0 0 0) 0x7f6c9c0d0020 con 0x56386329dbc0 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 10 client.6474 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T01:17:17.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 10 client.6474 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 20 client.6474 populate_metadata read hostname 'smithi070' 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632f6f90 0x5638632f9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 4 client.6474 mds_command: new command op to 6330 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 --> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563863086ab0 con 0x5638632f6f90 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca4997640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632f6f90 0x5638632f9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 10 client.6474 ms_handle_connect on v2:172.21.15.70:6836/4194165476 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6ca4997640 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632f6f90 0x5638632f9380 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0456a57d97f3bf9 server_cookie=28e8fef78dd3074e in_seq=0 out_seq=0 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 1 -- 172.21.15.70:0/1406087484 <== mds.0 v2:172.21.15.70:6836/4194165476 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563863086ab0 con 0x5638632f6f90 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 10 client.6474 handle_command_reply: tid=0 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 client.6474 shutdown 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 msgr2=0x5638632cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 0x5638632cbf20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c01cac0 tx=0x7f6c9c019820 comp rx=0 tx=0).stop 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 shutdown_connections 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e7810 0x5638632cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x5638632cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632f6f90 0x5638632f9380 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 --2- 172.21.15.70:0/1406087484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56386329dbc0 0x5638632cbf20 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 >> 172.21.15.70:0/1406087484 conn(0x563863299b70 msgr2=0x56386329c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6c7ffff640 0 client.6474 ms_handle_reset on v2:172.21.15.70:6836/4194165476 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 shutdown_connections 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 1 -- 172.21.15.70:0/1406087484 wait complete. 2024-04-22T01:17:17.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.058+0000 7f6caa575000 20 client.6474 trim_cache size 0 max 16384 2024-04-22T01:17:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 Processor -- start 2024-04-22T01:17:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- start start 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 0x56386329d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x56386329bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0200db0 con 0x5638632e7810 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca006e890 con 0x56386329dbc0 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca0200760 con 0x5638632e6ef0 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60316/0 (socket says 172.21.15.70:60316) 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 -- 172.21.15.70:0/3610214284 learned_addr learned my addr 172.21.15.70:0/3610214284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 -- 172.21.15.70:0/3610214284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 msgr2=0x56386329bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x56386329bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 -- 172.21.15.70:0/3610214284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 msgr2=0x56386329d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 0x56386329d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 -- 172.21.15.70:0/3610214284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563863073e80 con 0x5638632e7810 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6ca5999640 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329c4a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6c98056720 tx=0x7f6c98056690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548ec209fe909239 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6c8a7fc640 1 -- 172.21.15.70:0/3610214284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98056e90 con 0x5638632e7810 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6c8a7fc640 1 -- 172.21.15.70:0/3610214284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c98056790 con 0x5638632e7810 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6c8a7fc640 1 -- 172.21.15.70:0/3610214284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9805ab40 con 0x5638632e7810 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- 172.21.15.70:0/3610214284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 msgr2=0x56386329c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329c4a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6c98056720 tx=0x7f6c98056690 comp rx=0 tx=0).stop 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- 172.21.15.70:0/3610214284 shutdown_connections 2024-04-22T01:17:17.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x56386329c4a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5638632e6ef0 0x56386329bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 --2- 172.21.15.70:0/3610214284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56386329dbc0 0x56386329d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- 172.21.15.70:0/3610214284 >> 172.21.15.70:0/3610214284 conn(0x563863299b70 msgr2=0x5638632e57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- 172.21.15.70:0/3610214284 shutdown_connections 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.078+0000 7f6caa575000 1 -- 172.21.15.70:0/3610214284 wait complete. 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- start start 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 Processor -- start 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x56386329dbc0 0x5638633577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e6ef0 0x563863357d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x5638632f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5638630f7180 con 0x5638632e7810 2024-04-22T01:17:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638630b4810 con 0x5638632e6ef0 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x563863083dd0 con 0x56386329dbc0 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 10 client.0 _ll_register_callbacks cb 0x7f6c9c0b0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x5638632f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x5638632f9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60328/0 (socket says 172.21.15.70:60328) 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 -- 172.21.15.70:0/4221295596 learned_addr learned my addr 172.21.15.70:0/4221295596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5198640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x56386329dbc0 0x5638633577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca4997640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e6ef0 0x563863357d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 -- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x56386329dbc0 msgr2=0x5638633577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x56386329dbc0 0x5638633577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 -- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e6ef0 msgr2=0x563863357d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e6ef0 0x563863357d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 -- 172.21.15.70:0/4221295596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638630b8760 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5198640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x56386329dbc0 0x5638633577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca4997640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e6ef0 0x563863357d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5999640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x5638632f9af0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6c98056720 tx=0x7f6c98059040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809a2c45113a6a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 1 -- 172.21.15.70:0/4221295596 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98086270 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c98056790 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 1 -- 172.21.15.70:0/4221295596 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f6c98055be0 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 1 -- 172.21.15.70:0/4221295596 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c9805f370 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 10 client.6480 fetch_fsmap learned FSMap version 31 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) -- 0x5638631449e0 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 1 -- 172.21.15.70:0/4221295596 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c98056790 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 1 -- 172.21.15.70:0/4221295596 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 3494+0+0 (secure 0 0 0) 0x7f6c980524e0 con 0x5638632e7810 2024-04-22T01:17:17.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 10 client.6480 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 10 client.6480 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 20 client.6480 populate_metadata read hostname 'smithi070' 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632fa030 0x5638632fa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 4 client.6480 mds_command: new command op to 6330 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 --> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] -- command(tid 0: {"prefix": "status"}) -- 0x5638630f8b20 con 0x5638632fa030 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5198640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632fa030 0x5638632fa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 10 client.6480 ms_handle_connect on v2:172.21.15.70:6836/4194165476 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca5198640 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632fa030 0x5638632fa410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a9bcdaeab4ac11 server_cookie=39f148fbb818cb94 in_seq=0 out_seq=0 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 1 -- 172.21.15.70:0/4221295596 <== mds.0 v2:172.21.15.70:6836/4194165476 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5638630f8b20 con 0x5638632fa030 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 10 client.6480 handle_command_reply: tid=0 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 client.6480 shutdown 2024-04-22T01:17:17.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 msgr2=0x5638632f9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x5638632f9af0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6c98056720 tx=0x7f6c98059040 comp rx=0 tx=0).stop 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 shutdown_connections 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:6836/4194165476,v1:172.21.15.70:6837/4194165476] conn(0x5638632fa030 0x5638632fa410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5638632e7810 0x5638632f9af0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638632e6ef0 0x563863357d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6c7ffff640 0 client.6480 ms_handle_reset on v2:172.21.15.70:6836/4194165476 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 --2- 172.21.15.70:0/4221295596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x56386329dbc0 0x5638633577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 >> 172.21.15.70:0/4221295596 conn(0x563863299b70 msgr2=0x5638632e6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 shutdown_connections 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 1 -- 172.21.15.70:0/4221295596 wait complete. 2024-04-22T01:17:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6caa575000 20 client.6480 trim_cache size 0 max 16384 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 6330, 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 4.7094595200000002, 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 31, 2024-04-22T01:17:17.095 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 64, 2024-04-22T01:17:17.096 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 64, 2024-04-22T01:17:17.096 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 10.09072984 2024-04-22T01:17:17.096 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c840678f0 msgr2=0x7f6c84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c840678f0 0x7f6c84069db0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0af000 tx=0x7f6c9c0046d0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 msgr2=0x7f6ca01f8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 0x7f6ca01f8860 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6c98007d10 tx=0x7f6c980115a0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 shutdown_connections 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6c840678f0 0x7f6c84069db0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ca01f83e0 0x7f6ca01f8860 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca006b250 0x7f6ca006b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 --2- 172.21.15.70:0/2851300642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6c8c003a30 0x7f6ca0136d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.082+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 >> 172.21.15.70:0/2851300642 conn(0x7f6ca00cf9e0 msgr2=0x7f6ca01434f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.086+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 shutdown_connections 2024-04-22T01:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.086+0000 7f6ca73fc640 1 -- 172.21.15.70:0/2851300642 wait complete. 2024-04-22T01:17:17.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:17:17.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.214+0000 7efc48fda640 1 Processor -- start 2024-04-22T01:17:17.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.214+0000 7efc48fda640 1 -- start start 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc48fda640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efc440cea80 0x7efc440ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc48fda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc440d2110 0x7efc440d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc48fda640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc4415e390 con 0x7efc4415dfb0 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc48fda640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efc440cf420 con 0x7efc440cea80 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc48fda640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efc44161e00 con 0x7efc440d2110 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efc440cea80 0x7efc440ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc4259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efc4415dfb0 0x7efc440b95d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49198/0 (socket says 172.21.15.70:49198) 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efc440cea80 0x7efc440ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60336/0 (socket says 172.21.15.70:60336) 2024-04-22T01:17:17.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc4259c640 1 -- 172.21.15.70:0/3958709175 learned_addr learned my addr 172.21.15.70:0/3958709175 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:17.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc42d9d640 1 --2- 172.21.15.70:0/3958709175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc440d2110 0x7efc440d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 -- 172.21.15.70:0/3958709175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc440d2110 msgr2=0x7efc440d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:17.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 --2- 172.21.15.70:0/3958709175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc440d2110 0x7efc440d25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 -- 172.21.15.70:0/3958709175 >> v1:172.21.15.70:6789/0 conn(0x7efc4415dfb0 legacy=0x7efc440b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 -- 172.21.15.70:0/3958709175 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc441620e0 con 0x7efc440cea80 2024-04-22T01:17:17.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc41d9b640 1 -- 172.21.15.70:0/3958709175 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850374929 0 0) 0x7efc4415e390 con 0x7efc4415dfb0 2024-04-22T01:17:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc39d9b640 1 --2- 172.21.15.70:0/3958709175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc440ceee0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0097c0 tx=0x7efc2c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9daa86605610a6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc41d9b640 1 -- 172.21.15.70:0/3958709175 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0b1070 con 0x7efc440cea80 2024-04-22T01:17:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc41d9b640 1 -- 172.21.15.70:0/3958709175 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2c0ac410 con 0x7efc440cea80 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.218+0000 7efc41d9b640 1 -- 172.21.15.70:0/3958709175 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0ac7e0 con 0x7efc440cea80 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/3958709175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 msgr2=0x7efc440ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 --2- 172.21.15.70:0/3958709175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc440ceee0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0097c0 tx=0x7efc2c01c4a0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/3958709175 shutdown_connections 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 --2- 172.21.15.70:0/3958709175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc440d2110 0x7efc440d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 --2- 172.21.15.70:0/3958709175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc440ceee0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/3958709175 >> 172.21.15.70:0/3958709175 conn(0x7efc4406c9b0 msgr2=0x7efc4406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/3958709175 shutdown_connections 2024-04-22T01:17:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/3958709175 wait complete. 2024-04-22T01:17:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 Processor -- start 2024-04-22T01:17:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- start start 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc4415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440d2110 0x7efc44155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc4415dfb0 0x7efc440cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc4416af50 con 0x7efc440cea80 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc4416add0 con 0x7efc440d2110 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efc4416b0d0 con 0x7efc4415dfb0 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc4415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc4415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60342/0 (socket says 172.21.15.70:60342) 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 -- 172.21.15.70:0/827840365 learned_addr learned my addr 172.21.15.70:0/827840365 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 -- 172.21.15.70:0/827840365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc4415dfb0 msgr2=0x7efc440cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc4415dfb0 0x7efc440cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 -- 172.21.15.70:0/827840365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440d2110 msgr2=0x7efc44155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440d2110 0x7efc44155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 -- 172.21.15.70:0/827840365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4415ead0 con 0x7efc440cea80 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc4259c640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc4415e590 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efc3400c480 tx=0x7efc3400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34018070 con 0x7efc440cea80 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc34007500 con 0x7efc440cea80 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc440cb640 con 0x7efc440cea80 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34004a30 con 0x7efc440cea80 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc440cbc00 con 0x7efc440cea80 2024-04-22T01:17:17.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.222+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc44068210 con 0x7efc440cea80 2024-04-22T01:17:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.226+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc3401b070 con 0x7efc440cea80 2024-04-22T01:17:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.226+0000 7efc3b7fe640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc240678c0 0x7efc24069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.226+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7efc34085100 con 0x7efc440cea80 2024-04-22T01:17:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.226+0000 7efc39d9b640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc240678c0 0x7efc24069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.226+0000 7efc39d9b640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc240678c0 0x7efc24069d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0ae000 tx=0x7efc2c004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.230+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7efc34055b40 con 0x7efc440cea80 2024-04-22T01:17:17.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.410+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efc440b6450 con 0x7efc440cea80 2024-04-22T01:17:17.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.410+0000 7efc3b7fe640 1 -- 172.21.15.70:0/827840365 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) ==== 53+0+73 (secure 0 0 0) 0x7efc34055440 con 0x7efc440cea80 2024-04-22T01:17:17.412 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:17:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc240678c0 msgr2=0x7efc24069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc240678c0 0x7efc24069d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0ae000 tx=0x7efc2c004740 comp rx=0 tx=0).stop 2024-04-22T01:17:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 msgr2=0x7efc4415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc4415e590 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efc3400c480 tx=0x7efc3400c940 comp rx=0 tx=0).stop 2024-04-22T01:17:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 shutdown_connections 2024-04-22T01:17:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc240678c0 0x7efc24069d80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc4415dfb0 0x7efc440cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440d2110 0x7efc44155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 --2- 172.21.15.70:0/827840365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440cea80 0x7efc4415e590 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 >> 172.21.15.70:0/827840365 conn(0x7efc4406c9b0 msgr2=0x7efc440b5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 shutdown_connections 2024-04-22T01:17:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17.414+0000 7efc48fda640 1 -- 172.21.15.70:0/827840365 wait complete. 2024-04-22T01:17:17.431 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:17:17.431 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:17:17.432 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:17:17.432 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:17:17.432 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:17.432 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:17:17.432 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:17:17.433 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:17:17.433 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:17:17.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:17.433 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:17:17.439 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:17.440 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42384sec preferred_lft 42384sec 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.441 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:17.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:17.442 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:17:17.490 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:17:17.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:17.492 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:17:17.543 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:17:17.543 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:17.545 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:17.545 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:17:17.545 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:17:17.545 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:17:17.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:17:17.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:17:17.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:17:17.546 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:17:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:17:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:17.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:17.663 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:17:17.710 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:17:17.710 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:17:17.710 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:17:17.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:17.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:17:17.766 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:17:17.766 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:17.766 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:17.814 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:17:17.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:17.814 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:17.861 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:17:17.919 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:17:17.971 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:17:17.972 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:17.973 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:17.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:17.974 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:17:18.013 INFO:tasks.ceph.mds.c.smithi070.stderr:2024-04-22T01:17:18.014+0000 7f8d824cf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:17:18.018 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:17:18.018 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:17:18.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:17:18.081 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:18.129 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:18.130 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:17:18.131 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:17:18.132 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:17:18.132 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:17:18.132 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:18.132 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:18.132 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:18.132 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:18.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:18.133 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:17:18.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:17:18.142+0000 7f41692a4580 -1 init, newargv = 0x55fddbbf6c50 newargc=15 2024-04-22T01:17:18.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[41734]: starting ceph client 2024-04-22T01:17:18.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[41734]: starting fuse 2024-04-22T01:17:19.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:17:19.264 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:17:19.311 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:17:19.312 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:19.313 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:19.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:19.314 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:17:19.362 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:17:19.362 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:17:19.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:17:19.362 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:17:19.362 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:17:19.362 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:17:19.363 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:17:19.364 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:17:19.486 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.41734.asok 2024-04-22T01:17:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:19.498 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41734.asok 2024-04-22T01:17:19.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:19.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41734.asok status 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:17:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "41734", 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 6495, 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 6495 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1891456097 2024-04-22T01:17:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1891456097 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.6495 192.168.0.1:0/1891456097", 2024-04-22T01:17:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/1891456097", 2024-04-22T01:17:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:17:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 31, 2024-04-22T01:17:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 64, 2024-04-22T01:17:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:17:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:17:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:17:19.659 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:17:19.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:19.669 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:17:19.669 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:19.674 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:17:19.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:19.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:19.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:19.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:17:19.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.890+0000 7f4dad9f6640 1 Processor -- start 2024-04-22T01:17:19.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.890+0000 7f4dad9f6640 1 -- start start 2024-04-22T01:17:19.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4da80ceee0 0x7f4da80d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:19.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da815e390 con 0x7f4da815dfb0 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da8161e00 con 0x7f4da80cea80 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4da81620e0 con 0x7f4da80ceee0 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4da80ceee0 0x7f4da80d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da6ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4da815dfb0 0x7f4da80b95d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49206/0 (socket says 172.21.15.70:49206) 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da6ffd640 1 -- 172.21.15.70:0/3383197610 learned_addr learned my addr 172.21.15.70:0/3383197610 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da77fe640 1 -- 172.21.15.70:0/3383197610 >> v1:172.21.15.98:6789/0 conn(0x7f4da80cea80 legacy=0x7f4da80d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da77fe640 1 -- 172.21.15.70:0/3383197610 >> v1:172.21.15.70:6789/0 conn(0x7f4da815dfb0 legacy=0x7f4da80b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da77fe640 1 -- 172.21.15.70:0/3383197610 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da81623c0 con 0x7f4da80ceee0 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da77fe640 1 --2- 172.21.15.70:0/3383197610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80ceee0 0x7f4da80d2820 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009d90 tx=0x7f4da001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a398ba8c71ef4f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da5ffb640 1 -- 172.21.15.70:0/3383197610 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da00b2070 con 0x7f4da80ceee0 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da5ffb640 1 -- 172.21.15.70:0/3383197610 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da00a5900 con 0x7f4da80ceee0 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4da5ffb640 1 -- 172.21.15.70:0/3383197610 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da00a5cd0 con 0x7f4da80ceee0 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- 172.21.15.70:0/3383197610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80ceee0 msgr2=0x7f4da80d2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/3383197610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80ceee0 0x7f4da80d2820 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009d90 tx=0x7f4da001c930 comp rx=0 tx=0).stop 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- 172.21.15.70:0/3383197610 shutdown_connections 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/3383197610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80ceee0 0x7f4da80d2820 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- 172.21.15.70:0/3383197610 >> 172.21.15.70:0/3383197610 conn(0x7f4da806c9b0 msgr2=0x7f4da806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:19.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- 172.21.15.70:0/3383197610 shutdown_connections 2024-04-22T01:17:19.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.894+0000 7f4dad9f6640 1 -- 172.21.15.70:0/3383197610 wait complete. 2024-04-22T01:17:19.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 Processor -- start 2024-04-22T01:17:19.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- start start 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 0x7f4da8152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4da80ceee0 0x7f4da80caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da815dfb0 0x7f4da80cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4da816bf60 con 0x7f4da80cea80 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4da816bde0 con 0x7f4da815dfb0 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4da816c0e0 con 0x7f4da80ceee0 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 0x7f4da8152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 0x7f4da8152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60368/0 (socket says 172.21.15.70:60368) 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 -- 172.21.15.70:0/2808146115 learned_addr learned my addr 172.21.15.70:0/2808146115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 -- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4da80ceee0 msgr2=0x7f4da80caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4da80ceee0 0x7f4da80caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:19.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da77fe640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da815dfb0 0x7f4da80cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 -- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da815dfb0 msgr2=0x7f4da80cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da815dfb0 0x7f4da80cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 -- 172.21.15.70:0/2808146115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da815f020 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4da6ffd640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 0x7f4da8152d70 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800c3a0 tx=0x7f4d9800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98017070 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d98005cf0 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da815f300 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d980072f0 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da815f8c0 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.898+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da8068210 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.902+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4d9801a070 con 0x7f4da80cea80 2024-04-22T01:17:19.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.902+0000 7f4d8ffff640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d940678f0 0x7f4d94069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:19.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.902+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f4d98085210 con 0x7f4da80cea80 2024-04-22T01:17:19.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.902+0000 7f4da67fc640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d940678f0 0x7f4d94069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:19.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.902+0000 7f4da67fc640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d940678f0 0x7f4d94069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009650 tx=0x7f4d9c006bb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:19.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:19.906+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4d98055b30 con 0x7f4da80cea80 2024-04-22T01:17:20.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4da80d1950 con 0x7f4da80cea80 2024-04-22T01:17:20.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4d8ffff640 1 -- 172.21.15.70:0/2808146115 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) ==== 53+0+73 (secure 0 0 0) 0x7f4d98055430 con 0x7f4da80cea80 2024-04-22T01:17:20.102 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d940678f0 msgr2=0x7f4d94069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d940678f0 0x7f4d94069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009650 tx=0x7f4d9c006bb0 comp rx=0 tx=0).stop 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 msgr2=0x7f4da8152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 0x7f4da8152d70 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800c3a0 tx=0x7f4d9800c860 comp rx=0 tx=0).stop 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 shutdown_connections 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4d940678f0 0x7f4d94069db0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da815dfb0 0x7f4da80cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:20.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4da80ceee0 0x7f4da80caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:20.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 --2- 172.21.15.70:0/2808146115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4da80cea80 0x7f4da8152d70 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:20.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 >> 172.21.15.70:0/2808146115 conn(0x7f4da806c9b0 msgr2=0x7f4da80d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:20.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 shutdown_connections 2024-04-22T01:17:20.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:20.102+0000 7f4dad9f6640 1 -- 172.21.15.70:0/2808146115 wait complete. 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:17:20.119 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:17:20.120 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:17:20.120 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:17:20.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.120 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:17:20.128 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42386sec preferred_lft 42386sec 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:17:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:17:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:20.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.131 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:17:20.186 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:17:20.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.188 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:20.240 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:17:20.241 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:17:20.241 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:17:20.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:17:20.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:17:20.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:17:20.241 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:17:20.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:17:20.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:20.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.369 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:17:20.374 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:17:20.375 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:17:20.375 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:17:20.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:17:20.434 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:17:20.435 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:20.435 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:20.482 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:17:20.483 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:20.483 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:20.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:17:20.588 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:20.635 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:17:20.635 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:17:20.635 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:17:20.635 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:17:20.636 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:20.637 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:20.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.638 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:17:20.686 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:17:20.686 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:17:20.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:17:20.745 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:20.794 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:17:20.794 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:17:20.795 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:20.797 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:20.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:20.798 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:17:20.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:17:20.808+0000 7f7110272580 -1 init, newargv = 0x5598a5e3ddc0 newargc=15 2024-04-22T01:17:20.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34213]: starting ceph client 2024-04-22T01:17:20.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34213]: starting fuse 2024-04-22T01:17:21.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:17:21.932 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:17:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:17:21.980 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:21.981 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:21.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:21.982 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:17:22.030 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:17:22.030 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:17:22.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:17:22.030 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:22.031 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:17:22.032 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:17:22.155 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.34213.asok 2024-04-22T01:17:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:17:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:22.167 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34213.asok 2024-04-22T01:17:22.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34213.asok status 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:17:22.322 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "34213", 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5090, 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5090 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1912498596 2024-04-22T01:17:22.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1912498596 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5090 192.168.0.1:0/1912498596", 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/1912498596", 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 31, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 64, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:17:22.324 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:17:22.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.335 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:17:22.336 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:22.341 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:17:22.342 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:22.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:22.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.398 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-22T01:17:22.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.415 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:22.464 INFO:teuthology.orchestra.run.smithi070.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-04-22T01:17:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:17:22.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.465 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10K ./subdir) 2024-04-22T01:17:22.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.520 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:22.569 INFO:teuthology.orchestra.run.smithi070.stdout:10240 2024-04-22T01:17:22.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.569 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 100Ki ./subdir) 2024-04-22T01:17:22.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.629 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:22.682 INFO:teuthology.orchestra.run.smithi070.stdout:102400 2024-04-22T01:17:22.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.682 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10M ./subdir) 2024-04-22T01:17:22.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.737 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:22.790 INFO:teuthology.orchestra.run.smithi070.stdout:10485760 2024-04-22T01:17:22.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.790 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 100Mi ./subdir) 2024-04-22T01:17:22.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.845 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:22.897 INFO:teuthology.orchestra.run.smithi070.stdout:104857600 2024-04-22T01:17:22.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.898 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 2G ./subdir) 2024-04-22T01:17:22.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:22.953 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:23.007 INFO:teuthology.orchestra.run.smithi070.stdout:2147483648 2024-04-22T01:17:23.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:23.008 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 4Gi ./subdir) 2024-04-22T01:17:23.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:23.061 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:23.114 INFO:teuthology.orchestra.run.smithi070.stdout:4294967296 2024-04-22T01:17:23.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:23.115 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1T ./subdir) 2024-04-22T01:17:23.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:23.168 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:23.218 INFO:teuthology.orchestra.run.smithi070.stdout:1099511627776 2024-04-22T01:17:23.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:23.218 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 2Ti ./subdir) 2024-04-22T01:17:23.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:23.273 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-22T01:17:23.326 INFO:teuthology.orchestra.run.smithi070.stdout:2199023255552 2024-04-22T01:17:23.327 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:17:23.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:17:23.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:17:23.443 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:17:23.443 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:17:23.444 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:17:23.444 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:23.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:23.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.626+0000 7fd718db0640 1 Processor -- start 2024-04-22T01:17:23.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.626+0000 7fd718db0640 1 -- start start 2024-04-22T01:17:23.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd718db0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 0x7fd7140b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd718db0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71406dbd0 con 0x7fd7140aee40 2024-04-22T01:17:23.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd718db0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71406de70 con 0x7fd71415bb50 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd718db0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd71406e150 con 0x7fd71415aa30 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd7140aee40 0x7fd7140b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49210/0 (socket says 172.21.15.70:49210) 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 -- 172.21.15.70:0/3393509732 learned_addr learned my addr 172.21.15.70:0/3393509732 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71259c640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 0x7fd7140b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217855931 0 0) 0x7fd71406dbd0 con 0x7fd7140aee40 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6fc003530 con 0x7fd7140aee40 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3823578913 0 0) 0x7fd6fc003530 con 0x7fd7140aee40 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 msgr2=0x7fd7140b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 0x7fd7140b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 >> v1:172.21.15.98:6789/0 conn(0x7fd71415bb50 legacy=0x7fd7140ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71406e430 con 0x7fd7140aee40 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fd70c001a00 con 0x7fd7140aee40 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 >> v1:172.21.15.70:6789/0 conn(0x7fd7140aee40 legacy=0x7fd7140b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 0x7fd6fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd71406e150 con 0x7fd6fc004d50 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc00bb60 con 0x7fd6fc0039f0 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6fc00bd30 con 0x7fd71415aa30 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71259c640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd711d9b640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 0x7fd6fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 -- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 msgr2=0x7fd6fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd71415aa30 0x7fd6fc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 -- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc0039f0 msgr2=0x7fd6fc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 -- 172.21.15.70:0/3393509732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc003530 con 0x7fd6fc004d50 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd71259c640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.630+0000 7fd712d9d640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00a890 tx=0x7fd70c01d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e12ed26e0d3e2c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00a480 con 0x7fd6fc004d50 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70c0ad630 con 0x7fd6fc004d50 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd71159a640 1 -- 172.21.15.70:0/3393509732 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0ada40 con 0x7fd6fc004d50 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/3393509732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6fc004d50 msgr2=0x7fd6fc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 --2- 172.21.15.70:0/3393509732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00a890 tx=0x7fd70c01d380 comp rx=0 tx=0).stop 2024-04-22T01:17:23.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 -- 172.21.15.70:0/3393509732 reap_dead start 2024-04-22T01:17:23.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/3393509732 shutdown_connections 2024-04-22T01:17:23.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/3393509732 >> 172.21.15.70:0/3393509732 conn(0x7fd71406b1f0 msgr2=0x7fd71406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:23.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/3393509732 shutdown_connections 2024-04-22T01:17:23.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/3393509732 wait complete. 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 Processor -- start 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- start start 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc004d50 0x7fd71414c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd714145de0 0x7fd71414eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 0x7fd71414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd71415ae10 con 0x7fd71414f1e0 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd71415bfb0 con 0x7fd6fc004d50 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd71415af90 con 0x7fd714145de0 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 0x7fd71414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 0x7fd71414da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60396/0 (socket says 172.21.15.70:60396) 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 -- 172.21.15.70:0/280469107 learned_addr learned my addr 172.21.15.70:0/280469107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd711d9b640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd714145de0 0x7fd71414eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd71259c640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc004d50 0x7fd71414c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 -- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd714145de0 msgr2=0x7fd71414eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd714145de0 0x7fd71414eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 -- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc004d50 msgr2=0x7fd71414c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc004d50 0x7fd71414c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 -- 172.21.15.70:0/280469107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7141490c0 con 0x7fd71414f1e0 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd712d9d640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 0x7fd71414da60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00a890 tx=0x7fd6fc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0be070 con 0x7fd71414f1e0 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70c0b3040 con 0x7fd71414f1e0 2024-04-22T01:17:23.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00a480 con 0x7fd71414f1e0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd714148300 con 0x7fd71414f1e0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.634+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7141488c0 con 0x7fd71414f1e0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.638+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd714068210 con 0x7fd71414f1e0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.638+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd70c0c1070 con 0x7fd71414f1e0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.638+0000 7fd702ffd640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6f40678f0 0x7fd6f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.638+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fd70c1204b0 con 0x7fd71414f1e0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.638+0000 7fd71259c640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6f40678f0 0x7fd6f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.638+0000 7fd71259c640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6f40678f0 0x7fd6f4069db0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c890 tx=0x7fd704008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:23.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.642+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd70c017090 con 0x7fd71414f1e0 2024-04-22T01:17:23.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.806+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd71406b8e0 con 0x7fd71414f1e0 2024-04-22T01:17:23.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.806+0000 7fd702ffd640 1 -- 172.21.15.70:0/280469107 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16848 (secure 0 0 0) 0x7fd70c0f0ef0 con 0x7fd71414f1e0 2024-04-22T01:17:23.808 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:23.808 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:13.375919+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6f40678f0 msgr2=0x7fd6f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6f40678f0 0x7fd6f4069db0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c890 tx=0x7fd704008040 comp rx=0 tx=0).stop 2024-04-22T01:17:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 msgr2=0x7fd71414da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 0x7fd71414da60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00a890 tx=0x7fd6fc005840 comp rx=0 tx=0).stop 2024-04-22T01:17:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 shutdown_connections 2024-04-22T01:17:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6f40678f0 0x7fd6f4069db0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd71414f1e0 0x7fd71414da60 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd714145de0 0x7fd71414eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 --2- 172.21.15.70:0/280469107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6fc004d50 0x7fd71414c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 >> 172.21.15.70:0/280469107 conn(0x7fd71406b1f0 msgr2=0x7fd71415b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 shutdown_connections 2024-04-22T01:17:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.810+0000 7fd718db0640 1 -- 172.21.15.70:0/280469107 wait complete. 2024-04-22T01:17:23.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:09.983208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:10.713378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:23.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:23.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 Processor -- start 2024-04-22T01:17:23.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 -- start start 2024-04-22T01:17:23.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8bdc0b4450 0x7f8bdc071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0720b0 0x7f8bdc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8bdc0cd050 0x7f8bdc15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8bdc072510 con 0x7f8bdc0cd050 2024-04-22T01:17:23.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8bdc15c290 con 0x7f8bdc0b4450 2024-04-22T01:17:23.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8be13e5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8bdc15c570 con 0x7f8bdc0720b0 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8bdc0cd050 0x7f8bdc15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8bdc0cd050 0x7f8bdc15f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60398/0 (socket says 172.21.15.70:60398) 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/4199151411 learned_addr learned my addr 172.21.15.70:0/4199151411 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bda7fc640 1 --2- 172.21.15.70:0/4199151411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0720b0 0x7f8bdc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdaffd640 1 --2- 172.21.15.70:0/4199151411 >> v2:172.21.15.98:3300/0 conn(0x7f8bdc0b4450 0x7f8bdc071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/4199151411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0720b0 msgr2=0x7f8bdc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 --2- 172.21.15.70:0/4199151411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0720b0 0x7f8bdc0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/4199151411 >> v2:172.21.15.98:3300/0 conn(0x7f8bdc0b4450 msgr2=0x7f8bdc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 --2- 172.21.15.70:0/4199151411 >> v2:172.21.15.98:3300/0 conn(0x7f8bdc0b4450 0x7f8bdc071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.938+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/4199151411 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bdc15c850 con 0x7f8bdc0cd050 2024-04-22T01:17:23.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdb7fe640 1 --2- 172.21.15.70:0/4199151411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc15f770 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc06fce0 tx=0x7f8bd401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e87fcd269af2d51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:23.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bd9ffb640 1 -- 172.21.15.70:0/4199151411 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40a5510 con 0x7f8bdc0cd050 2024-04-22T01:17:23.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bd9ffb640 1 -- 172.21.15.70:0/4199151411 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd40a56b0 con 0x7f8bdc0cd050 2024-04-22T01:17:23.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bd9ffb640 1 -- 172.21.15.70:0/4199151411 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40a5a80 con 0x7f8bdc0cd050 2024-04-22T01:17:23.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- 172.21.15.70:0/4199151411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 msgr2=0x7f8bdc15f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- 172.21.15.70:0/4199151411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc15f770 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc06fce0 tx=0x7f8bd401c700 comp rx=0 tx=0).stop 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- 172.21.15.70:0/4199151411 shutdown_connections 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- 172.21.15.70:0/4199151411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc15f770 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- 172.21.15.70:0/4199151411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0720b0 0x7f8bdc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- 172.21.15.70:0/4199151411 >> v2:172.21.15.98:3300/0 conn(0x7f8bdc0b4450 0x7f8bdc071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- 172.21.15.70:0/4199151411 >> 172.21.15.70:0/4199151411 conn(0x7f8bdc0b1d40 msgr2=0x7f8bdc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- 172.21.15.70:0/4199151411 shutdown_connections 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- 172.21.15.70:0/4199151411 wait complete. 2024-04-22T01:17:23.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 Processor -- start 2024-04-22T01:17:23.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- start start 2024-04-22T01:17:23.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bdc0720b0 0x7f8bdc14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0b4450 0x7f8bdc1554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc0b49b0 con 0x7f8bdc0cd050 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc0b4830 con 0x7f8bdc0720b0 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8be13e5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8bdc0cd800 con 0x7f8bdc0b4450 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60406/0 (socket says 172.21.15.70:60406) 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/1261791455 learned_addr learned my addr 172.21.15.70:0/1261791455 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdaffd640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bdc0720b0 0x7f8bdc14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bda7fc640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0b4450 0x7f8bdc1554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0b4450 msgr2=0x7f8bdc1554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.942+0000 7f8bdb7fe640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0b4450 0x7f8bdc1554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bdc0720b0 msgr2=0x7f8bdc14db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:23.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bdb7fe640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bdc0720b0 0x7f8bdc14db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bdb7fe640 1 -- 172.21.15.70:0/1261791455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bdc14e150 con 0x7f8bdc0cd050 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bdb7fe640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc146aa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8bd401cfa0 tx=0x7f8bd40194a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40bc020 con 0x7f8bdc0cd050 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd40b8690 con 0x7f8bdc0cd050 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40a5510 con 0x7f8bdc0cd050 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bdc14e430 con 0x7f8bdc0cd050 2024-04-22T01:17:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bdc14e9f0 con 0x7f8bdc0cd050 2024-04-22T01:17:23.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bdc068210 con 0x7f8bdc0cd050 2024-04-22T01:17:23.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8bd40bf020 con 0x7f8bdc0cd050 2024-04-22T01:17:23.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bc37fe640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8bc80678c0 0x7f8bc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:23.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bdaffd640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8bc80678c0 0x7f8bc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:23.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f8bd4120ab0 con 0x7f8bdc0cd050 2024-04-22T01:17:23.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.946+0000 7f8bdaffd640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8bc80678c0 0x7f8bc8069d80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009990 tx=0x7f8bcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:23.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:23.950+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8bd40c6030 con 0x7f8bdc0cd050 2024-04-22T01:17:24.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8bdc0d01e0 con 0x7f8bdc0cd050 2024-04-22T01:17:24.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8bc37fe640 1 -- 172.21.15.70:0/1261791455 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) ==== 93+0+10 (secure 0 0 0) 0x7f8bd40165e0 con 0x7f8bdc0cd050 2024-04-22T01:17:24.111 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8bc80678c0 msgr2=0x7f8bc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8bc80678c0 0x7f8bc8069d80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009990 tx=0x7f8bcc008040 comp rx=0 tx=0).stop 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 msgr2=0x7f8bdc146aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc146aa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8bd401cfa0 tx=0x7f8bd40194a0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 shutdown_connections 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.110+0000 7f8be13e5640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8bc80678c0 0x7f8bc8069d80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.114+0000 7f8be13e5640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bdc0cd050 0x7f8bdc146aa0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.114+0000 7f8be13e5640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bdc0b4450 0x7f8bdc1554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.114+0000 7f8be13e5640 1 --2- 172.21.15.70:0/1261791455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bdc0720b0 0x7f8bdc14db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.114+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 >> 172.21.15.70:0/1261791455 conn(0x7f8bdc0b1d40 msgr2=0x7f8bdc0b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:24.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.114+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 shutdown_connections 2024-04-22T01:17:24.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.114+0000 7f8be13e5640 1 -- 172.21.15.70:0/1261791455 wait complete. 2024-04-22T01:17:24.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:17:24.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.246+0000 7f13c37ed640 1 Processor -- start 2024-04-22T01:17:24.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.246+0000 7f13c37ed640 1 -- start start 2024-04-22T01:17:24.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c37ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13bc155370 0x7f13bc06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c37ed640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13bc0b3800 con 0x7f13bc15e6e0 2024-04-22T01:17:24.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c37ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13bc0b3ae0 con 0x7f13bc0ce920 2024-04-22T01:17:24.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c37ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f13bc0cde40 con 0x7f13bc155370 2024-04-22T01:17:24.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1589640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f13bc15e6e0 0x7f13bc15eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49226/0 (socket says 172.21.15.70:49226) 2024-04-22T01:17:24.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1589640 1 -- 172.21.15.70:0/1988848995 learned_addr learned my addr 172.21.15.70:0/1988848995 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:24.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4116220572 0 0) 0x7f13bc0b3800 con 0x7f13bc15e6e0 2024-04-22T01:17:24.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1988848995 >> v2:172.21.15.98:3300/0 conn(0x7f13bc155370 0x7f13bc06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13a8003530 con 0x7f13bc15e6e0 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 382403193 0 0) 0x7f13a8003530 con 0x7f13bc15e6e0 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 >> v2:172.21.15.98:3300/0 conn(0x7f13bc155370 msgr2=0x7f13bc06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 --2- 172.21.15.70:0/1988848995 >> v2:172.21.15.98:3300/0 conn(0x7f13bc155370 0x7f13bc06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 >> v1:172.21.15.98:6789/0 conn(0x7f13bc0ce920 legacy=0x7f13bc154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13bc0ce120 con 0x7f13bc15e6e0 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f13b8001680 con 0x7f13bc15e6e0 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 >> v1:172.21.15.70:6789/0 conn(0x7f13bc15e6e0 legacy=0x7f13bc15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13a80039c0 0x7f13a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a8004810 0x7f13a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a80051d0 0x7f13a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc0cde40 con 0x7f13a80051d0 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13a800c070 con 0x7f13a8004810 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f13a800c270 con 0x7f13a80039c0 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a80051d0 0x7f13a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c0d88640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a8004810 0x7f13a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1589640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13a80039c0 0x7f13a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13a80039c0 msgr2=0x7f13a8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13a80039c0 0x7f13a8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a8004810 msgr2=0x7f13a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a8004810 0x7f13a8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1988848995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a8003530 con 0x7f13a80051d0 2024-04-22T01:17:24.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.250+0000 7f13c0d88640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a8004810 0x7f13a8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1988848995 reap_dead start 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a80051d0 0x7f13a800b9d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009fc0 tx=0x7f13b401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4544b12a2d1cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4009c10 con 0x7f13a80051d0 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b40a5d10 con 0x7f13a80051d0 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13b3fff640 1 -- 172.21.15.70:0/1988848995 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4003be0 con 0x7f13a80051d0 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1988848995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a80051d0 msgr2=0x7f13a800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a80051d0 0x7f13a800b9d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009fc0 tx=0x7f13b401ccc0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1988848995 shutdown_connections 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1988848995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a80051d0 0x7f13a800b9d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1988848995 >> 172.21.15.70:0/1988848995 conn(0x7f13bc0b1d40 msgr2=0x7f13bc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1988848995 shutdown_connections 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1988848995 wait complete. 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 Processor -- start 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- start start 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a80051d0 0x7f13bc141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13bc151fc0 0x7f13bc152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 0x7f13bc15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc06f260 con 0x7f13bc15c6e0 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc06f0e0 con 0x7f13a80051d0 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f13bc0cedd0 con 0x7f13bc151fc0 2024-04-22T01:17:24.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 0x7f13bc15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 0x7f13bc15cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60434/0 (socket says 172.21.15.70:60434) 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1354170303 learned_addr learned my addr 172.21.15.70:0/1354170303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c0d88640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13bc151fc0 0x7f13bc152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1589640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a80051d0 0x7f13bc141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13bc151fc0 msgr2=0x7f13bc152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13bc151fc0 0x7f13bc152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a80051d0 msgr2=0x7f13bc141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a80051d0 0x7f13bc141240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 -- 172.21.15.70:0/1354170303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13bc15d2f0 con 0x7f13bc15c6e0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c1d8a640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 0x7f13bc15cb60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f13b40aeb40 tx=0x7f13b4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4009c10 con 0x7f13bc15c6e0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b40bd070 con 0x7f13bc15c6e0 2024-04-22T01:17:24.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13bc15d5d0 con 0x7f13bc15c6e0 2024-04-22T01:17:24.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b40039a0 con 0x7f13bc15c6e0 2024-04-22T01:17:24.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.254+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13bc15db90 con 0x7f13bc15c6e0 2024-04-22T01:17:24.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.258+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13bc068210 con 0x7f13bc15c6e0 2024-04-22T01:17:24.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.258+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13b40a5d10 con 0x7f13bc15c6e0 2024-04-22T01:17:24.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.258+0000 7f13b1ffb640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a00678f0 0x7f13a0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.258+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f13b4121680 con 0x7f13bc15c6e0 2024-04-22T01:17:24.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.258+0000 7f13c1589640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a00678f0 0x7f13a0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.258+0000 7f13c1589640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a00678f0 0x7f13a0069db0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f13b8001930 tx=0x7f13b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.262+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f13b40c7020 con 0x7f13bc15c6e0 2024-04-22T01:17:24.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.410+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f13bc0d08e0 con 0x7f13bc15c6e0 2024-04-22T01:17:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.410+0000 7f13b1ffb640 1 -- 172.21.15.70:0/1354170303 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) ==== 104+0+11 (secure 0 0 0) 0x7f13b40169d0 con 0x7f13bc15c6e0 2024-04-22T01:17:24.413 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a00678f0 msgr2=0x7f13a0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a00678f0 0x7f13a0069db0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f13b8001930 tx=0x7f13b8009040 comp rx=0 tx=0).stop 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 msgr2=0x7f13bc15cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 0x7f13bc15cb60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f13b40aeb40 tx=0x7f13b4009750 comp rx=0 tx=0).stop 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 shutdown_connections 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a00678f0 0x7f13a0069db0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13bc15c6e0 0x7f13bc15cb60 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13bc151fc0 0x7f13bc152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 --2- 172.21.15.70:0/1354170303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a80051d0 0x7f13bc141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 >> 172.21.15.70:0/1354170303 conn(0x7f13bc0b1d40 msgr2=0x7f13bc15eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 shutdown_connections 2024-04-22T01:17:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.414+0000 7f13c37ed640 1 -- 172.21.15.70:0/1354170303 wait complete. 2024-04-22T01:17:24.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:17:24.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.550+0000 7f040e60e640 1 Processor -- start 2024-04-22T01:17:24.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.550+0000 7f040e60e640 1 -- start start 2024-04-22T01:17:24.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040e60e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f0408166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040e60e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040815c290 con 0x7f0408072280 2024-04-22T01:17:24.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040e60e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040815c570 con 0x7f04080b4450 2024-04-22T01:17:24.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040e60e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f04080b3a50 con 0x7f04080cd220 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0408072280 0x7f04080ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49232/0 (socket says 172.21.15.70:49232) 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 -- 172.21.15.70:0/2566782269 learned_addr learned my addr 172.21.15.70:0/2566782269 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414255646 0 0) 0x7f040815c290 con 0x7f0408072280 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f0003530 con 0x7f0408072280 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040cbab640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f0408166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3790833219 0 0) 0x7f03f0003530 con 0x7f0408072280 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 msgr2=0x7f0408166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f0408166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 >> v1:172.21.15.98:6789/0 conn(0x7f04080b4450 legacy=0x7f0408071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040815c850 con 0x7f0408072280 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f03fc001680 con 0x7f0408072280 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 >> v1:172.21.15.70:6789/0 conn(0x7f0408072280 legacy=0x7f04080ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f03f00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f03f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f0004d50 0x7f03f00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04080b3a50 con 0x7f03f0004020 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03f000bb90 con 0x7f03f0004d50 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f03f000bd90 con 0x7f04080cd220 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f03f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0407fff640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f03f00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040cbab640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f0004d50 0x7f03f00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 -- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 msgr2=0x7f03f00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f03f00039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 -- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f0004d50 msgr2=0x7f03f00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f0004d50 0x7f03f00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 -- 172.21.15.70:0/2566782269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f0003530 con 0x7f03f0004020 2024-04-22T01:17:24.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f0407fff640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04080cd220 0x7f03f00039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:24.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f040cbab640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f0004d50 0x7f03f00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.554+0000 7f04077fe640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f03f0004810 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c1c0 tx=0x7f03fc01f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88797255989f24bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc007d00 con 0x7f03f0004020 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03fc005b00 con 0x7f03f0004020 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0406ffd640 1 -- 172.21.15.70:0/2566782269 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc007070 con 0x7f03f0004020 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/2566782269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 msgr2=0x7f03f0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 --2- 172.21.15.70:0/2566782269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f03f0004810 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c1c0 tx=0x7f03fc01f3d0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040cbab640 1 -- 172.21.15.70:0/2566782269 reap_dead start 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/2566782269 shutdown_connections 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/2566782269 >> 172.21.15.70:0/2566782269 conn(0x7f04080b1d40 msgr2=0x7f04080cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/2566782269 shutdown_connections 2024-04-22T01:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/2566782269 wait complete. 2024-04-22T01:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 Processor -- start 2024-04-22T01:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- start start 2024-04-22T01:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f0408150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f040813e360 0x7f040814ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040814f3c0 0x7f04081492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04080b4830 con 0x7f03f0004020 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f040815d050 con 0x7f040814f3c0 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f04080b49b0 con 0x7f040813e360 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f0408150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f0408150f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60454/0 (socket says 172.21.15.70:60454) 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 -- 172.21.15.70:0/3948388167 learned_addr learned my addr 172.21.15.70:0/3948388167 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f04077fe640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f040813e360 0x7f040814ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 -- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f040813e360 msgr2=0x7f040814ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040cbab640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040814f3c0 0x7f04081492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f040813e360 0x7f040814ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 -- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040814f3c0 msgr2=0x7f04081492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040814f3c0 0x7f04081492c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 -- 172.21.15.70:0/3948388167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040813e7c0 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0407fff640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f0408150f20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f03f800d770 tx=0x7f03f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8005270 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f8017070 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f800b6b0 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0408151550 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.558+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0408151b10 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.562+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0408068210 con 0x7f03f0004020 2024-04-22T01:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.562+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f03f800b850 con 0x7f03f0004020 2024-04-22T01:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.562+0000 7f0404ff9640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03f40678c0 0x7f03f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.562+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f03f8085d80 con 0x7f03f0004020 2024-04-22T01:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.562+0000 7f04077fe640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03f40678c0 0x7f03f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.562+0000 7f04077fe640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03f40678c0 0x7f03f4069d80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0ae000 tx=0x7f040815c460 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.566+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f03f8019310 con 0x7f03f0004020 2024-04-22T01:17:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.730+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f04080d01e0 con 0x7f03f0004020 2024-04-22T01:17:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.730+0000 7f0404ff9640 1 -- 172.21.15.70:0/3948388167 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) ==== 100+0+11 (secure 0 0 0) 0x7f03f8056730 con 0x7f03f0004020 2024-04-22T01:17:24.732 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:17:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03f40678c0 msgr2=0x7f03f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03f40678c0 0x7f03f4069d80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0ae000 tx=0x7f040815c460 comp rx=0 tx=0).stop 2024-04-22T01:17:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 msgr2=0x7f0408150f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f0408150f20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f03f800d770 tx=0x7f03f800dc30 comp rx=0 tx=0).stop 2024-04-22T01:17:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 shutdown_connections 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03f40678c0 0x7f03f4069d80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040814f3c0 0x7f04081492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f040813e360 0x7f040814ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 --2- 172.21.15.70:0/3948388167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03f0004020 0x7f0408150f20 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 >> 172.21.15.70:0/3948388167 conn(0x7f04080b1d40 msgr2=0x7f04080b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 shutdown_connections 2024-04-22T01:17:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.734+0000 7f040e60e640 1 -- 172.21.15.70:0/3948388167 wait complete. 2024-04-22T01:17:24.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:17:24.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f87a45640 1 Processor -- start 2024-04-22T01:17:24.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f87a45640 1 -- start start 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f87a45640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8f8015bf30 0x7f8f8006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f87a45640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f800cd7a0 con 0x7f8f8015bb50 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f87a45640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8015fb00 con 0x7f8f801605e0 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f87a45640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8f800cd580 con 0x7f8f8015bf30 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f84fe0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8f8015bf30 0x7f8f8006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f857e1640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8f8015bb50 0x7f8f800cce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49242/0 (socket says 172.21.15.70:49242) 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f857e1640 1 -- 172.21.15.70:0/2564581747 learned_addr learned my addr 172.21.15.70:0/2564581747 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f84fe0640 1 -- 172.21.15.70:0/2564581747 >> v1:172.21.15.98:6789/0 conn(0x7f8f801605e0 legacy=0x7f8f80071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f84fe0640 1 -- 172.21.15.70:0/2564581747 >> v1:172.21.15.70:6789/0 conn(0x7f8f8015bb50 legacy=0x7f8f800cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f73fff640 1 -- 172.21.15.70:0/2564581747 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2205204554 0 0) 0x7f8f800cd7a0 con 0x7f8f8015bb50 2024-04-22T01:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f84fe0640 1 -- 172.21.15.70:0/2564581747 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8015fde0 con 0x7f8f8015bf30 2024-04-22T01:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f84fe0640 1 --2- 172.21.15.70:0/2564581747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f8015bf30 0x7f8f8006ed70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0097f0 tx=0x7f8f7c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d8fe0b62bbaac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f73fff640 1 -- 172.21.15.70:0/2564581747 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0b0070 con 0x7f8f8015bf30 2024-04-22T01:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f73fff640 1 -- 172.21.15.70:0/2564581747 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7c0ab600 con 0x7f8f8015bf30 2024-04-22T01:17:24.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.866+0000 7f8f73fff640 1 -- 172.21.15.70:0/2564581747 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0ab9d0 con 0x7f8f8015bf30 2024-04-22T01:17:24.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/2564581747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f8015bf30 msgr2=0x7f8f8006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 --2- 172.21.15.70:0/2564581747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f8015bf30 0x7f8f8006ed70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0097f0 tx=0x7f8f7c01c3d0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/2564581747 shutdown_connections 2024-04-22T01:17:24.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 --2- 172.21.15.70:0/2564581747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f8015bf30 0x7f8f8006ed70 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/2564581747 >> 172.21.15.70:0/2564581747 conn(0x7f8f800cf9e0 msgr2=0x7f8f800cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:24.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/2564581747 shutdown_connections 2024-04-22T01:17:24.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/2564581747 wait complete. 2024-04-22T01:17:24.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 Processor -- start 2024-04-22T01:17:24.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- start start 2024-04-22T01:17:24.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8015bb50 0x7f8f80152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f8015bf30 0x7f8f8015df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 0x7f8f8013cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8015f8b0 con 0x7f8f801605e0 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80169e70 con 0x7f8f8015bb50 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8f80169ff0 con 0x7f8f8015bf30 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 0x7f8f8013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 0x7f8f8013cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60470/0 (socket says 172.21.15.70:60470) 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 -- 172.21.15.70:0/4087426655 learned_addr learned my addr 172.21.15.70:0/4087426655 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f857e1640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8015bb50 0x7f8f80152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f84fe0640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f8015bf30 0x7f8f8015df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 -- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f8015bf30 msgr2=0x7f8f8015df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f8015bf30 0x7f8f8015df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 -- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8015bb50 msgr2=0x7f8f80152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8015bb50 0x7f8f80152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 -- 172.21.15.70:0/4087426655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f80158cc0 con 0x7f8f801605e0 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f857e1640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8015bb50 0x7f8f80152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f85fe2640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 0x7f8f8013cf90 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800b990 tx=0x7f8f7800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78019070 con 0x7f8f801605e0 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7800cba0 con 0x7f8f801605e0 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78014440 con 0x7f8f801605e0 2024-04-22T01:17:24.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f80158fa0 con 0x7f8f801605e0 2024-04-22T01:17:24.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.870+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f801f5520 con 0x7f8f801605e0 2024-04-22T01:17:24.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.874+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8004f510 con 0x7f8f801605e0 2024-04-22T01:17:24.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.874+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8f7801c070 con 0x7f8f801605e0 2024-04-22T01:17:24.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.874+0000 7f8f71ffb640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8f640678c0 0x7f8f64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:24.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.874+0000 7f8f857e1640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8f640678c0 0x7f8f64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:24.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.874+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f8f78085ba0 con 0x7f8f801605e0 2024-04-22T01:17:24.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.874+0000 7f8f857e1640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8f640678c0 0x7f8f64069d80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8f74000f70 tx=0x7f8f74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:24.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:24.878+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8f78056550 con 0x7f8f801605e0 2024-04-22T01:17:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8f800d0740 con 0x7f8f801605e0 2024-04-22T01:17:25.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f71ffb640 1 -- 172.21.15.70:0/4087426655 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) ==== 80+0+0 (secure 0 0 0) 0x7f8f78055e50 con 0x7f8f801605e0 2024-04-22T01:17:25.050 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:17:25.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8f640678c0 msgr2=0x7f8f64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8f640678c0 0x7f8f64069d80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8f74000f70 tx=0x7f8f74009040 comp rx=0 tx=0).stop 2024-04-22T01:17:25.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 msgr2=0x7f8f8013cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 0x7f8f8013cf90 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800b990 tx=0x7f8f7800be50 comp rx=0 tx=0).stop 2024-04-22T01:17:25.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 shutdown_connections 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8f640678c0 0x7f8f64069d80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f801605e0 0x7f8f8013cf90 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f8015bf30 0x7f8f8015df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 --2- 172.21.15.70:0/4087426655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8015bb50 0x7f8f80152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 >> 172.21.15.70:0/4087426655 conn(0x7f8f800cf9e0 msgr2=0x7f8f80153e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 shutdown_connections 2024-04-22T01:17:25.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.050+0000 7f8f87a45640 1 -- 172.21.15.70:0/4087426655 wait complete. 2024-04-22T01:17:25.069 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:17:25.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:25.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:25.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[41734]: fuse finished with error 0 and tester_r 0 2024-04-22T01:17:25.082 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:17:25.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:25.082 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:17:25.140 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:17:25.140 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:25.140 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:25.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:25.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.302+0000 7ff18ca91640 1 Processor -- start 2024-04-22T01:17:25.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.302+0000 7ff18ca91640 1 -- start start 2024-04-22T01:17:25.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18ca91640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff188072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18ca91640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff18806ed40 0x7ff1880b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18ca91640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff18815bf20 con 0x7ff1880b3b00 2024-04-22T01:17:25.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18ca91640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff18806e8b0 con 0x7ff18806ed40 2024-04-22T01:17:25.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18ca91640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff1880aec30 con 0x7ff18815bb40 2024-04-22T01:17:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff188072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff185d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff18806ed40 0x7ff1880b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff186d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff1880b3b00 0x7ff188161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:32994/0 (socket says 172.21.15.70:32994) 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff186d9d640 1 -- 172.21.15.70:0/2096138562 learned_addr learned my addr 172.21.15.70:0/2096138562 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff185d9b640 1 -- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 msgr2=0x7ff188072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff185d9b640 1 --2- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff188072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff185d9b640 1 -- 172.21.15.70:0/2096138562 >> v1:172.21.15.98:6789/0 conn(0x7ff1880b3b00 legacy=0x7ff188161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff185d9b640 1 -- 172.21.15.70:0/2096138562 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1880aef10 con 0x7ff18806ed40 2024-04-22T01:17:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.306+0000 7ff18659c640 1 --2- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff188072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:17:25.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff185d9b640 1 --2- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff18806ed40 0x7ff1880b35c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0097c0 tx=0x7ff17c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54d1432499da73a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18559a640 1 -- 172.21.15.70:0/2096138562 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0b2070 con 0x7ff18806ed40 2024-04-22T01:17:25.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18559a640 1 -- 172.21.15.70:0/2096138562 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff17c0ad640 con 0x7ff18806ed40 2024-04-22T01:17:25.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18559a640 1 -- 172.21.15.70:0/2096138562 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0ada10 con 0x7ff18806ed40 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff18806ed40 msgr2=0x7ff1880b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 --2- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff18806ed40 0x7ff1880b35c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0097c0 tx=0x7ff17c01c4a0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- 172.21.15.70:0/2096138562 shutdown_connections 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 --2- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff18806ed40 0x7ff1880b35c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 --2- 172.21.15.70:0/2096138562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff188072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- 172.21.15.70:0/2096138562 >> 172.21.15.70:0/2096138562 conn(0x7ff18806b1f0 msgr2=0x7ff18806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- 172.21.15.70:0/2096138562 shutdown_connections 2024-04-22T01:17:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- 172.21.15.70:0/2096138562 wait complete. 2024-04-22T01:17:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 Processor -- start 2024-04-22T01:17:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- start start 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 0x7ff18814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1880b3b00 0x7ff188145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff1881460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff18806f740 con 0x7ff18806ed40 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff18806f5c0 con 0x7ff1880b3b00 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18ca91640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff18816ae60 con 0x7ff18815bb40 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 0x7ff18814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 0x7ff18814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60486/0 (socket says 172.21.15.70:60486) 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 -- 172.21.15.70:0/4150214260 learned_addr learned my addr 172.21.15.70:0/4150214260 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff186d9d640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff1881460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff185d9b640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1880b3b00 0x7ff188145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 -- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 msgr2=0x7ff1881460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff1881460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 -- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1880b3b00 msgr2=0x7ff188145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1880b3b00 0x7ff188145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 -- 172.21.15.70:0/4150214260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18814fd90 con 0x7ff18806ed40 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff185d9b640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1880b3b00 0x7ff188145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.310+0000 7ff18659c640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 0x7ff18814d920 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff17800e980 tx=0x7ff17800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178017070 con 0x7ff18806ed40 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff17800c9a0 con 0x7ff18806ed40 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18814a090 con 0x7ff18806ed40 2024-04-22T01:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17800cd70 con 0x7ff18806ed40 2024-04-22T01:17:25.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18814a650 con 0x7ff18806ed40 2024-04-22T01:17:25.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff188068210 con 0x7ff18806ed40 2024-04-22T01:17:25.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff17801a070 con 0x7ff18806ed40 2024-04-22T01:17:25.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff176ffd640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1680678c0 0x7ff168069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff185d9b640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1680678c0 0x7ff168069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7ff178085cd0 con 0x7ff18806ed40 2024-04-22T01:17:25.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.314+0000 7ff185d9b640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1680678c0 0x7ff168069d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0af000 tx=0x7ff17c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.318+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff178056680 con 0x7ff18806ed40 2024-04-22T01:17:25.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.478+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff18814e700 con 0x7ff18806ed40 2024-04-22T01:17:25.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.482+0000 7ff176ffd640 1 -- 172.21.15.70:0/4150214260 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16848 (secure 0 0 0) 0x7ff178055f80 con 0x7ff18806ed40 2024-04-22T01:17:25.483 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:25.483 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:13.375919+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.482+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1680678c0 msgr2=0x7ff168069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.482+0000 7ff18ca91640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1680678c0 0x7ff168069d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0af000 tx=0x7ff17c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.482+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 msgr2=0x7ff18814d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.482+0000 7ff18ca91640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 0x7ff18814d920 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff17800e980 tx=0x7ff17800ee40 comp rx=0 tx=0).stop 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 shutdown_connections 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff1680678c0 0x7ff168069d80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff18815bb40 0x7ff1881460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1880b3b00 0x7ff188145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 --2- 172.21.15.70:0/4150214260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff18806ed40 0x7ff18814d920 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 >> 172.21.15.70:0/4150214260 conn(0x7ff18806b1f0 msgr2=0x7ff1880b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:25.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 shutdown_connections 2024-04-22T01:17:25.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.486+0000 7ff18ca91640 1 -- 172.21.15.70:0/4150214260 wait complete. 2024-04-22T01:17:25.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:09.983208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:10.713378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:25.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:25.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.622+0000 7f773536b640 1 Processor -- start 2024-04-22T01:17:25.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.622+0000 7f773536b640 1 -- start start 2024-04-22T01:17:25.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f773536b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 0x7f77300b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f773536b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77300b6db0 0x7f77300d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f773536b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77300b75e0 con 0x7f77300cea10 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f773536b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f77300b78c0 con 0x7f77300b6db0 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f773536b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f77300b7ba0 con 0x7f773015de20 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f77300cea10 0x7f77300ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49246/0 (socket says 172.21.15.70:49246) 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 -- 172.21.15.70:0/3980855965 learned_addr learned my addr 172.21.15.70:0/3980855965 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772effd640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 0x7f77300b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772e7fc640 1 --2- 172.21.15.70:0/3980855965 >> v2:172.21.15.98:3300/0 conn(0x7f77300b6db0 0x7f77300d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545243839 0 0) 0x7f77300b75e0 con 0x7f77300cea10 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f771c003530 con 0x7f77300cea10 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1243824809 0 0) 0x7f771c003530 con 0x7f77300cea10 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 msgr2=0x7f77300b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 0x7f77300b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 >> v2:172.21.15.98:3300/0 conn(0x7f77300b6db0 msgr2=0x7f77300d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 --2- 172.21.15.70:0/3980855965 >> v2:172.21.15.98:3300/0 conn(0x7f77300b6db0 0x7f77300d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77300b7e80 con 0x7f77300cea10 2024-04-22T01:17:25.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7728001680 con 0x7f77300cea10 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 >> v1:172.21.15.70:6789/0 conn(0x7f77300cea10 legacy=0x7f77300ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 0x7f771c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771c003f40 0x7f771c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f771c004d50 0x7f771c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77300b78c0 con 0x7f771c004d50 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f77300b7ba0 con 0x7f771c003f40 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f771c00ba00 con 0x7f773015de20 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f771c004d50 0x7f771c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772effd640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 0x7f771c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772e7fc640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771c003f40 0x7f771c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 -- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 msgr2=0x7f771c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f773015de20 0x7f771c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 -- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771c003f40 msgr2=0x7f771c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f771c003f40 0x7f771c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.626+0000 7f772f7fe640 1 -- 172.21.15.70:0/3980855965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771c003530 con 0x7f771c004d50 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772f7fe640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f771c004d50 0x7f771c0051d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f772800c1c0 tx=0x7f772801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cd3ccff5756cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728007d00 con 0x7f771c004d50 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280a9020 con 0x7f771c004d50 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772dffb640 1 -- 172.21.15.70:0/3980855965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280a5b60 con 0x7f771c004d50 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- 172.21.15.70:0/3980855965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f771c004d50 msgr2=0x7f771c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 --2- 172.21.15.70:0/3980855965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f771c004d50 0x7f771c0051d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f772800c1c0 tx=0x7f772801cc60 comp rx=0 tx=0).stop 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772f7fe640 1 -- 172.21.15.70:0/3980855965 reap_dead start 2024-04-22T01:17:25.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- 172.21.15.70:0/3980855965 shutdown_connections 2024-04-22T01:17:25.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- 172.21.15.70:0/3980855965 >> 172.21.15.70:0/3980855965 conn(0x7f7730072d20 msgr2=0x7f7730073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:25.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- 172.21.15.70:0/3980855965 shutdown_connections 2024-04-22T01:17:25.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- 172.21.15.70:0/3980855965 wait complete. 2024-04-22T01:17:25.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 Processor -- start 2024-04-22T01:17:25.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- start start 2024-04-22T01:17:25.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7730158410 0x7f77301f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 0x7f77300cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77301f41e0 0x7f77301f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77300d22c0 con 0x7f77301587f0 2024-04-22T01:17:25.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f773015e200 con 0x7f77301f41e0 2024-04-22T01:17:25.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f773536b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f77300b71f0 con 0x7f7730158410 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 0x7f77300cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 0x7f77300cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60518/0 (socket says 172.21.15.70:60518) 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 -- 172.21.15.70:0/3029032767 learned_addr learned my addr 172.21.15.70:0/3029032767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772effd640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7730158410 0x7f77301f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772f7fe640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77301f41e0 0x7f77301f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 -- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7730158410 msgr2=0x7f77301f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7730158410 0x7f77301f7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 -- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77301f41e0 msgr2=0x7f77301f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77301f41e0 0x7f77301f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 -- 172.21.15.70:0/3029032767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77301f77a0 con 0x7f77301587f0 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772f7fe640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77301f41e0 0x7f77301f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.630+0000 7f772e7fc640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 0x7f77300cc870 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7718007f00 tx=0x7f77180115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771801a020 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7718016470 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77301f7a80 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718016860 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77301f3cc0 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7730068210 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f771801d020 con 0x7f77301587f0 2024-04-22T01:17:25.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f77277fe640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77100678c0 0x7f7710069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f7718086ce0 con 0x7f77301587f0 2024-04-22T01:17:25.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f772effd640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77100678c0 0x7f7710069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.634+0000 7f772effd640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77100678c0 0x7f7710069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f77300ca890 tx=0x7f7720008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.638+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7718024030 con 0x7f77301587f0 2024-04-22T01:17:25.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77300d4b30 con 0x7f77301587f0 2024-04-22T01:17:25.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f77277fe640 1 -- 172.21.15.70:0/3029032767 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) ==== 93+0+10 (secure 0 0 0) 0x7f7718057720 con 0x7f77301587f0 2024-04-22T01:17:25.802 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77100678c0 msgr2=0x7f7710069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77100678c0 0x7f7710069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f77300ca890 tx=0x7f7720008040 comp rx=0 tx=0).stop 2024-04-22T01:17:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 msgr2=0x7f77300cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 0x7f77300cc870 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7718007f00 tx=0x7f77180115a0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 shutdown_connections 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77100678c0 0x7f7710069d80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77301f41e0 0x7f77301f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77301587f0 0x7f77300cc870 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 --2- 172.21.15.70:0/3029032767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7730158410 0x7f77301f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 >> 172.21.15.70:0/3029032767 conn(0x7f7730072d20 msgr2=0x7f77300ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 shutdown_connections 2024-04-22T01:17:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.802+0000 7f773536b640 1 -- 172.21.15.70:0/3029032767 wait complete. 2024-04-22T01:17:25.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:17:25.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 Processor -- start 2024-04-22T01:17:25.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 -- start start 2024-04-22T01:17:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe14415e6e0 0x7fe14415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1440ce920 0x7fe144154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1440cedd0 0x7fe1441551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe14415f000 con 0x7fe14415e6e0 2024-04-22T01:17:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe1440b3800 con 0x7fe1440ce920 2024-04-22T01:17:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14bda7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe1440b3ae0 con 0x7fe1440cedd0 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe14415e6e0 0x7fe14415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe14415e6e0 0x7fe14415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60532/0 (socket says 172.21.15.70:60532) 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 -- 172.21.15.70:0/2734364200 learned_addr learned my addr 172.21.15.70:0/2734364200 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe14a344640 1 --2- 172.21.15.70:0/2734364200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1440cedd0 0x7fe1441551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149342640 1 --2- 172.21.15.70:0/2734364200 >> v2:172.21.15.98:3300/0 conn(0x7fe1440ce920 0x7fe144154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 -- 172.21.15.70:0/2734364200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1440cedd0 msgr2=0x7fe1441551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 --2- 172.21.15.70:0/2734364200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1440cedd0 0x7fe1441551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 -- 172.21.15.70:0/2734364200 >> v2:172.21.15.98:3300/0 conn(0x7fe1440ce920 msgr2=0x7fe144154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 --2- 172.21.15.70:0/2734364200 >> v2:172.21.15.98:3300/0 conn(0x7fe1440ce920 0x7fe144154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.946+0000 7fe149b43640 1 -- 172.21.15.70:0/2734364200 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1440cde40 con 0x7fe14415e6e0 2024-04-22T01:17:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe149b43640 1 --2- 172.21.15.70:0/2734364200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe14415e6e0 0x7fe14415eac0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe1440b26e0 tx=0x7fe14001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484d3a92eb4d7841 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe148b41640 1 -- 172.21.15.70:0/2734364200 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1400aa610 con 0x7fe14415e6e0 2024-04-22T01:17:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe148b41640 1 -- 172.21.15.70:0/2734364200 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1400af070 con 0x7fe14415e6e0 2024-04-22T01:17:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe148b41640 1 -- 172.21.15.70:0/2734364200 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1400aa9d0 con 0x7fe14415e6e0 2024-04-22T01:17:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 -- 172.21.15.70:0/2734364200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe14415e6e0 msgr2=0x7fe14415eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 --2- 172.21.15.70:0/2734364200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe14415e6e0 0x7fe14415eac0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe1440b26e0 tx=0x7fe14001c4f0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 -- 172.21.15.70:0/2734364200 shutdown_connections 2024-04-22T01:17:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 --2- 172.21.15.70:0/2734364200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1440cedd0 0x7fe1441551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 --2- 172.21.15.70:0/2734364200 >> v2:172.21.15.98:3300/0 conn(0x7fe1440ce920 0x7fe144154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 --2- 172.21.15.70:0/2734364200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe14415e6e0 0x7fe14415eac0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 -- 172.21.15.70:0/2734364200 >> 172.21.15.70:0/2734364200 conn(0x7fe1440b1d40 msgr2=0x7fe1440cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 -- 172.21.15.70:0/2734364200 shutdown_connections 2024-04-22T01:17:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 -- 172.21.15.70:0/2734364200 wait complete. 2024-04-22T01:17:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 Processor -- start 2024-04-22T01:17:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.950+0000 7fe14bda7640 1 -- start start 2024-04-22T01:17:25.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1440ce920 0x7fe144159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 0x7fe14415b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe14415e6e0 0x7fe14415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440b2c30 con 0x7fe1440cedd0 2024-04-22T01:17:25.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440b2ab0 con 0x7fe1440ce920 2024-04-22T01:17:25.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe1440b2db0 con 0x7fe14415e6e0 2024-04-22T01:17:25.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 0x7fe14415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 0x7fe14415b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60546/0 (socket says 172.21.15.70:60546) 2024-04-22T01:17:25.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 -- 172.21.15.70:0/59927104 learned_addr learned my addr 172.21.15.70:0/59927104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14a344640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe14415e6e0 0x7fe14415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149b43640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1440ce920 0x7fe144159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 -- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe14415e6e0 msgr2=0x7fe14415d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe14415e6e0 0x7fe14415d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 -- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1440ce920 msgr2=0x7fe144159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1440ce920 0x7fe144159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 -- 172.21.15.70:0/59927104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe144158a60 con 0x7fe1440cedd0 2024-04-22T01:17:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149342640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 0x7fe14415b880 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe13400d830 tx=0x7fe13400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134018020 con 0x7fe1440cedd0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe134009640 con 0x7fe1440cedd0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe144158d40 con 0x7fe1440cedd0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340099f0 con 0x7fe1440cedd0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe144159300 con 0x7fe1440cedd0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe144068210 con 0x7fe1440cedd0 2024-04-22T01:17:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe13401b020 con 0x7fe1440cedd0 2024-04-22T01:17:25.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe13a7fc640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1280678c0 0x7fe128069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:25.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fe1340865b0 con 0x7fe1440cedd0 2024-04-22T01:17:25.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149b43640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1280678c0 0x7fe128069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:25.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.954+0000 7fe149b43640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1280678c0 0x7fe128069d80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe1400ae000 tx=0x7fe140008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:25.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:25.962+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe134022020 con 0x7fe1440cedd0 2024-04-22T01:17:26.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.113+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe1440cca20 con 0x7fe1440cedd0 2024-04-22T01:17:26.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe13a7fc640 1 -- 172.21.15.70:0/59927104 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) ==== 104+0+11 (secure 0 0 0) 0x7fe134011340 con 0x7fe1440cedd0 2024-04-22T01:17:26.117 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:17:26.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1280678c0 msgr2=0x7fe128069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1280678c0 0x7fe128069d80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe1400ae000 tx=0x7fe140008040 comp rx=0 tx=0).stop 2024-04-22T01:17:26.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 msgr2=0x7fe14415b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 0x7fe14415b880 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe13400d830 tx=0x7fe13400dcf0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 shutdown_connections 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe1280678c0 0x7fe128069d80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe14415e6e0 0x7fe14415d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1440cedd0 0x7fe14415b880 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 --2- 172.21.15.70:0/59927104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1440ce920 0x7fe144159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 >> 172.21.15.70:0/59927104 conn(0x7fe1440b1d40 msgr2=0x7fe144153550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 shutdown_connections 2024-04-22T01:17:26.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.117+0000 7fe14bda7640 1 -- 172.21.15.70:0/59927104 wait complete. 2024-04-22T01:17:26.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:17:26.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.253+0000 7f87893e9640 1 Processor -- start 2024-04-22T01:17:26.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87893e9640 1 -- start start 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87893e9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8784071b70 0x7f8784072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87893e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f87840cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87893e9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f878406da70 con 0x7f878415a320 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87893e9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f878406f860 con 0x7f8784071b70 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87893e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f878406dd50 con 0x7f87840cd410 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f877bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8784071b70 0x7f8784072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f8782ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f878415a320 0x7f878406f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37208/0 (socket says 172.21.15.70:37208) 2024-04-22T01:17:26.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f8782ffd640 1 -- 172.21.15.70:0/3206084989 learned_addr learned my addr 172.21.15.70:0/3206084989 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:26.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87837fe640 1 --2- 172.21.15.70:0/3206084989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f87840cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f877bfff640 1 -- 172.21.15.70:0/3206084989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 msgr2=0x7f87840cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:26.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f877bfff640 1 --2- 172.21.15.70:0/3206084989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f87840cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f877bfff640 1 -- 172.21.15.70:0/3206084989 >> v1:172.21.15.70:6789/0 conn(0x7f878415a320 legacy=0x7f878406f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f877bfff640 1 -- 172.21.15.70:0/3206084989 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878406e030 con 0x7f8784071b70 2024-04-22T01:17:26.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f87827fc640 1 -- 172.21.15.70:0/3206084989 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511348531 0 0) 0x7f878406da70 con 0x7f878415a320 2024-04-22T01:17:26.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.257+0000 7f877bfff640 1 --2- 172.21.15.70:0/3206084989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8784071b70 0x7f8784072000 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f876c0097c0 tx=0x7f876c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719a488049b4736b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:26.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87827fc640 1 -- 172.21.15.70:0/3206084989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0b2070 con 0x7f8784071b70 2024-04-22T01:17:26.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87827fc640 1 -- 172.21.15.70:0/3206084989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876c0ad640 con 0x7f8784071b70 2024-04-22T01:17:26.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87827fc640 1 -- 172.21.15.70:0/3206084989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0ada10 con 0x7f8784071b70 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/3206084989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8784071b70 msgr2=0x7f8784072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 --2- 172.21.15.70:0/3206084989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8784071b70 0x7f8784072000 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f876c0097c0 tx=0x7f876c01c4a0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/3206084989 shutdown_connections 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 --2- 172.21.15.70:0/3206084989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f87840cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 --2- 172.21.15.70:0/3206084989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8784071b70 0x7f8784072000 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/3206084989 >> 172.21.15.70:0/3206084989 conn(0x7f87840b1d40 msgr2=0x7f87840cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/3206084989 shutdown_connections 2024-04-22T01:17:26.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/3206084989 wait complete. 2024-04-22T01:17:26.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 Processor -- start 2024-04-22T01:17:26.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- start start 2024-04-22T01:17:26.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784071b70 0x7f87840b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f878414edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 0x7f8784137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87840cf5a0 con 0x7f878415a320 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87840cf420 con 0x7f8784071b70 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8784166f30 con 0x7f87840cd410 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 0x7f8784137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 0x7f8784137b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39684/0 (socket says 172.21.15.70:39684) 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 -- 172.21.15.70:0/4238873198 learned_addr learned my addr 172.21.15.70:0/4238873198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 -- 172.21.15.70:0/4238873198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 msgr2=0x7f878414edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f878414edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 -- 172.21.15.70:0/4238873198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784071b70 msgr2=0x7f87840b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784071b70 0x7f87840b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 -- 172.21.15.70:0/4238873198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878414c910 con 0x7f878415a320 2024-04-22T01:17:26.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87837fe640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 0x7f8784137b40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f877c007e40 tx=0x7f877c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c014620 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c019070 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878414e2f0 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c014a20 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.261+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878414cbf0 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.265+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784068210 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.265+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f877c014bc0 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.265+0000 7f877b7fe640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87640678c0 0x7f8764069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.265+0000 7f8782ffd640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87640678c0 0x7f8764069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.265+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7f877c0866a0 con 0x7f878415a320 2024-04-22T01:17:26.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.265+0000 7f8782ffd640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87640678c0 0x7f8764069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f87740011f0 tx=0x7f8774007930 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:26.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.269+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f877c0570e0 con 0x7f878415a320 2024-04-22T01:17:26.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.429+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f87840727b0 con 0x7f878415a320 2024-04-22T01:17:26.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.429+0000 7f877b7fe640 1 -- 172.21.15.70:0/4238873198 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) ==== 100+0+11 (secure 0 0 0) 0x7f877c0569e0 con 0x7f878415a320 2024-04-22T01:17:26.432 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:17:26.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87640678c0 msgr2=0x7f8764069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87640678c0 0x7f8764069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f87740011f0 tx=0x7f8774007930 comp rx=0 tx=0).stop 2024-04-22T01:17:26.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 msgr2=0x7f8784137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 0x7f8784137b40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f877c007e40 tx=0x7f877c00e4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 shutdown_connections 2024-04-22T01:17:26.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87640678c0 0x7f8764069d80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f878415a320 0x7f8784137b40 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87840cd410 0x7f878414edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 --2- 172.21.15.70:0/4238873198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784071b70 0x7f87840b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 >> 172.21.15.70:0/4238873198 conn(0x7f87840b1d40 msgr2=0x7f8784158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 shutdown_connections 2024-04-22T01:17:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.433+0000 7f87893e9640 1 -- 172.21.15.70:0/4238873198 wait complete. 2024-04-22T01:17:26.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:17:26.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 Processor -- start 2024-04-22T01:17:26.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 -- start start 2024-04-22T01:17:26.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3140cdfe0 0x7fe31415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe31406b1f0 0x7fe31406b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe31415c470 con 0x7fe31415bd00 2024-04-22T01:17:26.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe31406bba0 con 0x7fe31406b1f0 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe31b01e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3140b1f80 con 0x7fe3140cdfe0 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe31406b1f0 0x7fe31406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe31406b1f0 0x7fe31406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39692/0 (socket says 172.21.15.70:39692) 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe318dba640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe31415bd00 0x7fe3140b1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37216/0 (socket says 172.21.15.70:37216) 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 -- 172.21.15.70:0/2979257274 learned_addr learned my addr 172.21.15.70:0/2979257274 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe313fff640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3140cdfe0 0x7fe31415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 -- 172.21.15.70:0/2979257274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3140cdfe0 msgr2=0x7fe31415cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3140cdfe0 0x7fe31415cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 -- 172.21.15.70:0/2979257274 >> v1:172.21.15.70:6789/0 conn(0x7fe31415bd00 legacy=0x7fe3140b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3137fe640 1 -- 172.21.15.70:0/2979257274 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272950858 0 0) 0x7fe31415c470 con 0x7fe31415bd00 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe3195bb640 1 -- 172.21.15.70:0/2979257274 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31415c710 con 0x7fe31406b1f0 2024-04-22T01:17:26.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.565+0000 7fe313fff640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3140cdfe0 0x7fe31415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:26.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe3195bb640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31406b1f0 0x7fe31406b660 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009870 tx=0x7fe30c01cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65babe53f761b0e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:26.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe3137fe640 1 -- 172.21.15.70:0/2979257274 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0aa620 con 0x7fe31406b1f0 2024-04-22T01:17:26.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe3137fe640 1 -- 172.21.15.70:0/2979257274 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c0af070 con 0x7fe31406b1f0 2024-04-22T01:17:26.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe3137fe640 1 -- 172.21.15.70:0/2979257274 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0aa9e0 con 0x7fe31406b1f0 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- 172.21.15.70:0/2979257274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31406b1f0 msgr2=0x7fe31406b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31406b1f0 0x7fe31406b660 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009870 tx=0x7fe30c01cf00 comp rx=0 tx=0).stop 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- 172.21.15.70:0/2979257274 shutdown_connections 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31406b1f0 0x7fe31406b660 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 --2- 172.21.15.70:0/2979257274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3140cdfe0 0x7fe31415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- 172.21.15.70:0/2979257274 >> 172.21.15.70:0/2979257274 conn(0x7fe314099a60 msgr2=0x7fe314099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- 172.21.15.70:0/2979257274 shutdown_connections 2024-04-22T01:17:26.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- 172.21.15.70:0/2979257274 wait complete. 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 Processor -- start 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- start start 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe31406b1f0 0x7fe314157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 0x7fe3140b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31415bd00 0x7fe314155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141602e0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe314160160 con 0x7fe31415bd00 2024-04-22T01:17:26.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe31b01e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe314160460 con 0x7fe31406b1f0 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 0x7fe3140b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 0x7fe3140b5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39702/0 (socket says 172.21.15.70:39702) 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 -- 172.21.15.70:0/1812888268 learned_addr learned my addr 172.21.15.70:0/1812888268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe318dba640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe31406b1f0 0x7fe314157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe3195bb640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31415bd00 0x7fe314155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 -- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe31406b1f0 msgr2=0x7fe314157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe31406b1f0 0x7fe314157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 -- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31415bd00 msgr2=0x7fe314155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31415bd00 0x7fe314155690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe313fff640 1 -- 172.21.15.70:0/1812888268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3140c8650 con 0x7fe3140cdfe0 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.569+0000 7fe318dba640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe31406b1f0 0x7fe314157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:26.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe313fff640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 0x7fe3140b5dc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c7c0 tx=0x7fe30800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308007a30 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe308007bd0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3140b41c0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3080113f0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3140af0f0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3140af2f0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe308011590 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe3117fa640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3000678c0 0x7fe300069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fe308086080 con 0x7fe3140cdfe0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe318dba640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3000678c0 0x7fe300069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:26.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.573+0000 7fe318dba640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3000678c0 0x7fe300069d80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe3040012f0 tx=0x7fe304009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:26.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.577+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe3140af2f0 con 0x7fe3140cdfe0 2024-04-22T01:17:26.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.745+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe31404f510 con 0x7fe3140cdfe0 2024-04-22T01:17:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe3117fa640 1 -- 172.21.15.70:0/1812888268 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) ==== 80+0+0 (secure 0 0 0) 0x7fe308015020 con 0x7fe3140cdfe0 2024-04-22T01:17:26.749 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:17:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3000678c0 msgr2=0x7fe300069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3000678c0 0x7fe300069d80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe3040012f0 tx=0x7fe304009040 comp rx=0 tx=0).stop 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 msgr2=0x7fe3140b5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 0x7fe3140b5dc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c7c0 tx=0x7fe30800cc80 comp rx=0 tx=0).stop 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 shutdown_connections 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3000678c0 0x7fe300069d80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31415bd00 0x7fe314155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3140cdfe0 0x7fe3140b5dc0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 --2- 172.21.15.70:0/1812888268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe31406b1f0 0x7fe314157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 >> 172.21.15.70:0/1812888268 conn(0x7fe314099a60 msgr2=0x7fe3140b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 shutdown_connections 2024-04-22T01:17:26.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:26.749+0000 7fe31b01e640 1 -- 172.21.15.70:0/1812888268 wait complete. 2024-04-22T01:17:26.767 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:17:26.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:26.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:26.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34213]: fuse finished with error 0 and tester_r 0 2024-04-22T01:17:26.782 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:17:26.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:26.782 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:17:26.854 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:17:26.855 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:17:26.855 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:26.861 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:17:26.861 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:17:26.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:26.861 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:17:26.909 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-22T01:17:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:17:26.911 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:17:26.911 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:26.911 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:26.917 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:17:26.917 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:17:26.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:26.917 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:17:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-22T01:17:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:17:26.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 Processor -- start 2024-04-22T01:17:27.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 -- start start 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8b01445a0 0x7ff8b00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff8b00705d0 0x7ff8b00b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8b00715f0 con 0x7ff8b00b2660 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff8b00b3f10 con 0x7ff8b00705d0 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b661c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff8b00d05f0 con 0x7ff8b01445a0 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff8b00705d0 0x7ff8b00b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff8b00705d0 0x7ff8b00b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39712/0 (socket says 172.21.15.70:39712) 2024-04-22T01:17:27.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 -- 172.21.15.70:0/3868300697 learned_addr learned my addr 172.21.15.70:0/3868300697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8b4bb9640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff8b00b2660 0x7ff8b00b2a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37230/0 (socket says 172.21.15.70:37230) 2024-04-22T01:17:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8affff640 1 --2- 172.21.15.70:0/3868300697 >> v2:172.21.15.98:3300/0 conn(0x7ff8b01445a0 0x7ff8b00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 -- 172.21.15.70:0/3868300697 >> v2:172.21.15.98:3300/0 conn(0x7ff8b01445a0 msgr2=0x7ff8b00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 --2- 172.21.15.70:0/3868300697 >> v2:172.21.15.98:3300/0 conn(0x7ff8b01445a0 0x7ff8b00cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 -- 172.21.15.70:0/3868300697 >> v1:172.21.15.70:6789/0 conn(0x7ff8b00b2660 legacy=0x7ff8b00b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.093+0000 7ff8a7fff640 1 -- 172.21.15.70:0/3868300697 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b0071030 con 0x7ff8b00705d0 2024-04-22T01:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8a7fff640 1 --2- 172.21.15.70:0/3868300697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b00b39d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff8b006cf60 tx=0x7ff89801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be88e8e122c18651 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8af7fe640 1 -- 172.21.15.70:0/3868300697 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980b1070 con 0x7ff8b00705d0 2024-04-22T01:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8af7fe640 1 -- 172.21.15.70:0/3868300697 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8980ac600 con 0x7ff8b00705d0 2024-04-22T01:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8af7fe640 1 -- 172.21.15.70:0/3868300697 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980ac9d0 con 0x7ff8b00705d0 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/3868300697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 msgr2=0x7ff8b00b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 --2- 172.21.15.70:0/3868300697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b00b39d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff8b006cf60 tx=0x7ff89801c490 comp rx=0 tx=0).stop 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/3868300697 shutdown_connections 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 --2- 172.21.15.70:0/3868300697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b00b39d0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 --2- 172.21.15.70:0/3868300697 >> v2:172.21.15.98:3300/0 conn(0x7ff8b01445a0 0x7ff8b00cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/3868300697 >> 172.21.15.70:0/3868300697 conn(0x7ff8b006c300 msgr2=0x7ff8b006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/3868300697 shutdown_connections 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/3868300697 wait complete. 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 Processor -- start 2024-04-22T01:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- start start 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b0137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b00b2660 0x7ff8b01426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b01445a0 0x7ff8b00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b0162890 con 0x7ff8b00705d0 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b0162710 con 0x7ff8b00b2660 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b0162a10 con 0x7ff8b01445a0 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b0137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b0137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39718/0 (socket says 172.21.15.70:39718) 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 -- 172.21.15.70:0/2512291235 learned_addr learned my addr 172.21.15.70:0/2512291235 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b4bb9640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b01445a0 0x7ff8b00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8a7fff640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b00b2660 0x7ff8b01426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 -- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b01445a0 msgr2=0x7ff8b00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b01445a0 0x7ff8b00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 -- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b00b2660 msgr2=0x7ff8b01426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b00b2660 0x7ff8b01426e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 -- 172.21.15.70:0/2512291235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b0138730 con 0x7ff8b00705d0 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b4bb9640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b01445a0 0x7ff8b00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8affff640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b0137e70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000d5f0 tx=0x7ff8a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a00125f0 con 0x7ff8b00705d0 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a0019070 con 0x7ff8b00705d0 2024-04-22T01:17:27.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a00129b0 con 0x7ff8b00705d0 2024-04-22T01:17:27.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b0138a10 con 0x7ff8b00705d0 2024-04-22T01:17:27.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.097+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b0138fd0 con 0x7ff8b00705d0 2024-04-22T01:17:27.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.101+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b0068210 con 0x7ff8b00705d0 2024-04-22T01:17:27.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.101+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff8a0012b50 con 0x7ff8b00705d0 2024-04-22T01:17:27.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.101+0000 7ff8ad7fa640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8900678c0 0x7ff890069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.101+0000 7ff8a7fff640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8900678c0 0x7ff890069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.101+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7ff8a00861d0 con 0x7ff8b00705d0 2024-04-22T01:17:27.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.101+0000 7ff8a7fff640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8900678c0 0x7ff890069d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff8980ae000 tx=0x7ff898008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.105+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff8a0056c10 con 0x7ff8b00705d0 2024-04-22T01:17:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8b0138470 con 0x7ff8b00705d0 2024-04-22T01:17:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8ad7fa640 1 -- 172.21.15.70:0/2512291235 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5179 (secure 0 0 0) 0x7ff8a0056510 con 0x7ff8b00705d0 2024-04-22T01:17:27.298 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:27.298 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:11.712206+0000","modified":"2024-04-22T01:17:13.727780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6330},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6330":{"gid":6330,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/4194165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4194165476},{"type":"v1","addr":"172.21.15.70:6837","nonce":4194165476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6330,"qdb_cluster":[6330]},"id":4}]} 2024-04-22T01:17:27.298 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8900678c0 msgr2=0x7ff890069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8900678c0 0x7ff890069d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff8980ae000 tx=0x7ff898008040 comp rx=0 tx=0).stop 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 msgr2=0x7ff8b0137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b0137e70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000d5f0 tx=0x7ff8a000dab0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 shutdown_connections 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8900678c0 0x7ff890069d80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b01445a0 0x7ff8b00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b00b2660 0x7ff8b01426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 --2- 172.21.15.70:0/2512291235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8b00705d0 0x7ff8b0137e70 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 >> 172.21.15.70:0/2512291235 conn(0x7ff8b006c300 msgr2=0x7ff8b006ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:27.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 shutdown_connections 2024-04-22T01:17:27.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.297+0000 7ff8b661c640 1 -- 172.21.15.70:0/2512291235 wait complete. 2024-04-22T01:17:27.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:27.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.429+0000 7fa2b259a640 1 Processor -- start 2024-04-22T01:17:27.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.429+0000 7fa2b259a640 1 -- start start 2024-04-22T01:17:27.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2b259a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2ac0cc150 0x7fa2ac071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2b259a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac0cd7b0 con 0x7fa2ac072060 2024-04-22T01:17:27.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2b259a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac0cc710 con 0x7fa2ac1455b0 2024-04-22T01:17:27.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2b259a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa2ac072490 con 0x7fa2ac0cc150 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2ac0cc150 0x7fa2ac071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2ac0cc150 0x7fa2ac071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39730/0 (socket says 172.21.15.70:39730) 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 -- 172.21.15.70:0/4089454608 learned_addr learned my addr 172.21.15.70:0/4089454608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2b0b37640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa2ac072060 0x7fa2ac06e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37234/0 (socket says 172.21.15.70:37234) 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 -- 172.21.15.70:0/4089454608 >> v1:172.21.15.98:6789/0 conn(0x7fa2ac1455b0 legacy=0x7fa2ac0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 -- 172.21.15.70:0/4089454608 >> v1:172.21.15.70:6789/0 conn(0x7fa2ac072060 legacy=0x7fa2ac06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 -- 172.21.15.70:0/4089454608 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac0cc9f0 con 0x7fa2ac0cc150 2024-04-22T01:17:27.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2ab7fe640 1 -- 172.21.15.70:0/4089454608 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3761451816 0 0) 0x7fa2ac0cd7b0 con 0x7fa2ac072060 2024-04-22T01:17:27.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2a37fe640 1 --2- 172.21.15.70:0/4089454608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac0cc150 0x7fa2ac071b20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa294009870 tx=0x7fa29401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59dee55d09fae58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2ab7fe640 1 -- 172.21.15.70:0/4089454608 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2940b0070 con 0x7fa2ac0cc150 2024-04-22T01:17:27.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2ab7fe640 1 -- 172.21.15.70:0/4089454608 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2940ab600 con 0x7fa2ac0cc150 2024-04-22T01:17:27.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.433+0000 7fa2ab7fe640 1 -- 172.21.15.70:0/4089454608 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2940ab9d0 con 0x7fa2ac0cc150 2024-04-22T01:17:27.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/4089454608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac0cc150 msgr2=0x7fa2ac071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 --2- 172.21.15.70:0/4089454608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac0cc150 0x7fa2ac071b20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa294009870 tx=0x7fa29401c400 comp rx=0 tx=0).stop 2024-04-22T01:17:27.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/4089454608 shutdown_connections 2024-04-22T01:17:27.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 --2- 172.21.15.70:0/4089454608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac0cc150 0x7fa2ac071b20 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/4089454608 >> 172.21.15.70:0/4089454608 conn(0x7fa2ac06b1f0 msgr2=0x7fa2ac06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:27.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/4089454608 shutdown_connections 2024-04-22T01:17:27.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/4089454608 wait complete. 2024-04-22T01:17:27.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 Processor -- start 2024-04-22T01:17:27.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- start start 2024-04-22T01:17:27.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 0x7fa2ac137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2ac0cc150 0x7fa2ac138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2ac1455b0 0x7fa2ac157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac169ee0 con 0x7fa2ac072060 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac169d60 con 0x7fa2ac0cc150 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ac16a060 con 0x7fa2ac1455b0 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 0x7fa2ac137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 0x7fa2ac137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39746/0 (socket says 172.21.15.70:39746) 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 -- 172.21.15.70:0/742435505 learned_addr learned my addr 172.21.15.70:0/742435505 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 -- 172.21.15.70:0/742435505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2ac1455b0 msgr2=0x7fa2ac157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2ac1455b0 0x7fa2ac157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 -- 172.21.15.70:0/742435505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2ac0cc150 msgr2=0x7fa2ac138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2ac0cc150 0x7fa2ac138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 -- 172.21.15.70:0/742435505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac155170 con 0x7fa2ac072060 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2abfff640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 0x7fa2ac137990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00b3e0 tx=0x7fa29c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c015020 con 0x7fa2ac072060 2024-04-22T01:17:27.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa29c011850 con 0x7fa2ac072060 2024-04-22T01:17:27.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c011c40 con 0x7fa2ac072060 2024-04-22T01:17:27.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac155450 con 0x7fa2ac072060 2024-04-22T01:17:27.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac155a10 con 0x7fa2ac072060 2024-04-22T01:17:27.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.437+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac068210 con 0x7fa2ac072060 2024-04-22T01:17:27.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.441+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa29c018020 con 0x7fa2ac072060 2024-04-22T01:17:27.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.441+0000 7fa2a97fa640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa28c0678c0 0x7fa28c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.441+0000 7fa2a37fe640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa28c0678c0 0x7fa28c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.441+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fa29c0863d0 con 0x7fa2ac072060 2024-04-22T01:17:27.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.441+0000 7fa2a37fe640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa28c0678c0 0x7fa28c069d80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa2940ad000 tx=0x7fa294008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.445+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa29c01f020 con 0x7fa2ac072060 2024-04-22T01:17:27.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.593+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2ac15f510 con 0x7fa2ac072060 2024-04-22T01:17:27.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.597+0000 7fa2a97fa640 1 -- 172.21.15.70:0/742435505 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16848 (secure 0 0 0) 0x7fa29c00f2e0 con 0x7fa2ac072060 2024-04-22T01:17:27.599 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:27.599 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":64,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:13.375919+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa28c0678c0 msgr2=0x7fa28c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa28c0678c0 0x7fa28c069d80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa2940ad000 tx=0x7fa294008040 comp rx=0 tx=0).stop 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 msgr2=0x7fa2ac137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 0x7fa2ac137990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00b3e0 tx=0x7fa29c00b8a0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 shutdown_connections 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa28c0678c0 0x7fa28c069d80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2ac1455b0 0x7fa2ac157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2ac0cc150 0x7fa2ac138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 --2- 172.21.15.70:0/742435505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2ac072060 0x7fa2ac137990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 >> 172.21.15.70:0/742435505 conn(0x7fa2ac06b1f0 msgr2=0x7fa2ac0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:27.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 shutdown_connections 2024-04-22T01:17:27.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.601+0000 7fa2b259a640 1 -- 172.21.15.70:0/742435505 wait complete. 2024-04-22T01:17:27.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:09.983208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:10.713378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:27.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 Processor -- start 2024-04-22T01:17:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 -- start start 2024-04-22T01:17:27.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f142815ac40 0x7f142815b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f14280ae260 0x7f14280ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f142806b290 con 0x7f14280aebe0 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f14280af4e0 con 0x7f14280ae260 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142fc90640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f14280af7c0 con 0x7f142815ac40 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142d22b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f14280ae260 0x7f14280ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142e22d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f14280aebe0 0x7f14280b07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37236/0 (socket says 172.21.15.70:37236) 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142e22d640 1 -- 172.21.15.70:0/3891373920 learned_addr learned my addr 172.21.15.70:0/3891373920 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142d22b640 1 -- 172.21.15.70:0/3891373920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f142815ac40 msgr2=0x7f142815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142d22b640 1 --2- 172.21.15.70:0/3891373920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f142815ac40 0x7f142815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142d22b640 1 -- 172.21.15.70:0/3891373920 >> v1:172.21.15.70:6789/0 conn(0x7f14280aebe0 legacy=0x7f14280b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142ca2a640 1 -- 172.21.15.70:0/3891373920 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649696600 0 0) 0x7f142806b290 con 0x7f14280aebe0 2024-04-22T01:17:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.741+0000 7f142d22b640 1 -- 172.21.15.70:0/3891373920 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142806b500 con 0x7f14280ae260 2024-04-22T01:17:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142d22b640 1 --2- 172.21.15.70:0/3891373920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f14280ae6a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f14180097c0 tx=0x7f141801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30be45a374cc473 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142ca2a640 1 -- 172.21.15.70:0/3891373920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14180b5070 con 0x7f14280ae260 2024-04-22T01:17:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142ca2a640 1 -- 172.21.15.70:0/3891373920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14180a8990 con 0x7f14280ae260 2024-04-22T01:17:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142ca2a640 1 -- 172.21.15.70:0/3891373920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14180a8d60 con 0x7f14280ae260 2024-04-22T01:17:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 -- 172.21.15.70:0/3891373920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 msgr2=0x7f14280ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 --2- 172.21.15.70:0/3891373920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f14280ae6a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f14180097c0 tx=0x7f141801ec90 comp rx=0 tx=0).stop 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 -- 172.21.15.70:0/3891373920 shutdown_connections 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 --2- 172.21.15.70:0/3891373920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f14280ae6a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 --2- 172.21.15.70:0/3891373920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f142815ac40 0x7f142815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 -- 172.21.15.70:0/3891373920 >> 172.21.15.70:0/3891373920 conn(0x7f1428099a60 msgr2=0x7f14280b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 -- 172.21.15.70:0/3891373920 shutdown_connections 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 -- 172.21.15.70:0/3891373920 wait complete. 2024-04-22T01:17:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 Processor -- start 2024-04-22T01:17:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.745+0000 7f142fc90640 1 -- start start 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f142815d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14280aebe0 0x7f142815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142815ac40 0x7f142809d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f142806cbb0 con 0x7f14280ae260 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f142806ca30 con 0x7f142815ac40 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f14280b5ff0 con 0x7f14280aebe0 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f142815d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f142815d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39766/0 (socket says 172.21.15.70:39766) 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 -- 172.21.15.70:0/2869324821 learned_addr learned my addr 172.21.15.70:0/2869324821 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142d22b640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14280aebe0 0x7f142815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142e22d640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142815ac40 0x7f142809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 -- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14280aebe0 msgr2=0x7f142815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14280aebe0 0x7f142815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 -- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142815ac40 msgr2=0x7f142809d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142815ac40 0x7f142809d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 -- 172.21.15.70:0/2869324821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142815dbf0 con 0x7f14280ae260 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142d22b640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14280aebe0 0x7f142815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142da2c640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f142815d280 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f142400b580 tx=0x7f142400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424014070 con 0x7f14280ae260 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1424010890 con 0x7f14280ae260 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142815ded0 con 0x7f14280ae260 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424010c80 con 0x7f14280ae260 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1428153350 con 0x7f14280ae260 2024-04-22T01:17:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1428068920 con 0x7f14280ae260 2024-04-22T01:17:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1424017070 con 0x7f14280ae260 2024-04-22T01:17:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f141e7fc640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f140c0678c0 0x7f140c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6246+0+0 (secure 0 0 0) 0x7f1424085cf0 con 0x7f14280ae260 2024-04-22T01:17:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142d22b640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f140c0678c0 0x7f140c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.749+0000 7f142d22b640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f140c0678c0 0x7f140c069d80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f14180b2000 tx=0x7f1418008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:27.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.753+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f142408a070 con 0x7f14280ae260 2024-04-22T01:17:27.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.901+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f14281558a0 con 0x7f14280ae260 2024-04-22T01:17:27.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.901+0000 7f141e7fc640 1 -- 172.21.15.70:0/2869324821 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) ==== 93+0+10 (secure 0 0 0) 0x7f1424010e20 con 0x7f14280ae260 2024-04-22T01:17:27.903 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:27.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f140c0678c0 msgr2=0x7f140c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f140c0678c0 0x7f140c069d80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f14180b2000 tx=0x7f1418008040 comp rx=0 tx=0).stop 2024-04-22T01:17:27.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 msgr2=0x7f142815d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:27.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f142815d280 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f142400b580 tx=0x7f142400ba40 comp rx=0 tx=0).stop 2024-04-22T01:17:27.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 shutdown_connections 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f140c0678c0 0x7f140c069d80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142815ac40 0x7f142809d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14280aebe0 0x7f142815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 --2- 172.21.15.70:0/2869324821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14280ae260 0x7f142815d280 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 >> 172.21.15.70:0/2869324821 conn(0x7f1428099a60 msgr2=0x7f1428066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 shutdown_connections 2024-04-22T01:17:27.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:27.905+0000 7f142fc90640 1 -- 172.21.15.70:0/2869324821 wait complete. 2024-04-22T01:17:27.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:17:28.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.029+0000 7fd02d678640 1 Processor -- start 2024-04-22T01:17:28.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.029+0000 7fd02d678640 1 -- start start 2024-04-22T01:17:28.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd02806d8f0 0x7fd02815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd02815f5c0 0x7fd02815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd02806de60 con 0x7fd028158650 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd02806e140 con 0x7fd02815f5c0 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd02806e420 con 0x7fd02806d8f0 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd02815f5c0 0x7fd02815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd02815f5c0 0x7fd02815f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39770/0 (socket says 172.21.15.70:39770) 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 -- 172.21.15.70:0/2751951000 learned_addr learned my addr 172.21.15.70:0/2751951000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd01e7fc640 1 --2- 172.21.15.70:0/2751951000 >> v2:172.21.15.98:3300/0 conn(0x7fd02806d8f0 0x7fd02815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 -- 172.21.15.70:0/2751951000 >> v2:172.21.15.98:3300/0 conn(0x7fd02806d8f0 msgr2=0x7fd02815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 --2- 172.21.15.70:0/2751951000 >> v2:172.21.15.98:3300/0 conn(0x7fd02806d8f0 0x7fd02815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 -- 172.21.15.70:0/2751951000 >> v1:172.21.15.98:6789/0 conn(0x7fd028158650 legacy=0x7fd02806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 -- 172.21.15.70:0/2751951000 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02806e700 con 0x7fd02815f5c0 2024-04-22T01:17:28.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd01e7fc640 1 --2- 172.21.15.70:0/2751951000 >> v2:172.21.15.98:3300/0 conn(0x7fd02806d8f0 0x7fd02815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0277fe640 1 --2- 172.21.15.70:0/2751951000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd02815f5c0 0x7fd02815f9d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd028072730 tx=0x7fd02001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bd670a4c7be573 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0267fc640 1 -- 172.21.15.70:0/2751951000 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0200a55e0 con 0x7fd02815f5c0 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0267fc640 1 -- 172.21.15.70:0/2751951000 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0200a5780 con 0x7fd02815f5c0 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd0267fc640 1 -- 172.21.15.70:0/2751951000 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0200a5b50 con 0x7fd02815f5c0 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 -- 172.21.15.70:0/2751951000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd02815f5c0 msgr2=0x7fd02815f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.033+0000 7fd02d678640 1 --2- 172.21.15.70:0/2751951000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd02815f5c0 0x7fd02815f9d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd028072730 tx=0x7fd02001c920 comp rx=0 tx=0).stop 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/2751951000 shutdown_connections 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 --2- 172.21.15.70:0/2751951000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd02815f5c0 0x7fd02815f9d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 --2- 172.21.15.70:0/2751951000 >> v2:172.21.15.98:3300/0 conn(0x7fd02806d8f0 0x7fd02815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/2751951000 >> 172.21.15.70:0/2751951000 conn(0x7fd0280b1d40 msgr2=0x7fd0280cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/2751951000 shutdown_connections 2024-04-22T01:17:28.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/2751951000 wait complete. 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 Processor -- start 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- start start 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd02806d8f0 0x7fd028138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 0x7fd02814cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd02815f5c0 0x7fd0280b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd02816add0 con 0x7fd028158650 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd02816ac50 con 0x7fd02806d8f0 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd02816af50 con 0x7fd02815f5c0 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 0x7fd02814cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 0x7fd02814cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39772/0 (socket says 172.21.15.70:39772) 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 -- 172.21.15.70:0/3716664196 learned_addr learned my addr 172.21.15.70:0/3716664196 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd0277fe640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd02815f5c0 0x7fd0280b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 -- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd02815f5c0 msgr2=0x7fd0280b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd02815f5c0 0x7fd0280b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd026ffd640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd02806d8f0 0x7fd028138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 -- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd02806d8f0 msgr2=0x7fd028138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd02806d8f0 0x7fd028138450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 -- 172.21.15.70:0/3716664196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02814d7c0 con 0x7fd028158650 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd0277fe640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd02815f5c0 0x7fd0280b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01e7fc640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 0x7fd02814cde0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c820 tx=0x7fd01000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd010015070 con 0x7fd028158650 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd010007a30 con 0x7fd028158650 2024-04-22T01:17:28.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02814daa0 con 0x7fd028158650 2024-04-22T01:17:28.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd010007e40 con 0x7fd028158650 2024-04-22T01:17:28.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd02814e060 con 0x7fd028158650 2024-04-22T01:17:28.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.037+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd028068210 con 0x7fd028158650 2024-04-22T01:17:28.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.041+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd010018070 con 0x7fd028158650 2024-04-22T01:17:28.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.041+0000 7fd01ffff640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd0080678c0 0x7fd008069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.041+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6246+0+0 (secure 0 0 0) 0x7fd010085d60 con 0x7fd028158650 2024-04-22T01:17:28.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.041+0000 7fd026ffd640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd0080678c0 0x7fd008069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.041+0000 7fd026ffd640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd0080678c0 0x7fd008069d80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd01800a750 tx=0x7fd018008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.045+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd010056710 con 0x7fd028158650 2024-04-22T01:17:28.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.201+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd02806f110 con 0x7fd028158650 2024-04-22T01:17:28.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.201+0000 7fd01ffff640 1 -- 172.21.15.70:0/3716664196 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v65) ==== 104+0+11 (secure 0 0 0) 0x7fd010056010 con 0x7fd028158650 2024-04-22T01:17:28.204 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd0080678c0 msgr2=0x7fd008069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd0080678c0 0x7fd008069d80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd01800a750 tx=0x7fd018008040 comp rx=0 tx=0).stop 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 msgr2=0x7fd02814cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 0x7fd02814cde0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c820 tx=0x7fd01000cce0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 shutdown_connections 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd0080678c0 0x7fd008069d80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd02815f5c0 0x7fd0280b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd028158650 0x7fd02814cde0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 --2- 172.21.15.70:0/3716664196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd02806d8f0 0x7fd028138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 >> 172.21.15.70:0/3716664196 conn(0x7fd0280b1d40 msgr2=0x7fd028155010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 shutdown_connections 2024-04-22T01:17:28.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.205+0000 7fd02d678640 1 -- 172.21.15.70:0/3716664196 wait complete. 2024-04-22T01:17:28.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:17:28.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.345+0000 7f181c87e640 1 Processor -- start 2024-04-22T01:17:28.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.345+0000 7f181c87e640 1 -- start start 2024-04-22T01:17:28.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181c87e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f181815e6e0 0x7f181815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181c87e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18180ce920 0x7f1818154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181c87e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181815f000 con 0x7f18180cedd0 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181c87e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f18180b3800 con 0x7f181815e6e0 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181c87e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f18180b3ae0 con 0x7f18180ce920 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f181815e6e0 0x7f181815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f1816d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f18180cedd0 0x7f181806d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37242/0 (socket says 172.21.15.70:37242) 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f1816d9d640 1 -- 172.21.15.70:0/1055251572 learned_addr learned my addr 172.21.15.70:0/1055251572 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181659c640 1 -- 172.21.15.70:0/1055251572 >> v2:172.21.15.98:3300/0 conn(0x7f18180ce920 msgr2=0x7f1818154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f1815d9b640 1 --2- 172.21.15.70:0/1055251572 >> v2:172.21.15.98:3300/0 conn(0x7f18180ce920 0x7f1818154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181559a640 1 -- 172.21.15.70:0/1055251572 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341608056 0 0) 0x7f181815f000 con 0x7f18180cedd0 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181659c640 1 --2- 172.21.15.70:0/1055251572 >> v2:172.21.15.98:3300/0 conn(0x7f18180ce920 0x7f1818154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181659c640 1 -- 172.21.15.70:0/1055251572 >> v1:172.21.15.70:6789/0 conn(0x7f18180cedd0 legacy=0x7f181806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f181659c640 1 -- 172.21.15.70:0/1055251572 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18180cde40 con 0x7f181815e6e0 2024-04-22T01:17:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.349+0000 7f1815d9b640 1 --2- 172.21.15.70:0/1055251572 >> v2:172.21.15.98:3300/0 conn(0x7f18180ce920 0x7f1818154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 --2- 172.21.15.70:0/1055251572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f181815e6e0 0x7f181815eac0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f18180b26e0 tx=0x7f180801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a94100c3304cf18 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181559a640 1 -- 172.21.15.70:0/1055251572 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080ab650 con 0x7f181815e6e0 2024-04-22T01:17:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181559a640 1 -- 172.21.15.70:0/1055251572 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18080b0070 con 0x7f181815e6e0 2024-04-22T01:17:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181559a640 1 -- 172.21.15.70:0/1055251572 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080aba10 con 0x7f181815e6e0 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/1055251572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f181815e6e0 msgr2=0x7f181815eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 --2- 172.21.15.70:0/1055251572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f181815e6e0 0x7f181815eac0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f18180b26e0 tx=0x7f180801c450 comp rx=0 tx=0).stop 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/1055251572 shutdown_connections 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 --2- 172.21.15.70:0/1055251572 >> v2:172.21.15.98:3300/0 conn(0x7f18180ce920 0x7f1818154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 --2- 172.21.15.70:0/1055251572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f181815e6e0 0x7f181815eac0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/1055251572 >> 172.21.15.70:0/1055251572 conn(0x7f18180b1d40 msgr2=0x7f18180cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/1055251572 shutdown_connections 2024-04-22T01:17:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/1055251572 wait complete. 2024-04-22T01:17:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 Processor -- start 2024-04-22T01:17:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- start start 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 0x7f1818165a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18180cedd0 0x7f181815c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f181815e6e0 0x7f1818157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18180b32c0 con 0x7f18180ce920 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18181555b0 con 0x7f181815e6e0 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f18180b3440 con 0x7f18180cedd0 2024-04-22T01:17:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 0x7f1818165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 0x7f1818165a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39786/0 (socket says 172.21.15.70:39786) 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 -- 172.21.15.70:0/2507033739 learned_addr learned my addr 172.21.15.70:0/2507033739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f1815d9b640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18180cedd0 0x7f181815c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 -- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18180cedd0 msgr2=0x7f181815c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18180cedd0 0x7f181815c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 -- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f181815e6e0 msgr2=0x7f1818157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f181815e6e0 0x7f1818157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 -- 172.21.15.70:0/2507033739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1818165f70 con 0x7f18180ce920 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181659c640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 0x7f1818165a30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f18080ad040 tx=0x7f180801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080bf610 con 0x7f18180ce920 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18080bf7d0 con 0x7f18180ce920 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181815ce50 con 0x7f18180ce920 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080bfbc0 con 0x7f18180ce920 2024-04-22T01:17:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.353+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f181815d3d0 con 0x7f18180ce920 2024-04-22T01:17:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.357+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f181815dcc0 con 0x7f18180ce920 2024-04-22T01:17:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.357+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f18080bd020 con 0x7f18180ce920 2024-04-22T01:17:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.357+0000 7f1806ffd640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17f80678c0 0x7f17f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.357+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6246+0+0 (secure 0 0 0) 0x7f1808120a70 con 0x7f18180ce920 2024-04-22T01:17:28.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.357+0000 7f1815d9b640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17f80678c0 0x7f17f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.357+0000 7f1815d9b640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17f80678c0 0x7f17f8069d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f180c006fd0 tx=0x7f180c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.361+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1808016070 con 0x7f18180ce920 2024-04-22T01:17:28.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.517+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1818145610 con 0x7f18180ce920 2024-04-22T01:17:28.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f1806ffd640 1 -- 172.21.15.70:0/2507033739 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v65) ==== 100+0+11 (secure 0 0 0) 0x7f18080f1390 con 0x7f18180ce920 2024-04-22T01:17:28.522 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:17:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17f80678c0 msgr2=0x7f17f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17f80678c0 0x7f17f8069d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f180c006fd0 tx=0x7f180c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 msgr2=0x7f1818165a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 0x7f1818165a30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f18080ad040 tx=0x7f180801ac90 comp rx=0 tx=0).stop 2024-04-22T01:17:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 shutdown_connections 2024-04-22T01:17:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17f80678c0 0x7f17f8069d80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f181815e6e0 0x7f1818157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18180cedd0 0x7f181815c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 --2- 172.21.15.70:0/2507033739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18180ce920 0x7f1818165a30 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 >> 172.21.15.70:0/2507033739 conn(0x7f18180b1d40 msgr2=0x7f1818152820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 shutdown_connections 2024-04-22T01:17:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.521+0000 7f181c87e640 1 -- 172.21.15.70:0/2507033739 wait complete. 2024-04-22T01:17:28.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:28.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.649+0000 7fdacc533640 1 Processor -- start 2024-04-22T01:17:28.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.649+0000 7fdacc533640 1 -- start start 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacc533640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac40aee40 0x7fdac40b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacc533640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac406dbd0 con 0x7fdac415bb50 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacc533640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac406de70 con 0x7fdac415aa30 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacc533640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdac406e150 con 0x7fdac40aee40 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacaad0640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdac415bb50 0x7fdac40b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37244/0 (socket says 172.21.15.70:37244) 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacaad0640 1 -- 172.21.15.70:0/68319931 learned_addr learned my addr 172.21.15.70:0/68319931 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac9ace640 1 --2- 172.21.15.70:0/68319931 >> v2:172.21.15.98:3300/0 conn(0x7fdac40aee40 0x7fdac40b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259775128 0 0) 0x7fdac406dbd0 con 0x7fdac415bb50 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaac005740 con 0x7fdac415bb50 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 253607131 0 0) 0x7fdaac005740 con 0x7fdac415bb50 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 >> v2:172.21.15.98:3300/0 conn(0x7fdac40aee40 msgr2=0x7fdac40b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 --2- 172.21.15.70:0/68319931 >> v2:172.21.15.98:3300/0 conn(0x7fdac40aee40 0x7fdac40b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 >> v1:172.21.15.98:6789/0 conn(0x7fdac415aa30 legacy=0x7fdac40ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac406e430 con 0x7fdac415bb50 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fdab4001680 con 0x7fdac415bb50 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 >> v1:172.21.15.70:6789/0 conn(0x7fdac415bb50 legacy=0x7fdac40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaac005bd0 0x7fdaac005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaac006a20 0x7fdaac006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac0073e0 0x7fdaac00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdac406e150 con 0x7fdaac005bd0 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac00e280 con 0x7fdaac0073e0 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdaac00e480 con 0x7fdaac006a20 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaac005bd0 0x7fdaac005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacaad0640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac0073e0 0x7fdaac00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 -- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaac006a20 msgr2=0x7fdaac006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaac006a20 0x7fdaac006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 -- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac0073e0 msgr2=0x7fdaac00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac0073e0 0x7fdaac00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 -- 172.21.15.70:0/68319931 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac005740 con 0x7fdaac005bd0 2024-04-22T01:17:28.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacaad0640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac0073e0 0x7fdaac00dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:28.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdacaad0640 1 -- 172.21.15.70:0/68319931 reap_dead start 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.653+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaac005bd0 0x7fdaac005ff0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fdabc001e70 tx=0x7fdabc01c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b63552a5f809f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc001a90 con 0x7fdaac005bd0 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc003b80 con 0x7fdaac005bd0 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdac92cd640 1 -- 172.21.15.70:0/68319931 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc005080 con 0x7fdaac005bd0 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- 172.21.15.70:0/68319931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaac005bd0 msgr2=0x7fdaac005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaac005bd0 0x7fdaac005ff0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fdabc001e70 tx=0x7fdabc01c710 comp rx=0 tx=0).stop 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- 172.21.15.70:0/68319931 shutdown_connections 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 --2- 172.21.15.70:0/68319931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaac005bd0 0x7fdaac005ff0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- 172.21.15.70:0/68319931 >> 172.21.15.70:0/68319931 conn(0x7fdac406b1f0 msgr2=0x7fdac406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- 172.21.15.70:0/68319931 shutdown_connections 2024-04-22T01:17:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- 172.21.15.70:0/68319931 wait complete. 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 Processor -- start 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- start start 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac005bd0 0x7fdac41f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac41503e0 0x7fdac414c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 0x7fdac40cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdac415af90 con 0x7fdac414e150 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdac415ae10 con 0x7fdaac005bd0 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacc533640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdac406e950 con 0x7fdac41503e0 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 0x7fdac40cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 0x7fdac40cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39802/0 (socket says 172.21.15.70:39802) 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 -- 172.21.15.70:0/4218488520 learned_addr learned my addr 172.21.15.70:0/4218488520 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdac9ace640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac41503e0 0x7fdac414c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac005bd0 0x7fdac41f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 -- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac41503e0 msgr2=0x7fdac414c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac41503e0 0x7fdac414c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 -- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac005bd0 msgr2=0x7fdac41f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac005bd0 0x7fdac41f35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.657+0000 7fdacaad0640 1 -- 172.21.15.70:0/4218488520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac414c560 con 0x7fdac414e150 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdacaad0640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 0x7fdac40cec60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdac4072ea0 tx=0x7fdab400d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40074e0 con 0x7fdac414e150 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab40047f0 con 0x7fdac414e150 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4004be0 con 0x7fdac414e150 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac4150b80 con 0x7fdac414e150 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac4151140 con 0x7fdac414e150 2024-04-22T01:17:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac4068210 con 0x7fdac414e150 2024-04-22T01:17:28.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdab4012410 con 0x7fdac414e150 2024-04-22T01:17:28.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdac2ffd640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdab00678f0 0x7fdab0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6246+0+0 (secure 0 0 0) 0x7fdab4085450 con 0x7fdac414e150 2024-04-22T01:17:28.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdab00678f0 0x7fdab0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.661+0000 7fdaca2cf640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdab00678f0 0x7fdab0069db0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdabc005000 tx=0x7fdabc001520 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.665+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdab4055e00 con 0x7fdac414e150 2024-04-22T01:17:28.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdac406f190 con 0x7fdac414e150 2024-04-22T01:17:28.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdac2ffd640 1 -- 172.21.15.70:0/4218488520 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5179 (secure 0 0 0) 0x7fdab4055700 con 0x7fdac414e150 2024-04-22T01:17:28.859 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:28.859 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:11.712206+0000","modified":"2024-04-22T01:17:13.727780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6330},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6330":{"gid":6330,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/4194165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4194165476},{"type":"v1","addr":"172.21.15.70:6837","nonce":4194165476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6330,"qdb_cluster":[6330]},"id":4}]} 2024-04-22T01:17:28.859 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdab00678f0 msgr2=0x7fdab0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdab00678f0 0x7fdab0069db0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdabc005000 tx=0x7fdabc001520 comp rx=0 tx=0).stop 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 msgr2=0x7fdac40cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 0x7fdac40cec60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdac4072ea0 tx=0x7fdab400d900 comp rx=0 tx=0).stop 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 shutdown_connections 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdab00678f0 0x7fdab0069db0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdac414e150 0x7fdac40cec60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac41503e0 0x7fdac414c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 --2- 172.21.15.70:0/4218488520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaac005bd0 0x7fdac41f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 >> 172.21.15.70:0/4218488520 conn(0x7fdac406b1f0 msgr2=0x7fdac41556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 shutdown_connections 2024-04-22T01:17:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.857+0000 7fdacc533640 1 -- 172.21.15.70:0/4218488520 wait complete. 2024-04-22T01:17:28.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:28.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.981+0000 7f0e9c007640 1 Processor -- start 2024-04-22T01:17:28.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.981+0000 7f0e9c007640 1 -- start start 2024-04-22T01:17:28.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e9415ac40 0x7f0e9415e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0e940d0f00 0x7f0e940ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e9415eec0 con 0x7f0e940ce730 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0e940ad9b0 con 0x7f0e940d0f00 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9415f180 con 0x7f0e9415ac40 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9a5a4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0e940d0f00 0x7f0e940ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e995a2640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0e940ce730 0x7f0e940d07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37250/0 (socket says 172.21.15.70:37250) 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e995a2640 1 -- 172.21.15.70:0/2405865732 learned_addr learned my addr 172.21.15.70:0/2405865732 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e99da3640 1 --2- 172.21.15.70:0/2405865732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e9415ac40 0x7f0e9415e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9a5a4640 1 -- 172.21.15.70:0/2405865732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e9415ac40 msgr2=0x7f0e9415e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9a5a4640 1 --2- 172.21.15.70:0/2405865732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e9415ac40 0x7f0e9415e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9a5a4640 1 -- 172.21.15.70:0/2405865732 >> v1:172.21.15.70:6789/0 conn(0x7f0e940ce730 legacy=0x7f0e940d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9a5a4640 1 -- 172.21.15.70:0/2405865732 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9415f460 con 0x7f0e940d0f00 2024-04-22T01:17:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9a5a4640 1 --2- 172.21.15.70:0/2405865732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940ade40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0e940adb30 tx=0x7f0e8c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab195dfb70a73fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e98da1640 1 -- 172.21.15.70:0/2405865732 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0b2070 con 0x7f0e940d0f00 2024-04-22T01:17:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e98da1640 1 -- 172.21.15.70:0/2405865732 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e8c0a5810 con 0x7f0e940d0f00 2024-04-22T01:17:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e98da1640 1 -- 172.21.15.70:0/2405865732 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0a5be0 con 0x7f0e940d0f00 2024-04-22T01:17:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- 172.21.15.70:0/2405865732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 msgr2=0x7f0e940ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2405865732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940ade40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0e940adb30 tx=0x7f0e8c01c950 comp rx=0 tx=0).stop 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- 172.21.15.70:0/2405865732 shutdown_connections 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2405865732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940ade40 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2405865732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e9415ac40 0x7f0e9415e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- 172.21.15.70:0/2405865732 >> 172.21.15.70:0/2405865732 conn(0x7f0e94099a60 msgr2=0x7f0e94099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- 172.21.15.70:0/2405865732 shutdown_connections 2024-04-22T01:17:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.985+0000 7f0e9c007640 1 -- 172.21.15.70:0/2405865732 wait complete. 2024-04-22T01:17:28.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 Processor -- start 2024-04-22T01:17:28.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- start start 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e940ce730 0x7f0e940c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e9415ac40 0x7f0e940c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e940aef20 con 0x7f0e940d0f00 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e940aeda0 con 0x7f0e9415ac40 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0e940af0a0 con 0x7f0e940ce730 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39818/0 (socket says 172.21.15.70:39818) 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 -- 172.21.15.70:0/2784453517 learned_addr learned my addr 172.21.15.70:0/2784453517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9a5a4640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e9415ac40 0x7f0e940c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e99da3640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e940ce730 0x7f0e940c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 -- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e940ce730 msgr2=0x7f0e940c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e940ce730 0x7f0e940c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 -- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e9415ac40 msgr2=0x7f0e940c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e9415ac40 0x7f0e940c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 -- 172.21.15.70:0/2784453517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e940c6b60 con 0x7f0e940d0f00 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e995a2640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940c8f40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400c420 tx=0x7f0e8400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84017070 con 0x7f0e940d0f00 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e84005cf0 con 0x7f0e940d0f00 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9414dc50 con 0x7f0e940d0f00 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e840072f0 con 0x7f0e940d0f00 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9414e4f0 con 0x7f0e940d0f00 2024-04-22T01:17:28.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e94068920 con 0x7f0e940d0f00 2024-04-22T01:17:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.989+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0e8401a070 con 0x7f0e940d0f00 2024-04-22T01:17:28.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.993+0000 7f0e8a7fc640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0e780678c0 0x7f0e78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:28.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.993+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6246+0+0 (secure 0 0 0) 0x7f0e84085280 con 0x7f0e940d0f00 2024-04-22T01:17:28.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.993+0000 7f0e99da3640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0e780678c0 0x7f0e78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:28.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.993+0000 7f0e99da3640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0e780678c0 0x7f0e78069d80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0e9414d210 tx=0x7f0e90008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:28.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:28.997+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0e84055ba0 con 0x7f0e940d0f00 2024-04-22T01:17:29.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.157+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e9406ed90 con 0x7f0e940d0f00 2024-04-22T01:17:29.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e8a7fc640 1 -- 172.21.15.70:0/2784453517 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+17088 (secure 0 0 0) 0x7f0e840554a0 con 0x7f0e940d0f00 2024-04-22T01:17:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:29.162 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":66,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:28.732496+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.1","mappings":[{"from":4,"to":3}]},{"pgid":"10.13","mappings":[{"from":1,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0e780678c0 msgr2=0x7f0e78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0e780678c0 0x7f0e78069d80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0e9414d210 tx=0x7f0e90008040 comp rx=0 tx=0).stop 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 msgr2=0x7f0e940c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940c8f40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400c420 tx=0x7f0e8400c8e0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 shutdown_connections 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0e780678c0 0x7f0e78069d80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e9415ac40 0x7f0e940c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e940d0f00 0x7f0e940c8f40 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 --2- 172.21.15.70:0/2784453517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e940ce730 0x7f0e940c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.161+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 >> 172.21.15.70:0/2784453517 conn(0x7f0e94099a60 msgr2=0x7f0e940cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.165+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 shutdown_connections 2024-04-22T01:17:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.165+0000 7f0e9c007640 1 -- 172.21.15.70:0/2784453517 wait complete. 2024-04-22T01:17:29.181 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:17:29.181 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.293+0000 7fc5177c2640 1 Processor -- start 2024-04-22T01:17:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.293+0000 7fc5177c2640 1 -- start start 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc5177c2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5100aeae0 0x7fc5100aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc5177c2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc510071b20 0x7fc510071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc5177c2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5100cf4a0 con 0x7fc51015a550 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc5177c2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc51006f010 con 0x7fc510071b20 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc5177c2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc51006f260 con 0x7fc5100aeae0 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc510071b20 0x7fc510071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc510071b20 0x7fc510071fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39830/0 (socket says 172.21.15.70:39830) 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 -- 172.21.15.70:0/1939779665 learned_addr learned my addr 172.21.15.70:0/1939779665 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc514d5d640 1 --2- 172.21.15.70:0/1939779665 >> v2:172.21.15.98:3300/0 conn(0x7fc5100aeae0 0x7fc5100aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 -- 172.21.15.70:0/1939779665 >> v2:172.21.15.98:3300/0 conn(0x7fc5100aeae0 msgr2=0x7fc5100aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 --2- 172.21.15.70:0/1939779665 >> v2:172.21.15.98:3300/0 conn(0x7fc5100aeae0 0x7fc5100aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 -- 172.21.15.70:0/1939779665 >> v1:172.21.15.98:6789/0 conn(0x7fc51015a550 legacy=0x7fc5100cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 -- 172.21.15.70:0/1939779665 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51006f540 con 0x7fc510071b20 2024-04-22T01:17:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc514d5d640 1 --2- 172.21.15.70:0/1939779665 >> v2:172.21.15.98:3300/0 conn(0x7fc5100aeae0 0x7fc5100aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc515d5f640 1 --2- 172.21.15.70:0/1939779665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc510071fb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc5100b3e80 tx=0x7fc50801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2018f66410177a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc507fff640 1 -- 172.21.15.70:0/1939779665 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5080a55f0 con 0x7fc510071b20 2024-04-22T01:17:29.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc507fff640 1 -- 172.21.15.70:0/1939779665 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5080a5790 con 0x7fc510071b20 2024-04-22T01:17:29.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.297+0000 7fc507fff640 1 -- 172.21.15.70:0/1939779665 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5080a5b60 con 0x7fc510071b20 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/1939779665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 msgr2=0x7fc510071fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 --2- 172.21.15.70:0/1939779665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc510071fb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc5100b3e80 tx=0x7fc50801c9b0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/1939779665 shutdown_connections 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 --2- 172.21.15.70:0/1939779665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc510071fb0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 --2- 172.21.15.70:0/1939779665 >> v2:172.21.15.98:3300/0 conn(0x7fc5100aeae0 0x7fc5100aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/1939779665 >> 172.21.15.70:0/1939779665 conn(0x7fc51006b1f0 msgr2=0x7fc51006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/1939779665 shutdown_connections 2024-04-22T01:17:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/1939779665 wait complete. 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 Processor -- start 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- start start 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc51014cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5100aeae0 0x7fc51014d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc51015a550 0x7fc51014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100b30d0 con 0x7fc510071b20 2024-04-22T01:17:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100b2f50 con 0x7fc51015a550 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc5100b3250 con 0x7fc5100aeae0 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc51014cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc51014cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39842/0 (socket says 172.21.15.70:39842) 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 -- 172.21.15.70:0/485474520 learned_addr learned my addr 172.21.15.70:0/485474520 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc514d5d640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5100aeae0 0x7fc51014d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc515d5f640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc51015a550 0x7fc51014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 -- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5100aeae0 msgr2=0x7fc51014d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5100aeae0 0x7fc51014d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 -- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc51015a550 msgr2=0x7fc51014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc51015a550 0x7fc51014fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 -- 172.21.15.70:0/485474520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc510155ec0 con 0x7fc510071b20 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc514d5d640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5100aeae0 0x7fc51014d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc51555e640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc51014cae0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00b440 tx=0x7fc50c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c015020 con 0x7fc510071b20 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc50c011890 con 0x7fc510071b20 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5101561a0 con 0x7fc510071b20 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c011c80 con 0x7fc510071b20 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.301+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc51014d020 con 0x7fc510071b20 2024-04-22T01:17:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.305+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc510068210 con 0x7fc510071b20 2024-04-22T01:17:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.305+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc50c018020 con 0x7fc510071b20 2024-04-22T01:17:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.305+0000 7fc505ffb640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4f40678c0 0x7fc4f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.305+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6246+0+0 (secure 0 0 0) 0x7fc50c085cd0 con 0x7fc510071b20 2024-04-22T01:17:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.305+0000 7fc514d5d640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4f40678c0 0x7fc4f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.305+0000 7fc514d5d640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4f40678c0 0x7fc4f4069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc510150d80 tx=0x7fc500008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.309+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc50c01f030 con 0x7fc510071b20 2024-04-22T01:17:29.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.497+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc51015cab0 con 0x7fc510071b20 2024-04-22T01:17:29.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc505ffb640 1 -- 172.21.15.70:0/485474520 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5179 (secure 0 0 0) 0x7fc50c00d040 con 0x7fc510071b20 2024-04-22T01:17:29.501 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:29.501 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:11.712206+0000","modified":"2024-04-22T01:17:13.727780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6330},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6330":{"gid":6330,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/4194165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4194165476},{"type":"v1","addr":"172.21.15.70:6837","nonce":4194165476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6330,"qdb_cluster":[6330]},"id":4}]} 2024-04-22T01:17:29.501 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-22T01:17:29.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4f40678c0 msgr2=0x7fc4f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4f40678c0 0x7fc4f4069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc510150d80 tx=0x7fc500008040 comp rx=0 tx=0).stop 2024-04-22T01:17:29.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 msgr2=0x7fc51014cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc51014cae0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00b440 tx=0x7fc50c00b900 comp rx=0 tx=0).stop 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 shutdown_connections 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc4f40678c0 0x7fc4f4069d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc51015a550 0x7fc51014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5100aeae0 0x7fc51014d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 --2- 172.21.15.70:0/485474520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc510071b20 0x7fc51014cae0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 >> 172.21.15.70:0/485474520 conn(0x7fc51006b1f0 msgr2=0x7fc5100b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 shutdown_connections 2024-04-22T01:17:29.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.501+0000 7fc5177c2640 1 -- 172.21.15.70:0/485474520 wait complete. 2024-04-22T01:17:29.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:29.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31d07f640 1 Processor -- start 2024-04-22T01:17:29.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31d07f640 1 -- start start 2024-04-22T01:17:29.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31d07f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3181455b0 0x7fe3180cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31d07f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3180d0250 con 0x7fe3180cc150 2024-04-22T01:17:29.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31d07f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3180cd7b0 con 0x7fe318072230 2024-04-22T01:17:29.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31d07f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe3180cc710 con 0x7fe3181455b0 2024-04-22T01:17:29.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe3180cc150 0x7fe318071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37256/0 (socket says 172.21.15.70:37256) 2024-04-22T01:17:29.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31659c640 1 -- 172.21.15.70:0/1954828068 learned_addr learned my addr 172.21.15.70:0/1954828068 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224309647 0 0) 0x7fe3180d0250 con 0x7fe3180cc150 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe300003530 con 0x7fe3180cc150 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1954828068 >> v2:172.21.15.98:3300/0 conn(0x7fe3181455b0 0x7fe3180cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1878506891 0 0) 0x7fe300003530 con 0x7fe3180cc150 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 >> v2:172.21.15.98:3300/0 conn(0x7fe3181455b0 msgr2=0x7fe3180cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 --2- 172.21.15.70:0/1954828068 >> v2:172.21.15.98:3300/0 conn(0x7fe3181455b0 0x7fe3180cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 >> v1:172.21.15.98:6789/0 conn(0x7fe318072230 legacy=0x7fe31806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3180cc9f0 con 0x7fe3180cc150 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fe30c001680 con 0x7fe3180cc150 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 >> v1:172.21.15.70:6789/0 conn(0x7fe3180cc150 legacy=0x7fe318071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3000039c0 0x7fe300003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe300004810 0x7fe300004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3000053d0 0x7fe30000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3180cc710 con 0x7fe3000039c0 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe30000c260 con 0x7fe3000053d0 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe30000c460 con 0x7fe300004810 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3000039c0 0x7fe300003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31759e640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3000053d0 0x7fe30000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31659c640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe300004810 0x7fe300004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 -- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe300004810 msgr2=0x7fe300004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe300004810 0x7fe300004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 -- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3000053d0 msgr2=0x7fe30000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3000053d0 0x7fe30000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe316d9d640 1 -- 172.21.15.70:0/1954828068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe300003530 con 0x7fe3000039c0 2024-04-22T01:17:29.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.625+0000 7fe31759e640 1 -- 172.21.15.70:0/1954828068 reap_dead start 2024-04-22T01:17:29.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3000039c0 0x7fe300003de0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe30800ba30 tx=0x7fe30801c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af7bbce50b7b4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3080b3070 con 0x7fe3000039c0 2024-04-22T01:17:29.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3080ae630 con 0x7fe3000039c0 2024-04-22T01:17:29.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe315d9b640 1 -- 172.21.15.70:0/1954828068 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3080aa050 con 0x7fe3000039c0 2024-04-22T01:17:29.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 -- 172.21.15.70:0/1954828068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3000039c0 msgr2=0x7fe300003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3000039c0 0x7fe300003de0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe30800ba30 tx=0x7fe30801c6a0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 -- 172.21.15.70:0/1954828068 shutdown_connections 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1954828068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3000039c0 0x7fe300003de0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 -- 172.21.15.70:0/1954828068 >> 172.21.15.70:0/1954828068 conn(0x7fe31806b1f0 msgr2=0x7fe31806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 -- 172.21.15.70:0/1954828068 shutdown_connections 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 -- 172.21.15.70:0/1954828068 wait complete. 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 Processor -- start 2024-04-22T01:17:29.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.629+0000 7fe31d07f640 1 -- start start 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3000039c0 0x7fe31815ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 0x7fe31813eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31815e4c0 0x7fe31815e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe318160130 con 0x7fe31813ea70 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe31815ffb0 con 0x7fe31815e4c0 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3181602b0 con 0x7fe3000039c0 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 0x7fe31813eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 0x7fe31813eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39872/0 (socket says 172.21.15.70:39872) 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 -- 172.21.15.70:0/1767281319 learned_addr learned my addr 172.21.15.70:0/1767281319 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:29.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3000039c0 0x7fe31815ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31759e640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31815e4c0 0x7fe31815e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 -- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3000039c0 msgr2=0x7fe31815ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3000039c0 0x7fe31815ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 -- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31815e4c0 msgr2=0x7fe31815e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31815e4c0 0x7fe31815e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 -- 172.21.15.70:0/1767281319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3181f98f0 con 0x7fe31813ea70 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3000039c0 0x7fe31815ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31659c640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 0x7fe31813eef0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00e7b0 tx=0x7fe30c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c00c6c0 con 0x7fe31813ea70 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c018070 con 0x7fe31813ea70 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31813f5e0 con 0x7fe31813ea70 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c00caa0 con 0x7fe31813ea70 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3181f8e10 con 0x7fe31813ea70 2024-04-22T01:17:29.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe318068210 con 0x7fe31813ea70 2024-04-22T01:17:29.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe30c00cc40 con 0x7fe31813ea70 2024-04-22T01:17:29.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe2ff7fe640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3040678c0 0x7fe304069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3040678c0 0x7fe304069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6246+0+0 (secure 0 0 0) 0x7fe30c085df0 con 0x7fe31813ea70 2024-04-22T01:17:29.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.633+0000 7fe316d9d640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3040678c0 0x7fe304069d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe3080b0000 tx=0x7fe3080046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.637+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe30c056710 con 0x7fe31813ea70 2024-04-22T01:17:29.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.817+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe31815f510 con 0x7fe31813ea70 2024-04-22T01:17:29.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe2ff7fe640 1 -- 172.21.15.70:0/1767281319 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5179 (secure 0 0 0) 0x7fe30c056010 con 0x7fe31813ea70 2024-04-22T01:17:29.821 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:29.821 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:11.712206+0000","modified":"2024-04-22T01:17:13.727780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6330},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6330":{"gid":6330,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/4194165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4194165476},{"type":"v1","addr":"172.21.15.70:6837","nonce":4194165476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6330,"qdb_cluster":[6330]},"id":4}]} 2024-04-22T01:17:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3040678c0 msgr2=0x7fe304069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3040678c0 0x7fe304069d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe3080b0000 tx=0x7fe3080046d0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 msgr2=0x7fe31813eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 0x7fe31813eef0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00e7b0 tx=0x7fe30c00ec70 comp rx=0 tx=0).stop 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 shutdown_connections 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe3040678c0 0x7fe304069d80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe31815e4c0 0x7fe31815e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe31813ea70 0x7fe31813eef0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 --2- 172.21.15.70:0/1767281319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3000039c0 0x7fe31815ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 >> 172.21.15.70:0/1767281319 conn(0x7fe31806b1f0 msgr2=0x7fe3180cf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 shutdown_connections 2024-04-22T01:17:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.821+0000 7fe31d07f640 1 -- 172.21.15.70:0/1767281319 wait complete. 2024-04-22T01:17:29.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:29.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.941+0000 7fb0e63d0640 1 Processor -- start 2024-04-22T01:17:29.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.941+0000 7fb0e63d0640 1 -- start start 2024-04-22T01:17:29.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.941+0000 7fb0e63d0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb0e00aee40 0x7fb0e00b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.941+0000 7fb0e63d0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e006dbd0 con 0x7fb0e015bb50 2024-04-22T01:17:29.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.941+0000 7fb0e63d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e006de70 con 0x7fb0e015aa30 2024-04-22T01:17:29.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.941+0000 7fb0e63d0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb0e006e150 con 0x7fb0e00aee40 2024-04-22T01:17:29.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e496d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb0e015bb50 0x7fb0e00b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37270/0 (socket says 172.21.15.70:37270) 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e496d640 1 -- 172.21.15.70:0/4245572989 learned_addr learned my addr 172.21.15.70:0/4245572989 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0df7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb0e00aee40 0x7fb0e00b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0deffd640 1 -- 172.21.15.70:0/4245572989 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149195955 0 0) 0x7fb0e006dbd0 con 0x7fb0e015bb50 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0deffd640 1 -- 172.21.15.70:0/4245572989 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c8003530 con 0x7fb0e015bb50 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0df7fe640 1 -- 172.21.15.70:0/4245572989 >> v1:172.21.15.98:6789/0 conn(0x7fb0e015aa30 legacy=0x7fb0e00ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0df7fe640 1 -- 172.21.15.70:0/4245572989 >> v1:172.21.15.70:6789/0 conn(0x7fb0e015bb50 legacy=0x7fb0e00b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0df7fe640 1 -- 172.21.15.70:0/4245572989 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e006e430 con 0x7fb0e00aee40 2024-04-22T01:17:29.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0df7fe640 1 --2- 172.21.15.70:0/4245572989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e00b4720 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40097c0 tx=0x7fb0d401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce43f920e8254a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0deffd640 1 -- 172.21.15.70:0/4245572989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d40ab680 con 0x7fb0e00aee40 2024-04-22T01:17:29.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0deffd640 1 -- 172.21.15.70:0/4245572989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0d40b0070 con 0x7fb0e00aee40 2024-04-22T01:17:29.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0deffd640 1 -- 172.21.15.70:0/4245572989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d40aba40 con 0x7fb0e00aee40 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e63d0640 1 -- 172.21.15.70:0/4245572989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 msgr2=0x7fb0e00b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/4245572989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e00b4720 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40097c0 tx=0x7fb0d401c460 comp rx=0 tx=0).stop 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e63d0640 1 -- 172.21.15.70:0/4245572989 shutdown_connections 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/4245572989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e00b4720 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.945+0000 7fb0e63d0640 1 -- 172.21.15.70:0/4245572989 >> 172.21.15.70:0/4245572989 conn(0x7fb0e006b1f0 msgr2=0x7fb0e006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- 172.21.15.70:0/4245572989 shutdown_connections 2024-04-22T01:17:29.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- 172.21.15.70:0/4245572989 wait complete. 2024-04-22T01:17:29.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 Processor -- start 2024-04-22T01:17:29.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- start start 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e014adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e015aa30 0x7fb0e014c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e015bb50 0x7fb0e014c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e006ead0 con 0x7fb0e00aee40 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e006e950 con 0x7fb0e015bb50 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e0168aa0 con 0x7fb0e015aa30 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e014adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e014adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39896/0 (socket says 172.21.15.70:39896) 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 -- 172.21.15.70:0/2075839313 learned_addr learned my addr 172.21.15.70:0/2075839313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 -- 172.21.15.70:0/2075839313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e015aa30 msgr2=0x7fb0e014c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e015aa30 0x7fb0e014c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 -- 172.21.15.70:0/2075839313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e015bb50 msgr2=0x7fb0e014c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:29.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e015bb50 0x7fb0e014c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 -- 172.21.15.70:0/2075839313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e0153760 con 0x7fb0e00aee40 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dffff640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e014adf0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c7f0 tx=0x7fb0d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0016020 con 0x7fb0e00aee40 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0d000aa50 con 0x7fb0e00aee40 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000ae40 con 0x7fb0e00aee40 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e0153a40 con 0x7fb0e00aee40 2024-04-22T01:17:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e0154000 con 0x7fb0e00aee40 2024-04-22T01:17:29.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.949+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e0068210 con 0x7fb0e00aee40 2024-04-22T01:17:29.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.953+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0d0019020 con 0x7fb0e00aee40 2024-04-22T01:17:29.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.953+0000 7fb0dcff9640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0cc0678c0 0x7fb0cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:29.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.953+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6246+0+0 (secure 0 0 0) 0x7fb0d0085f30 con 0x7fb0e00aee40 2024-04-22T01:17:29.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.953+0000 7fb0df7fe640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0cc0678c0 0x7fb0cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:29.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.953+0000 7fb0df7fe640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0cc0678c0 0x7fb0cc069d80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40af000 tx=0x7fb0d4004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:29.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:29.957+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb0d0020030 con 0x7fb0e00aee40 2024-04-22T01:17:30.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.105+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0e006f190 con 0x7fb0e00aee40 2024-04-22T01:17:30.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.109+0000 7fb0dcff9640 1 -- 172.21.15.70:0/2075839313 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+17088 (secure 0 0 0) 0x7fb0d0056850 con 0x7fb0e00aee40 2024-04-22T01:17:30.111 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:30.111 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":66,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:28.732496+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:09.983208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:10.713378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.1","mappings":[{"from":4,"to":3}]},{"pgid":"10.13","mappings":[{"from":1,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:30.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0cc0678c0 msgr2=0x7fb0cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0cc0678c0 0x7fb0cc069d80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40af000 tx=0x7fb0d4004740 comp rx=0 tx=0).stop 2024-04-22T01:17:30.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 msgr2=0x7fb0e014adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e014adf0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c7f0 tx=0x7fb0d000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 shutdown_connections 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0cc0678c0 0x7fb0cc069d80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e015bb50 0x7fb0e014c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e015aa30 0x7fb0e014c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 --2- 172.21.15.70:0/2075839313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e00aee40 0x7fb0e014adf0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 >> 172.21.15.70:0/2075839313 conn(0x7fb0e006b1f0 msgr2=0x7fb0e015d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 shutdown_connections 2024-04-22T01:17:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.113+0000 7fb0e63d0640 1 -- 172.21.15.70:0/2075839313 wait complete. 2024-04-22T01:17:30.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:17:30.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 Processor -- start 2024-04-22T01:17:30.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 -- start start 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01441445b0 0x7f0144072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f01440b47f0 0x7f01440ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0144144990 con 0x7f01440b4360 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f01440b3bd0 con 0x7f01440b47f0 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014b46f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f01440b2b50 con 0x7f01441445b0 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f01440b47f0 0x7f01440ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f01440b47f0 0x7f01440ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39912/0 (socket says 172.21.15.70:39912) 2024-04-22T01:17:30.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 -- 172.21.15.70:0/835076324 learned_addr learned my addr 172.21.15.70:0/835076324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:30.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f014920b640 1 --2- 172.21.15.70:0/835076324 >> v2:172.21.15.98:3300/0 conn(0x7f01441445b0 0x7f0144072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 -- 172.21.15.70:0/835076324 >> v2:172.21.15.98:3300/0 conn(0x7f01441445b0 msgr2=0x7f0144072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:30.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 --2- 172.21.15.70:0/835076324 >> v2:172.21.15.98:3300/0 conn(0x7f01441445b0 0x7f0144072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 -- 172.21.15.70:0/835076324 >> v1:172.21.15.98:6789/0 conn(0x7f01440b4360 legacy=0x7f01440cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.245+0000 7f0149a0c640 1 -- 172.21.15.70:0/835076324 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01440b2e10 con 0x7f01440b47f0 2024-04-22T01:17:30.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f0149a0c640 1 --2- 172.21.15.70:0/835076324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01440b47f0 0x7f01440ce3f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0144073480 tx=0x7f013c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e90d138d0ef0342 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:30.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f013bfff640 1 -- 172.21.15.70:0/835076324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c0a5880 con 0x7f01440b47f0 2024-04-22T01:17:30.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f013bfff640 1 -- 172.21.15.70:0/835076324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013c0a5a20 con 0x7f01440b47f0 2024-04-22T01:17:30.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f013bfff640 1 -- 172.21.15.70:0/835076324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c0a5df0 con 0x7f01440b47f0 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 -- 172.21.15.70:0/835076324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01440b47f0 msgr2=0x7f01440ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 --2- 172.21.15.70:0/835076324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01440b47f0 0x7f01440ce3f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0144073480 tx=0x7f013c01c9c0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 -- 172.21.15.70:0/835076324 shutdown_connections 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 --2- 172.21.15.70:0/835076324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01440b47f0 0x7f01440ce3f0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 --2- 172.21.15.70:0/835076324 >> v2:172.21.15.98:3300/0 conn(0x7f01441445b0 0x7f0144072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 -- 172.21.15.70:0/835076324 >> 172.21.15.70:0/835076324 conn(0x7f01440cf9e0 msgr2=0x7f01440cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 -- 172.21.15.70:0/835076324 shutdown_connections 2024-04-22T01:17:30.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 -- 172.21.15.70:0/835076324 wait complete. 2024-04-22T01:17:30.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 Processor -- start 2024-04-22T01:17:30.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.249+0000 7f014b46f640 1 -- start start 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440b4360 0x7f01441370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01440b47f0 0x7f014415c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 0x7f0144153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0144072a40 con 0x7f01441445b0 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01440b4c20 con 0x7f01440b47f0 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f014406edb0 con 0x7f01440b4360 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 0x7f0144153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 0x7f0144153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39928/0 (socket says 172.21.15.70:39928) 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 -- 172.21.15.70:0/162000475 learned_addr learned my addr 172.21.15.70:0/162000475 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014920b640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440b4360 0x7f01441370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 -- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440b4360 msgr2=0x7f01441370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440b4360 0x7f01441370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 -- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01440b47f0 msgr2=0x7f014415c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0148a0a640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01440b47f0 0x7f014415c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01440b47f0 0x7f014415c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 -- 172.21.15.70:0/162000475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014415b300 con 0x7f01441445b0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014920b640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440b4360 0x7f01441370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0149a0c640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 0x7f0144153c80 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f013c0af040 tx=0x7f013c019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c0bc020 con 0x7f01441445b0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013c0b8670 con 0x7f01441445b0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01440ce460 con 0x7f01441445b0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c0b8a60 con 0x7f01441445b0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0144137620 con 0x7f01441445b0 2024-04-22T01:17:30.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01440ce250 con 0x7f01441445b0 2024-04-22T01:17:30.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f013c0bf020 con 0x7f01441445b0 2024-04-22T01:17:30.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0139ffb640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01280678c0 0x7f0128069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6246+0+0 (secure 0 0 0) 0x7f013c121000 con 0x7f01441445b0 2024-04-22T01:17:30.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014920b640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01280678c0 0x7f0128069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.253+0000 7f014920b640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01280678c0 0x7f0128069d80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f01440cee80 tx=0x7f0140008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:30.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.257+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f013c0c6030 con 0x7f01441445b0 2024-04-22T01:17:30.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.453+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f014404f510 con 0x7f01441445b0 2024-04-22T01:17:30.457 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:17:30.457+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:17:30.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.457+0000 7f0139ffb640 1 -- 172.21.15.70:0/162000475 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7f013c0165e0 con 0x7f01441445b0 2024-04-22T01:17:30.460 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01280678c0 msgr2=0x7f0128069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01280678c0 0x7f0128069d80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f01440cee80 tx=0x7f0140008040 comp rx=0 tx=0).stop 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 msgr2=0x7f0144153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 0x7f0144153c80 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f013c0af040 tx=0x7f013c019760 comp rx=0 tx=0).stop 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 shutdown_connections 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01280678c0 0x7f0128069d80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01441445b0 0x7f0144153c80 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01440b47f0 0x7f014415c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 --2- 172.21.15.70:0/162000475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440b4360 0x7f01441370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 >> 172.21.15.70:0/162000475 conn(0x7f01440cf9e0 msgr2=0x7f0144153320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 shutdown_connections 2024-04-22T01:17:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.461+0000 7f014b46f640 1 -- 172.21.15.70:0/162000475 wait complete. 2024-04-22T01:17:30.472 INFO:tasks.ceph.mds.c.smithi070.stderr: -3167> 2024-04-22T01:17:18.014+0000 7f8d824cf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:17:30.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:17:30.512 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:17:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.609+0000 7ff104edf640 1 Processor -- start 2024-04-22T01:17:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.609+0000 7ff104edf640 1 -- start start 2024-04-22T01:17:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147c00 0x7ff100161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff100162000 0x7ff1000b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1001624a0 con 0x7ff100147820 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff100162a60 con 0x7ff100162000 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff100162780 con 0x7ff100147c00 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff100162000 0x7ff1000b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff100162000 0x7ff1000b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39954/0 (socket says 172.21.15.70:39954) 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 -- 172.21.15.70:0/671761230 learned_addr learned my addr 172.21.15.70:0/671761230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fdd9b640 1 --2- 172.21.15.70:0/671761230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147c00 0x7ff100161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 -- 172.21.15.70:0/671761230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147c00 msgr2=0x7ff100161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 --2- 172.21.15.70:0/671761230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147c00 0x7ff100161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 -- 172.21.15.70:0/671761230 >> v1:172.21.15.98:6789/0 conn(0x7ff100147820 legacy=0x7ff100072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 -- 172.21.15.70:0/671761230 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff100072070 con 0x7ff100162000 2024-04-22T01:17:30.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fed9d640 1 --2- 172.21.15.70:0/671761230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1000b3870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009e50 tx=0x7ff0f801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b077f9f268ea59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fd59a640 1 -- 172.21.15.70:0/671761230 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80a5770 con 0x7ff100162000 2024-04-22T01:17:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fd59a640 1 -- 172.21.15.70:0/671761230 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f80a5910 con 0x7ff100162000 2024-04-22T01:17:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff0fd59a640 1 -- 172.21.15.70:0/671761230 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80a5ce0 con 0x7ff100162000 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- 172.21.15.70:0/671761230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 msgr2=0x7ff1000b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 --2- 172.21.15.70:0/671761230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1000b3870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009e50 tx=0x7ff0f801c9a0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- 172.21.15.70:0/671761230 shutdown_connections 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 --2- 172.21.15.70:0/671761230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1000b3870 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 --2- 172.21.15.70:0/671761230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147c00 0x7ff100161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- 172.21.15.70:0/671761230 >> 172.21.15.70:0/671761230 conn(0x7ff10006b1f0 msgr2=0x7ff10006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- 172.21.15.70:0/671761230 shutdown_connections 2024-04-22T01:17:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.613+0000 7ff104edf640 1 -- 172.21.15.70:0/671761230 wait complete. 2024-04-22T01:17:30.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 Processor -- start 2024-04-22T01:17:30.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 -- start start 2024-04-22T01:17:30.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147820 0x7ff100143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff100147c00 0x7ff10013b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1001609e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff10016ae60 con 0x7ff100162000 2024-04-22T01:17:30.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff10006ecb0 con 0x7ff100147c00 2024-04-22T01:17:30.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff10016afe0 con 0x7ff100147820 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1001609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1001609e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39970/0 (socket says 172.21.15.70:39970) 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 -- 172.21.15.70:0/1633781861 learned_addr learned my addr 172.21.15.70:0/1633781861 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fdd9b640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff100147c00 0x7ff10013b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fe59c640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147820 0x7ff100143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 -- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147820 msgr2=0x7ff100143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147820 0x7ff100143ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 -- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff100147c00 msgr2=0x7ff10013b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff100147c00 0x7ff10013b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 -- 172.21.15.70:0/1633781861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10015eab0 con 0x7ff100162000 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0fed9d640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1001609e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80af040 tx=0x7ff0f8019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80bc020 con 0x7ff100162000 2024-04-22T01:17:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f80b8670 con 0x7ff100162000 2024-04-22T01:17:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10015edf0 con 0x7ff100162000 2024-04-22T01:17:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80b8a40 con 0x7ff100162000 2024-04-22T01:17:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.617+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10015f630 con 0x7ff100162000 2024-04-22T01:17:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.621+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff100068210 con 0x7ff100162000 2024-04-22T01:17:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.621+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff0f80bf020 con 0x7ff100162000 2024-04-22T01:17:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.621+0000 7ff0eeffd640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0e00678c0 0x7ff0e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.621+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6332+0+0 (secure 0 0 0) 0x7ff0f8120e10 con 0x7ff100162000 2024-04-22T01:17:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.621+0000 7ff0fe59c640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0e00678c0 0x7ff0e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.621+0000 7ff0fe59c640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0e00678c0 0x7ff0e0069d80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0001650 tx=0x7ff0f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:30.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.625+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff0f80c6030 con 0x7ff100162000 2024-04-22T01:17:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:30.817+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff10006e070 con 0x7ff100162000 2024-04-22T01:17:31.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.465+0000 7ff0eeffd640 1 -- 172.21.15.70:0/1633781861 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7ff10015edf0 con 0x7ff100162000 2024-04-22T01:17:31.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0e00678c0 msgr2=0x7ff0e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0e00678c0 0x7ff0e0069d80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0001650 tx=0x7ff0f0009040 comp rx=0 tx=0).stop 2024-04-22T01:17:31.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 msgr2=0x7ff1001609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1001609e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80af040 tx=0x7ff0f8019740 comp rx=0 tx=0).stop 2024-04-22T01:17:31.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 shutdown_connections 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0e00678c0 0x7ff0e0069d80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff100162000 0x7ff1001609e0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff100147c00 0x7ff10013b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 --2- 172.21.15.70:0/1633781861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff100147820 0x7ff100143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 >> 172.21.15.70:0/1633781861 conn(0x7ff10006b1f0 msgr2=0x7ff1000ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 shutdown_connections 2024-04-22T01:17:31.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.469+0000 7ff104edf640 1 -- 172.21.15.70:0/1633781861 wait complete. 2024-04-22T01:17:31.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:17:31.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dfb9d640 1 Processor -- start 2024-04-22T01:17:31.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dfb9d640 1 -- start start 2024-04-22T01:17:31.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dfb9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 0x7f91d806f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dfb9d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d80d1a70 con 0x7f91d8161970 2024-04-22T01:17:31.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dfb9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d80d0cb0 con 0x7f91d806b1f0 2024-04-22T01:17:31.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dfb9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f91d80cd570 con 0x7f91d806efb0 2024-04-22T01:17:31.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f91d8161970 0x7f91d8161e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37300/0 (socket says 172.21.15.70:37300) 2024-04-22T01:17:31.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 -- 172.21.15.70:0/1920906315 learned_addr learned my addr 172.21.15.70:0/1920906315 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:31.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dd138640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 0x7f91d806f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:31.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480233294 0 0) 0x7f91d80d1a70 con 0x7f91d8161970 2024-04-22T01:17:31.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91c0003530 con 0x7f91d8161970 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2642521131 0 0) 0x7f91c0003530 con 0x7f91d8161970 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 msgr2=0x7f91d806f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 0x7f91d806f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 >> v1:172.21.15.98:6789/0 conn(0x7f91d806b1f0 legacy=0x7f91d806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d80d0f90 con 0x7f91d8161970 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f91c8001680 con 0x7f91d8161970 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 >> v1:172.21.15.70:6789/0 conn(0x7f91d8161970 legacy=0x7f91d8161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 0x7f91c00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91c0003fc0 0x7f91c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91c0004d50 0x7f91c00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91c000bae0 con 0x7f91c0004d50 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91c000bc90 con 0x7f91c0003fc0 2024-04-22T01:17:31.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f91c000be90 con 0x7f91d806efb0 2024-04-22T01:17:31.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91c0004d50 0x7f91c00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:31.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 -- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 msgr2=0x7f91c00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:31.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d806efb0 0x7f91c00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 -- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91c0003fc0 msgr2=0x7f91c0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:31.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91c0003fc0 0x7f91c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.661+0000 7f91de13a640 1 -- 172.21.15.70:0/1920906315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c0003530 con 0x7f91c0004d50 2024-04-22T01:17:31.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91de13a640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91c0004d50 0x7f91c00051d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f91c800c190 tx=0x7f91c801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23e3ed0c0258d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:31.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8007c70 con 0x7f91c0004d50 2024-04-22T01:17:31.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c80a9070 con 0x7f91c0004d50 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dc937640 1 -- 172.21.15.70:0/1920906315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8005dc0 con 0x7f91c0004d50 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1920906315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91c0004d50 msgr2=0x7f91c00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1920906315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91c0004d50 0x7f91c00051d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f91c800c190 tx=0x7f91c801cbe0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91de13a640 1 -- 172.21.15.70:0/1920906315 reap_dead start 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1920906315 shutdown_connections 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1920906315 >> 172.21.15.70:0/1920906315 conn(0x7f91d80733e0 msgr2=0x7f91d8099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1920906315 shutdown_connections 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1920906315 wait complete. 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 Processor -- start 2024-04-22T01:17:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.665+0000 7f91dfb9d640 1 -- start start 2024-04-22T01:17:31.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 0x7f91d815f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d81f3c10 0x7f91d815fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91d8159690 0x7f91d8159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91d80d15e0 con 0x7f91d8155ae0 2024-04-22T01:17:31.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91d806fd20 con 0x7f91d8159690 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f91d815e0b0 con 0x7f91d81f3c10 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 0x7f91d815f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 0x7f91d815f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39980/0 (socket says 172.21.15.70:39980) 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 -- 172.21.15.70:0/1589976372 learned_addr learned my addr 172.21.15.70:0/1589976372 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd138640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d81f3c10 0x7f91d815fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91de13a640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91d8159690 0x7f91d8159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 -- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d81f3c10 msgr2=0x7f91d815fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d81f3c10 0x7f91d815fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 -- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91d8159690 msgr2=0x7f91d8159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91d8159690 0x7f91d8159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 -- 172.21.15.70:0/1589976372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d815a430 con 0x7f91d8155ae0 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd939640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 0x7f91d815f8a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f91d000c790 tx=0x7f91d000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0003b80 con 0x7f91d8155ae0 2024-04-22T01:17:31.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d0003d40 con 0x7f91d8155ae0 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d815a710 con 0x7f91d8155ae0 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d000aa90 con 0x7f91d8155ae0 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d80b2080 con 0x7f91d8155ae0 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d8068920 con 0x7f91d8155ae0 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f91d000ac30 con 0x7f91d8155ae0 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91d67fc640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f91c40678c0 0x7f91c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd138640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f91c40678c0 0x7f91c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:31.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6332+0+0 (secure 0 0 0) 0x7f91d0085de0 con 0x7f91d8155ae0 2024-04-22T01:17:31.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.669+0000 7f91dd138640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f91c40678c0 0x7f91c4069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0047c0 tx=0x7f91cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:31.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.673+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f91d0056740 con 0x7f91d8155ae0 2024-04-22T01:17:31.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:31.837+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f91d815d200 con 0x7f91d8155ae0 2024-04-22T01:17:32.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.749+0000 7f91d67fc640 1 -- 172.21.15.70:0/1589976372 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) ==== 181+0+0 (secure 0 0 0) 0x7f91d0056040 con 0x7f91d8155ae0 2024-04-22T01:17:32.752 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:17:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f91c40678c0 msgr2=0x7f91c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f91c40678c0 0x7f91c4069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0047c0 tx=0x7f91cc008040 comp rx=0 tx=0).stop 2024-04-22T01:17:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 msgr2=0x7f91d815f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 0x7f91d815f8a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f91d000c790 tx=0x7f91d000cc50 comp rx=0 tx=0).stop 2024-04-22T01:17:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 shutdown_connections 2024-04-22T01:17:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f91c40678c0 0x7f91c4069d80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91d8159690 0x7f91d8159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91d81f3c10 0x7f91d815fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 --2- 172.21.15.70:0/1589976372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91d8155ae0 0x7f91d815f8a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 >> 172.21.15.70:0/1589976372 conn(0x7f91d80733e0 msgr2=0x7f91d80ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:32.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 shutdown_connections 2024-04-22T01:17:32.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.753+0000 7f91dfb9d640 1 -- 172.21.15.70:0/1589976372 wait complete. 2024-04-22T01:17:32.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:17:32.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3499a58640 1 Processor -- start 2024-04-22T01:17:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3499a58640 1 -- start start 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3499a58640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 0x7f3494154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3499a58640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34940b3800 con 0x7f34940ceda0 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3499a58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34940b3ae0 con 0x7f349415e6e0 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3499a58640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f34940cde40 con 0x7f34940ce920 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34937fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f34940ceda0 0x7f349406d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37312/0 (socket says 172.21.15.70:37312) 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34937fe640 1 -- 172.21.15.70:0/1375736556 learned_addr learned my addr 172.21.15.70:0/1375736556 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 0x7f3494154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2479197838 0 0) 0x7f34940b3800 con 0x7f34940ceda0 2024-04-22T01:17:32.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f347c003530 con 0x7f34940ceda0 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2507370546 0 0) 0x7f347c003530 con 0x7f34940ceda0 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 msgr2=0x7f3494154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 0x7f3494154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 >> v1:172.21.15.98:6789/0 conn(0x7f349415e6e0 legacy=0x7f349415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34940ce120 con 0x7f34940ceda0 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f348c001a00 con 0x7f34940ceda0 2024-04-22T01:17:32.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 >> v1:172.21.15.70:6789/0 conn(0x7f34940ceda0 legacy=0x7f349406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0039f0 0x7f347c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f347c0043a0 0x7f347c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 0x7f347c0050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34940cde40 con 0x7f347c0043a0 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f347c00dc10 con 0x7f347c0039f0 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f347c00de10 con 0x7f34940ce920 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f347c0043a0 0x7f347c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f3492ffd640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0039f0 0x7f347c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34937fe640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 0x7f347c0050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 -- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 msgr2=0x7f347c0050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34940ce920 0x7f347c0050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 -- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0039f0 msgr2=0x7f347c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0039f0 0x7f347c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.905+0000 7f34927fc640 1 -- 172.21.15.70:0/1375736556 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c003530 con 0x7f347c0043a0 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f34927fc640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f347c0043a0 0x7f347c004870 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3488009ae0 tx=0x7f348801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437217b7611ecbad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34880b2070 con 0x7f347c0043a0 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34880a5660 con 0x7f347c0043a0 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3491ffb640 1 -- 172.21.15.70:0/1375736556 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34880096b0 con 0x7f347c0043a0 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 -- 172.21.15.70:0/1375736556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f347c0043a0 msgr2=0x7f347c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 --2- 172.21.15.70:0/1375736556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f347c0043a0 0x7f347c004870 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3488009ae0 tx=0x7f348801ca70 comp rx=0 tx=0).stop 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f34937fe640 1 -- 172.21.15.70:0/1375736556 reap_dead start 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 -- 172.21.15.70:0/1375736556 shutdown_connections 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 -- 172.21.15.70:0/1375736556 >> 172.21.15.70:0/1375736556 conn(0x7f34940b1d40 msgr2=0x7f34940cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 -- 172.21.15.70:0/1375736556 shutdown_connections 2024-04-22T01:17:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 -- 172.21.15.70:0/1375736556 wait complete. 2024-04-22T01:17:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 Processor -- start 2024-04-22T01:17:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.909+0000 7f3499a58640 1 -- start start 2024-04-22T01:17:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0043a0 0x7f349415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 0x7f34941e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3494159820 0x7f3494159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f349406f260 con 0x7f3494147460 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f349406f0e0 con 0x7f347c0043a0 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f349406e000 con 0x7f3494159820 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 0x7f34941e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 0x7f34941e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40000/0 (socket says 172.21.15.70:40000) 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 -- 172.21.15.70:0/778737964 learned_addr learned my addr 172.21.15.70:0/778737964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34937fe640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3494159820 0x7f3494159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3492ffd640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0043a0 0x7f349415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 -- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3494159820 msgr2=0x7f3494159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3494159820 0x7f3494159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 -- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0043a0 msgr2=0x7f349415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0043a0 0x7f349415d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 -- 172.21.15.70:0/778737964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34941e4e70 con 0x7f3494147460 2024-04-22T01:17:32.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f34927fc640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 0x7f34941e4930 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f3488009ae0 tx=0x7f347c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34880b2040 con 0x7f3494147460 2024-04-22T01:17:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348800f000 con 0x7f3494147460 2024-04-22T01:17:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34941e5140 con 0x7f3494147460 2024-04-22T01:17:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34880096b0 con 0x7f3494147460 2024-04-22T01:17:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34941e5700 con 0x7f3494147460 2024-04-22T01:17:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3494068210 con 0x7f3494147460 2024-04-22T01:17:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f34880c0030 con 0x7f3494147460 2024-04-22T01:17:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f347bfff640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34800678f0 0x7f3480069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5865+0+0 (secure 0 0 0) 0x7f348811f990 con 0x7f3494147460 2024-04-22T01:17:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3492ffd640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34800678f0 0x7f3480069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.913+0000 7f3492ffd640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34800678f0 0x7f3480069db0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f34840124b0 tx=0x7f348400a9b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:32.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:32.917+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f34880c7060 con 0x7f3494147460 2024-04-22T01:17:33.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.089+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f34940d08e0 con 0x7f3494147460 2024-04-22T01:17:33.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.757+0000 7f347bfff640 1 -- 172.21.15.70:0/778737964 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) ==== 169+0+0 (secure 0 0 0) 0x7f3488016780 con 0x7f3494147460 2024-04-22T01:17:33.761 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:17:33.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34800678f0 msgr2=0x7f3480069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34800678f0 0x7f3480069db0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f34840124b0 tx=0x7f348400a9b0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 msgr2=0x7f34941e4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 0x7f34941e4930 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f3488009ae0 tx=0x7f347c005840 comp rx=0 tx=0).stop 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 shutdown_connections 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f34800678f0 0x7f3480069db0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3494159820 0x7f3494159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3494147460 0x7f34941e4930 secure :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f3488009ae0 tx=0x7f347c005840 comp rx=0 tx=0).stop 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 --2- 172.21.15.70:0/778737964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0043a0 0x7f349415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 >> 172.21.15.70:0/778737964 conn(0x7f34940b1d40 msgr2=0x7f34940b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:33.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 shutdown_connections 2024-04-22T01:17:33.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.765+0000 7f3499a58640 1 -- 172.21.15.70:0/778737964 wait complete. 2024-04-22T01:17:33.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-22T01:17:33.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 Processor -- start 2024-04-22T01:17:33.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 -- start start 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9c640b4450 0x7f9c64071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c640720b0 0x7f9c640ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c6415c290 con 0x7f9c640cd050 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9c640b3a50 con 0x7f9c640b4450 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c69c87640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9c6415c570 con 0x7f9c640720b0 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c637fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9c640b4450 0x7f9c64071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c63fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9c640cd050 0x7f9c6415f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37320/0 (socket says 172.21.15.70:37320) 2024-04-22T01:17:33.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c637fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9c640b4450 0x7f9c64071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40014/0 (socket says 172.21.15.70:40014) 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c63fff640 1 -- 172.21.15.70:0/1904959808 learned_addr learned my addr 172.21.15.70:0/1904959808 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c62ffd640 1 --2- 172.21.15.70:0/1904959808 >> v2:172.21.15.98:3300/0 conn(0x7f9c640720b0 0x7f9c640ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c637fe640 1 -- 172.21.15.70:0/1904959808 >> v2:172.21.15.98:3300/0 conn(0x7f9c640720b0 msgr2=0x7f9c640ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c637fe640 1 --2- 172.21.15.70:0/1904959808 >> v2:172.21.15.98:3300/0 conn(0x7f9c640720b0 0x7f9c640ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c637fe640 1 -- 172.21.15.70:0/1904959808 >> v1:172.21.15.70:6789/0 conn(0x7f9c640cd050 legacy=0x7f9c6415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c627fc640 1 -- 172.21.15.70:0/1904959808 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3792024293 0 0) 0x7f9c6415c290 con 0x7f9c640cd050 2024-04-22T01:17:33.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.913+0000 7f9c637fe640 1 -- 172.21.15.70:0/1904959808 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6415c850 con 0x7f9c640b4450 2024-04-22T01:17:33.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c637fe640 1 --2- 172.21.15.70:0/1904959808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640b4450 0x7f9c64071b70 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9c640cf9a0 tx=0x7f9c4c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448b4fa49a077ff5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:33.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c627fc640 1 -- 172.21.15.70:0/1904959808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0b0070 con 0x7f9c640b4450 2024-04-22T01:17:33.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c627fc640 1 -- 172.21.15.70:0/1904959808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c4c0ab600 con 0x7f9c640b4450 2024-04-22T01:17:33.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c627fc640 1 -- 172.21.15.70:0/1904959808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0ab9d0 con 0x7f9c640b4450 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 -- 172.21.15.70:0/1904959808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640b4450 msgr2=0x7f9c64071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1904959808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640b4450 0x7f9c64071b70 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9c640cf9a0 tx=0x7f9c4c01c480 comp rx=0 tx=0).stop 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 -- 172.21.15.70:0/1904959808 shutdown_connections 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1904959808 >> v2:172.21.15.98:3300/0 conn(0x7f9c640720b0 0x7f9c640ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1904959808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640b4450 0x7f9c64071b70 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 -- 172.21.15.70:0/1904959808 >> 172.21.15.70:0/1904959808 conn(0x7f9c640b1d40 msgr2=0x7f9c640cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 -- 172.21.15.70:0/1904959808 shutdown_connections 2024-04-22T01:17:33.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 -- 172.21.15.70:0/1904959808 wait complete. 2024-04-22T01:17:33.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 Processor -- start 2024-04-22T01:17:33.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.917+0000 7f9c69c87640 1 -- start start 2024-04-22T01:17:33.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c640720b0 0x7f9c6414b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c640b4450 0x7f9c64155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 0x7f9c6414aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c640b49b0 con 0x7f9c640cd050 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c640b4830 con 0x7f9c640b4450 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c640cd800 con 0x7f9c640720b0 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 0x7f9c6414aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 0x7f9c6414aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40028/0 (socket says 172.21.15.70:40028) 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 -- 172.21.15.70:0/1484071369 learned_addr learned my addr 172.21.15.70:0/1484071369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c62ffd640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c640b4450 0x7f9c64155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 -- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c640720b0 msgr2=0x7f9c6414b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c637fe640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c640720b0 0x7f9c6414b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c640720b0 0x7f9c6414b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 -- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c640b4450 msgr2=0x7f9c64155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:33.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c640b4450 0x7f9c64155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 -- 172.21.15.70:0/1484071369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c641fa280 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c63fff640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 0x7f9c6414aae0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00c7a0 tx=0x7f9c5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c018070 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c5c00a640 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c641fa560 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c00aab0 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c641fab20 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c64068210 con 0x7f9c640cd050 2024-04-22T01:17:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9c5c01b070 con 0x7f9c640cd050 2024-04-22T01:17:33.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c5bfff640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c440678c0 0x7f9c44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:33.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7f9c5c0851a0 con 0x7f9c640cd050 2024-04-22T01:17:33.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c637fe640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c440678c0 0x7f9c44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:33.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.921+0000 7f9c637fe640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c440678c0 0x7f9c44069d80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c0ad000 tx=0x7f9c4c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:33.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:33.925+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9c5c055e50 con 0x7f9c640cd050 2024-04-22T01:17:34.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.085+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f9c640d01e0 con 0x7f9c640cd050 2024-04-22T01:17:34.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c5bfff640 1 -- 172.21.15.70:0/1484071369 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7f9c5c055750 con 0x7f9c640cd050 2024-04-22T01:17:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c440678c0 msgr2=0x7f9c44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c440678c0 0x7f9c44069d80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c0ad000 tx=0x7f9c4c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 msgr2=0x7f9c6414aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 0x7f9c6414aae0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00c7a0 tx=0x7f9c5c00cc60 comp rx=0 tx=0).stop 2024-04-22T01:17:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 shutdown_connections 2024-04-22T01:17:34.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c440678c0 0x7f9c44069d80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c640cd050 0x7f9c6414aae0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c640b4450 0x7f9c64155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 --2- 172.21.15.70:0/1484071369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c640720b0 0x7f9c6414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 >> 172.21.15.70:0/1484071369 conn(0x7f9c640b1d40 msgr2=0x7f9c640cf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 shutdown_connections 2024-04-22T01:17:34.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.093+0000 7f9c69c87640 1 -- 172.21.15.70:0/1484071369 wait complete. 2024-04-22T01:17:34.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values' 2024-04-22T01:17:34.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.217+0000 7fa4c50aa640 1 Processor -- start 2024-04-22T01:17:34.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.217+0000 7fa4c50aa640 1 -- start start 2024-04-22T01:17:34.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4c50aa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4c015bb40 0x7fa4c0072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4c50aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4c006ed40 0x7fa4c00b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4c50aa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c015bf20 con 0x7fa4c00b3b00 2024-04-22T01:17:34.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4c50aa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c00b3f70 con 0x7fa4c015bb40 2024-04-22T01:17:34.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4c50aa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c006e8b0 con 0x7fa4c006ed40 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4c015bb40 0x7fa4c0072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4c015bb40 0x7fa4c0072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40038/0 (socket says 172.21.15.70:40038) 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 -- 172.21.15.70:0/1876696619 learned_addr learned my addr 172.21.15.70:0/1876696619 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4be59c640 1 --2- 172.21.15.70:0/1876696619 >> v2:172.21.15.98:3300/0 conn(0x7fa4c006ed40 0x7fa4c00b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 -- 172.21.15.70:0/1876696619 >> v2:172.21.15.98:3300/0 conn(0x7fa4c006ed40 msgr2=0x7fa4c00b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 --2- 172.21.15.70:0/1876696619 >> v2:172.21.15.98:3300/0 conn(0x7fa4c006ed40 0x7fa4c00b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 -- 172.21.15.70:0/1876696619 >> v1:172.21.15.98:6789/0 conn(0x7fa4c00b3b00 legacy=0x7fa4c0161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 -- 172.21.15.70:0/1876696619 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c00aec30 con 0x7fa4c015bb40 2024-04-22T01:17:34.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4be59c640 1 --2- 172.21.15.70:0/1876696619 >> v2:172.21.15.98:3300/0 conn(0x7fa4c006ed40 0x7fa4c00b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:34.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bed9d640 1 --2- 172.21.15.70:0/1876696619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c0072530 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00b2d10 tx=0x7fa4b001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532952962f35fd42 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:34.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bdd9b640 1 -- 172.21.15.70:0/1876696619 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00aa610 con 0x7fa4c015bb40 2024-04-22T01:17:34.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bdd9b640 1 -- 172.21.15.70:0/1876696619 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b00af070 con 0x7fa4c015bb40 2024-04-22T01:17:34.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.221+0000 7fa4bdd9b640 1 -- 172.21.15.70:0/1876696619 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00aa9d0 con 0x7fa4c015bb40 2024-04-22T01:17:34.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1876696619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 msgr2=0x7fa4c0072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1876696619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c0072530 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00b2d10 tx=0x7fa4b001c4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1876696619 shutdown_connections 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1876696619 >> v2:172.21.15.98:3300/0 conn(0x7fa4c006ed40 0x7fa4c00b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1876696619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c0072530 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1876696619 >> 172.21.15.70:0/1876696619 conn(0x7fa4c006b1f0 msgr2=0x7fa4c006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1876696619 shutdown_connections 2024-04-22T01:17:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1876696619 wait complete. 2024-04-22T01:17:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 Processor -- start 2024-04-22T01:17:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- start start 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c006ed40 0x7fa4c0146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c00b3b00 0x7fa4c00ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c00ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c00ae9d0 con 0x7fa4c015bb40 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c00ae850 con 0x7fa4c006ed40 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c006f5c0 con 0x7fa4c00b3b00 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c00ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c00ad570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40046/0 (socket says 172.21.15.70:40046) 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 -- 172.21.15.70:0/1517512798 learned_addr learned my addr 172.21.15.70:0/1517512798 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4be59c640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c00b3b00 0x7fa4c00ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bed9d640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c006ed40 0x7fa4c0146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 -- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c00b3b00 msgr2=0x7fa4c00ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c00b3b00 0x7fa4c00ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 -- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c006ed40 msgr2=0x7fa4c0146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c006ed40 0x7fa4c0146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 -- 172.21.15.70:0/1517512798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c014e2c0 con 0x7fa4c015bb40 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4be59c640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c00b3b00 0x7fa4c00ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bed9d640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c006ed40 0x7fa4c0146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4bf59e640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c00ad570 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c780 tx=0x7fa4b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8004b70 con 0x7fa4c015bb40 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b8004d10 con 0x7fa4c015bb40 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c014e5a0 con 0x7fa4c015bb40 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b800aa90 con 0x7fa4c015bb40 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.225+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c0148e10 con 0x7fa4c015bb40 2024-04-22T01:17:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.229+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c0149010 con 0x7fa4c015bb40 2024-04-22T01:17:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.229+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa4b800ac30 con 0x7fa4c015bb40 2024-04-22T01:17:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.229+0000 7fa4a77fe640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa4ac0678c0 0x7fa4ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.229+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7fa4b8085dc0 con 0x7fa4c015bb40 2024-04-22T01:17:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.229+0000 7fa4bed9d640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa4ac0678c0 0x7fa4ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.229+0000 7fa4bed9d640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa4ac0678c0 0x7fa4ac069d80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa4b00ae000 tx=0x7fa4b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:34.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.233+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa4c0149010 con 0x7fa4c015bb40 2024-04-22T01:17:34.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.437+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]} v 0) -- 0x7fa4c0068210 con 0x7fa4c015bb40 2024-04-22T01:17:34.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.761+0000 7fa4a77fe640 1 -- 172.21.15.70:0/1517512798 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]}]=0 v176) ==== 145+0+0 (secure 0 0 0) 0x7fa4b8020070 con 0x7fa4c015bb40 2024-04-22T01:17:34.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa4ac0678c0 msgr2=0x7fa4ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa4ac0678c0 0x7fa4ac069d80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa4b00ae000 tx=0x7fa4b0008040 comp rx=0 tx=0).stop 2024-04-22T01:17:34.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 msgr2=0x7fa4c00ad570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c00ad570 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c780 tx=0x7fa4b800cc40 comp rx=0 tx=0).stop 2024-04-22T01:17:34.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 shutdown_connections 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa4ac0678c0 0x7fa4ac069d80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c015bb40 0x7fa4c00ad570 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c00b3b00 0x7fa4c00ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 --2- 172.21.15.70:0/1517512798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c006ed40 0x7fa4c0146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 >> 172.21.15.70:0/1517512798 conn(0x7fa4c006b1f0 msgr2=0x7fa4c00cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 shutdown_connections 2024-04-22T01:17:34.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.765+0000 7fa4c50aa640 1 -- 172.21.15.70:0/1517512798 wait complete. 2024-04-22T01:17:34.787 INFO:tasks.cephfs_test_runner:test human-readable values for setting ceph.quota.max_bytes ... ok 2024-04-22T01:17:34.788 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_df (tasks.cephfs.test_quota.TestQuota) That when a client modifies the quota on a directory used 2024-04-22T01:17:34.788 INFO:tasks.cephfs_test_runner:test_remote_update_df (tasks.cephfs.test_quota.TestQuota) 2024-04-22T01:17:34.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.909+0000 7f51a65e7640 1 Processor -- start 2024-04-22T01:17:34.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.909+0000 7f51a65e7640 1 -- start start 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f51a65e7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f51a015f3d0 0x7f51a0099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f51a65e7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a015f7b0 con 0x7f51a009a200 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f51a65e7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a00cd3e0 con 0x7f51a00d06c0 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f51a65e7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f51a00d0ba0 con 0x7f51a015f3d0 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f51a015f3d0 0x7f51a0099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f51a009a200 0x7f51a00c5630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37334/0 (socket says 172.21.15.70:37334) 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519f7fe640 1 -- 172.21.15.70:0/1586770483 learned_addr learned my addr 172.21.15.70:0/1586770483 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:34.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519ffff640 1 -- 172.21.15.70:0/1586770483 >> v1:172.21.15.98:6789/0 conn(0x7f51a00d06c0 legacy=0x7f51a00c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519effd640 1 -- 172.21.15.70:0/1586770483 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150749256 0 0) 0x7f51a015f7b0 con 0x7f51a009a200 2024-04-22T01:17:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519ffff640 1 -- 172.21.15.70:0/1586770483 >> v1:172.21.15.70:6789/0 conn(0x7f51a009a200 legacy=0x7f51a00c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519ffff640 1 -- 172.21.15.70:0/1586770483 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a00cd6c0 con 0x7f51a015f3d0 2024-04-22T01:17:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519ffff640 1 --2- 172.21.15.70:0/1586770483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a015f3d0 0x7f51a0099cc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f51900097c0 tx=0x7f519001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3465b5e1323a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519effd640 1 -- 172.21.15.70:0/1586770483 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900b0070 con 0x7f51a015f3d0 2024-04-22T01:17:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519effd640 1 -- 172.21.15.70:0/1586770483 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51900ab600 con 0x7f51a015f3d0 2024-04-22T01:17:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.913+0000 7f519effd640 1 -- 172.21.15.70:0/1586770483 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900ab9d0 con 0x7f51a015f3d0 2024-04-22T01:17:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/1586770483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a015f3d0 msgr2=0x7f51a0099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 --2- 172.21.15.70:0/1586770483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a015f3d0 0x7f51a0099cc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f51900097c0 tx=0x7f519001c460 comp rx=0 tx=0).stop 2024-04-22T01:17:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/1586770483 shutdown_connections 2024-04-22T01:17:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 --2- 172.21.15.70:0/1586770483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a015f3d0 0x7f51a0099cc0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/1586770483 >> 172.21.15.70:0/1586770483 conn(0x7f51a00b35d0 msgr2=0x7f51a00b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/1586770483 shutdown_connections 2024-04-22T01:17:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/1586770483 wait complete. 2024-04-22T01:17:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 Processor -- start 2024-04-22T01:17:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- start start 2024-04-22T01:17:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 0x7f51a01665c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a00d06c0 0x7f51a0153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a015f3d0 0x7f51a0165860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f51a009aa60 con 0x7f51a009a200 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51a00cd190 con 0x7f51a015f3d0 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f51a009abe0 con 0x7f51a00d06c0 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 0x7f51a01665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 0x7f51a01665c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40066/0 (socket says 172.21.15.70:40066) 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 -- 172.21.15.70:0/3370580710 learned_addr learned my addr 172.21.15.70:0/3370580710 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519f7fe640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a00d06c0 0x7f51a0153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 -- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a00d06c0 msgr2=0x7f51a0153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a00d06c0 0x7f51a0153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 -- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a015f3d0 msgr2=0x7f51a0165860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a4b84640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a015f3d0 0x7f51a0165860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a015f3d0 0x7f51a0165860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 -- 172.21.15.70:0/3370580710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a0166c50 con 0x7f51a009a200 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519ffff640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 0x7f51a01665c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5190019070 tx=0x7f519001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900b0050 con 0x7f51a009a200 2024-04-22T01:17:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51900bc070 con 0x7f51a009a200 2024-04-22T01:17:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a00cae90 con 0x7f51a009a200 2024-04-22T01:17:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900b5a70 con 0x7f51a009a200 2024-04-22T01:17:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.917+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a00cb450 con 0x7f51a009a200 2024-04-22T01:17:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.921+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a0068920 con 0x7f51a009a200 2024-04-22T01:17:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.921+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f51900ab4c0 con 0x7f51a009a200 2024-04-22T01:17:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.921+0000 7f519cff9640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f518c0678c0 0x7f518c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.921+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7f51901203d0 con 0x7f51a009a200 2024-04-22T01:17:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.921+0000 7f519f7fe640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f518c0678c0 0x7f518c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.921+0000 7f519f7fe640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f518c0678c0 0x7f518c069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51a015edf0 tx=0x7f5194009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:34.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:34.925+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5190016050 con 0x7f51a009a200 2024-04-22T01:17:35.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.085+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51a00c75d0 con 0x7f51a009a200 2024-04-22T01:17:35.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f519cff9640 1 -- 172.21.15.70:0/3370580710 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+13258 (secure 0 0 0) 0x7f51900f11a0 con 0x7f51a009a200 2024-04-22T01:17:35.089 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:35.089 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":69,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:33.755055+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:6837/4194165476":"2024-04-23T01:17:30.459911+0000","172.21.15.70:6836/4194165476":"2024-04-23T01:17:30.459911+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f518c0678c0 msgr2=0x7f518c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f518c0678c0 0x7f518c069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51a015edf0 tx=0x7f5194009040 comp rx=0 tx=0).stop 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 msgr2=0x7f51a01665c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 0x7f51a01665c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5190019070 tx=0x7f519001aca0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 shutdown_connections 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f518c0678c0 0x7f518c069d80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a015f3d0 0x7f51a0165860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a00d06c0 0x7f51a0153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 --2- 172.21.15.70:0/3370580710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51a009a200 0x7f51a01665c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 >> 172.21.15.70:0/3370580710 conn(0x7f51a00b35d0 msgr2=0x7f51a015d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 shutdown_connections 2024-04-22T01:17:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.089+0000 7f51a65e7640 1 -- 172.21.15.70:0/3370580710 wait complete. 2024-04-22T01:17:35.109 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:35.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:35.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.221+0000 7f9b5901f640 1 Processor -- start 2024-04-22T01:17:35.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.221+0000 7f9b5901f640 1 -- start start 2024-04-22T01:17:35.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9b5406b900 0x7f9b54161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b54162040 con 0x7f9b540cd090 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b54162320 con 0x7f9b540cca20 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9b540b02b0 con 0x7f9b5406b900 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9b5406b900 0x7f9b54161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9b5406b900 0x7f9b54161a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40076/0 (socket says 172.21.15.70:40076) 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5359e640 1 --1- 172.21.15.70:0/542672825 >> v1:172.21.15.70:6789/0 conn(0x7f9b540cd090 0x7f9b5415e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37340/0 (socket says 172.21.15.70:37340) 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 -- 172.21.15.70:0/542672825 learned_addr learned my addr 172.21.15.70:0/542672825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 -- 172.21.15.70:0/542672825 >> v1:172.21.15.98:6789/0 conn(0x7f9b540cca20 legacy=0x7f9b5406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 -- 172.21.15.70:0/542672825 >> v1:172.21.15.70:6789/0 conn(0x7f9b540cd090 legacy=0x7f9b5415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 -- 172.21.15.70:0/542672825 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b54162600 con 0x7f9b5406b900 2024-04-22T01:17:35.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5259c640 1 --2- 172.21.15.70:0/542672825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b54161a60 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009870 tx=0x7f9b4801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32040c90879a49df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b51d9b640 1 -- 172.21.15.70:0/542672825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b480ab5f0 con 0x7f9b5406b900 2024-04-22T01:17:35.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b51d9b640 1 -- 172.21.15.70:0/542672825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b480b0070 con 0x7f9b5406b900 2024-04-22T01:17:35.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b51d9b640 1 -- 172.21.15.70:0/542672825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b480ab9b0 con 0x7f9b5406b900 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- 172.21.15.70:0/542672825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 msgr2=0x7f9b54161a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 --2- 172.21.15.70:0/542672825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b54161a60 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009870 tx=0x7f9b4801c520 comp rx=0 tx=0).stop 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- 172.21.15.70:0/542672825 shutdown_connections 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 --2- 172.21.15.70:0/542672825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b54161a60 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- 172.21.15.70:0/542672825 >> 172.21.15.70:0/542672825 conn(0x7f9b54099a60 msgr2=0x7f9b54099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- 172.21.15.70:0/542672825 shutdown_connections 2024-04-22T01:17:35.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.225+0000 7f9b5901f640 1 -- 172.21.15.70:0/542672825 wait complete. 2024-04-22T01:17:35.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 Processor -- start 2024-04-22T01:17:35.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- start start 2024-04-22T01:17:35.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b5414ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b540cca20 0x7f9b54159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b540cd090 0x7f9b54159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5415f290 con 0x7f9b5406b900 2024-04-22T01:17:35.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b541628e0 con 0x7f9b540cd090 2024-04-22T01:17:35.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9b5406fb70 con 0x7f9b540cca20 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b5414ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b5414ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40084/0 (socket says 172.21.15.70:40084) 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 -- 172.21.15.70:0/530389698 learned_addr learned my addr 172.21.15.70:0/530389698 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 -- 172.21.15.70:0/530389698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b540cca20 msgr2=0x7f9b54159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b540cca20 0x7f9b54159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 -- 172.21.15.70:0/530389698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b540cd090 msgr2=0x7f9b54159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b540cd090 0x7f9b54159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 -- 172.21.15.70:0/530389698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b540c5260 con 0x7f9b5406b900 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b52d9d640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b5414ecb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400c490 tx=0x7f9b44001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b44017070 con 0x7f9b5406b900 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b44003600 con 0x7f9b5406b900 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b540c5540 con 0x7f9b5406b900 2024-04-22T01:17:35.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b44003a30 con 0x7f9b5406b900 2024-04-22T01:17:35.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b540c5b00 con 0x7f9b5406b900 2024-04-22T01:17:35.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b54068920 con 0x7f9b5406b900 2024-04-22T01:17:35.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.229+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b4401a070 con 0x7f9b5406b900 2024-04-22T01:17:35.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.233+0000 7f9b3b7fe640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b400678c0 0x7f9b40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.233+0000 7f9b5259c640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b400678c0 0x7f9b40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.233+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7f9b44084ff0 con 0x7f9b5406b900 2024-04-22T01:17:35.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.233+0000 7f9b5259c640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b400678c0 0x7f9b40069d80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9b480af000 tx=0x7f9b48004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.237+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9b44055c20 con 0x7f9b5406b900 2024-04-22T01:17:35.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.393+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9b540cd6a0 con 0x7f9b5406b900 2024-04-22T01:17:35.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.393+0000 7f9b3b7fe640 1 -- 172.21.15.70:0/530389698 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) ==== 93+0+10 (secure 0 0 0) 0x7f9b44055520 con 0x7f9b5406b900 2024-04-22T01:17:35.397 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:35.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b400678c0 msgr2=0x7f9b40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b400678c0 0x7f9b40069d80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9b480af000 tx=0x7f9b48004740 comp rx=0 tx=0).stop 2024-04-22T01:17:35.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 msgr2=0x7f9b5414ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b5414ecb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400c490 tx=0x7f9b44001e80 comp rx=0 tx=0).stop 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 shutdown_connections 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9b400678c0 0x7f9b40069d80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b540cd090 0x7f9b54159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b540cca20 0x7f9b54159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 --2- 172.21.15.70:0/530389698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b5406b900 0x7f9b5414ecb0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 >> 172.21.15.70:0/530389698 conn(0x7f9b54099a60 msgr2=0x7f9b541608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 shutdown_connections 2024-04-22T01:17:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.397+0000 7f9b5901f640 1 -- 172.21.15.70:0/530389698 wait complete. 2024-04-22T01:17:35.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df' 2024-04-22T01:17:35.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1c4be640 1 Processor -- start 2024-04-22T01:17:35.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1c4be640 1 -- start start 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1c4be640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd140cd740 0x7fcd1406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1c4be640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd1406bb50 con 0x7fcd1406b730 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1c4be640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd1415c050 con 0x7fcd1415bc70 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1c4be640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcd140cfd10 con 0x7fcd140cd740 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcd1406b730 0x7fcd140d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37352/0 (socket says 172.21.15.70:37352) 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 -- 172.21.15.70:0/2669810548 learned_addr learned my addr 172.21.15.70:0/2669810548 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19a59640 1 --2- 172.21.15.70:0/2669810548 >> v2:172.21.15.98:3300/0 conn(0x7fcd140cd740 0x7fcd1406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053600566 0 0) 0x7fcd1406bb50 con 0x7fcd1406b730 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd00003530 con 0x7fcd1406b730 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1136621373 0 0) 0x7fcd00003530 con 0x7fcd1406b730 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 >> v2:172.21.15.98:3300/0 conn(0x7fcd140cd740 msgr2=0x7fcd1406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 --2- 172.21.15.70:0/2669810548 >> v2:172.21.15.98:3300/0 conn(0x7fcd140cd740 0x7fcd1406b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 >> v1:172.21.15.98:6789/0 conn(0x7fcd1415bc70 legacy=0x7fcd1415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1415ff90 con 0x7fcd1406b730 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fcd0c001a00 con 0x7fcd1406b730 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 >> v1:172.21.15.70:6789/0 conn(0x7fcd1406b730 legacy=0x7fcd140d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd000039c0 0x7fcd00003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd00004810 0x7fcd00004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd000051d0 0x7fcd0000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140cfd10 con 0x7fcd000051d0 2024-04-22T01:17:35.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0000c070 con 0x7fcd00004810 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd0000c270 con 0x7fcd000039c0 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd000051d0 0x7fcd0000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd19a59640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd00004810 0x7fcd00004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1a25a640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd000039c0 0x7fcd00003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 -- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd000039c0 msgr2=0x7fcd00003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd000039c0 0x7fcd00003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 -- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd00004810 msgr2=0x7fcd00004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd00004810 0x7fcd00004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.525+0000 7fcd1aa5b640 1 -- 172.21.15.70:0/2669810548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd00003530 con 0x7fcd000051d0 2024-04-22T01:17:35.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1aa5b640 1 -- 172.21.15.70:0/2669810548 reap_dead start 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1aa5b640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd000051d0 0x7fcd0000b9d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00a860 tx=0x7fcd0c01d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cfec8634269e30 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c00a480 con 0x7fcd000051d0 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c0ad630 con 0x7fcd000051d0 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19258640 1 -- 172.21.15.70:0/2669810548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0ada20 con 0x7fcd000051d0 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- 172.21.15.70:0/2669810548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd000051d0 msgr2=0x7fcd0000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd000051d0 0x7fcd0000b9d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00a860 tx=0x7fcd0c01d380 comp rx=0 tx=0).stop 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- 172.21.15.70:0/2669810548 shutdown_connections 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/2669810548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd000051d0 0x7fcd0000b9d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- 172.21.15.70:0/2669810548 >> 172.21.15.70:0/2669810548 conn(0x7fcd14099a60 msgr2=0x7fcd14099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- 172.21.15.70:0/2669810548 shutdown_connections 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- 172.21.15.70:0/2669810548 wait complete. 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 Processor -- start 2024-04-22T01:17:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- start start 2024-04-22T01:17:35.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd000051d0 0x7fcd140c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 0x7fcd1414ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd140b5c00 0x7fcd140b6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140cff60 con 0x7fcd1414ff00 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd14160550 con 0x7fcd000051d0 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1c4be640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1406b1f0 con 0x7fcd140b5c00 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 0x7fcd1414ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 0x7fcd1414ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40106/0 (socket says 172.21.15.70:40106) 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 -- 172.21.15.70:0/1016893429 learned_addr learned my addr 172.21.15.70:0/1016893429 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1aa5b640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd140b5c00 0x7fcd140b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1a25a640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd000051d0 0x7fcd140c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 -- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd140b5c00 msgr2=0x7fcd140b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd140b5c00 0x7fcd140b6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 -- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd000051d0 msgr2=0x7fcd140c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd000051d0 0x7fcd140c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd19a59640 1 -- 172.21.15.70:0/1016893429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd140c7fe0 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.529+0000 7fcd1aa5b640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd140b5c00 0x7fcd140b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd19a59640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 0x7fcd1414ccf0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcd04011590 tx=0x7fcd04011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04015870 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd04015a10 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd140c9350 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04015dc0 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd140c8590 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd14068920 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd04013400 con 0x7fcd1414ff00 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd0affd640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccf80678f0 0x7fccf8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd1a25a640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccf80678f0 0x7fccf8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7fcd040862e0 con 0x7fcd1414ff00 2024-04-22T01:17:35.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.533+0000 7fcd1a25a640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccf80678f0 0x7fccf8069db0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcd10005490 tx=0x7fcd10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.537+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcd04019030 con 0x7fcd1414ff00 2024-04-22T01:17:35.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.741+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]} v 0) -- 0x7fcd140cd520 con 0x7fcd1414ff00 2024-04-22T01:17:35.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd0affd640 1 -- 172.21.15.70:0/1016893429 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]}]=0 v177) ==== 137+0+0 (secure 0 0 0) 0x7fcd04056f90 con 0x7fcd1414ff00 2024-04-22T01:17:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccf80678f0 msgr2=0x7fccf8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccf80678f0 0x7fccf8069db0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcd10005490 tx=0x7fcd10009040 comp rx=0 tx=0).stop 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 msgr2=0x7fcd1414ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 0x7fcd1414ccf0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcd04011590 tx=0x7fcd04011a50 comp rx=0 tx=0).stop 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 shutdown_connections 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccf80678f0 0x7fccf8069db0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd140b5c00 0x7fcd140b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1414ff00 0x7fcd1414ccf0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 --2- 172.21.15.70:0/1016893429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd000051d0 0x7fcd140c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 >> 172.21.15.70:0/1016893429 conn(0x7fcd14099a60 msgr2=0x7fcd140af790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 shutdown_connections 2024-04-22T01:17:35.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.765+0000 7fcd1c4be640 1 -- 172.21.15.70:0/1016893429 wait complete. 2024-04-22T01:17:35.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-22T01:17:35.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.889+0000 7f812ac9c640 1 Processor -- start 2024-04-22T01:17:35.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.889+0000 7f812ac9c640 1 -- start start 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f812ac9c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81240b98a0 0x7f81240d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f812ac9c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f81240cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f812ac9c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81240b9ce0 con 0x7f812415de20 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f812ac9c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f81240cf3c0 con 0x7f81240b98a0 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f812ac9c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f81240b75e0 con 0x7f81240cea10 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8128a38640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f812415de20 0x7f81240b6db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37356/0 (socket says 172.21.15.70:37356) 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8128a38640 1 -- 172.21.15.70:0/484132246 learned_addr learned my addr 172.21.15.70:0/484132246 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f81240cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8123fff640 1 --2- 172.21.15.70:0/484132246 >> v2:172.21.15.98:3300/0 conn(0x7f81240b98a0 0x7f81240d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232307849 0 0) 0x7f81240b9ce0 con 0x7f812415de20 2024-04-22T01:17:35.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f810c003530 con 0x7f812415de20 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1449214622 0 0) 0x7f810c003530 con 0x7f812415de20 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 msgr2=0x7f81240cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f81240cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 >> v2:172.21.15.98:3300/0 conn(0x7f81240b98a0 msgr2=0x7f81240d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 --2- 172.21.15.70:0/484132246 >> v2:172.21.15.98:3300/0 conn(0x7f81240b98a0 0x7f81240d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81240b78c0 con 0x7f812415de20 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f81180016a0 con 0x7f812415de20 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 >> v1:172.21.15.70:6789/0 conn(0x7f812415de20 legacy=0x7f81240b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f810c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f810c003f90 0x7f810c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f810c004d50 0x7f810c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81240cf3c0 con 0x7f810c004d50 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81240b75e0 con 0x7f810c003f90 2024-04-22T01:17:35.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f810c00ba50 con 0x7f81240cea10 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f810c004d50 0x7f810c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8128a38640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f810c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 -- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 msgr2=0x7f810c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f810c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8123fff640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f810c003f90 0x7f810c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 -- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f810c003f90 msgr2=0x7f810c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f810c003f90 0x7f810c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 -- 172.21.15.70:0/484132246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c003530 con 0x7f810c004d50 2024-04-22T01:17:35.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8128a38640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240cea10 0x7f810c0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:35.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f8129239640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f810c004d50 0x7f810c0051d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f811c009fc0 tx=0x7f811c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3007402998228bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b2070 con 0x7f810c004d50 2024-04-22T01:17:35.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811c0a5c90 con 0x7f810c004d50 2024-04-22T01:17:35.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.893+0000 7f81237fe640 1 -- 172.21.15.70:0/484132246 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c009c10 con 0x7f810c004d50 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/484132246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f810c004d50 msgr2=0x7f810c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 --2- 172.21.15.70:0/484132246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f810c004d50 0x7f810c0051d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f811c009fc0 tx=0x7f811c01cc80 comp rx=0 tx=0).stop 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 -- 172.21.15.70:0/484132246 reap_dead start 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/484132246 shutdown_connections 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/484132246 >> 172.21.15.70:0/484132246 conn(0x7f8124072d20 msgr2=0x7f8124073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/484132246 shutdown_connections 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/484132246 wait complete. 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 Processor -- start 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- start start 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81241585a0 0x7f81240d2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240ccfd0 0x7f81240d3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 0x7f81240c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81240b8160 con 0x7f81240d4490 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81240b7e80 con 0x7f81241585a0 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f812415e200 con 0x7f81240ccfd0 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 0x7f81240c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 0x7f81240c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40114/0 (socket says 172.21.15.70:40114) 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 -- 172.21.15.70:0/2359112097 learned_addr learned my addr 172.21.15.70:0/2359112097 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8123fff640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240ccfd0 0x7f81240d3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8128a38640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81241585a0 0x7f81240d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 -- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240ccfd0 msgr2=0x7f81240d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240ccfd0 0x7f81240d3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 -- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81241585a0 msgr2=0x7f81240d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81241585a0 0x7f81240d2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 -- 172.21.15.70:0/2359112097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81240d1dc0 con 0x7f81240d4490 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8123fff640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240ccfd0 0x7f81240d3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:35.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f8129239640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 0x7f81240c7fe0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f811c0b0040 tx=0x7f811c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0bd070 con 0x7f81240d4490 2024-04-22T01:17:35.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811c0a5a90 con 0x7f81240d4490 2024-04-22T01:17:35.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81240c7500 con 0x7f81240d4490 2024-04-22T01:17:35.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b2040 con 0x7f81240d4490 2024-04-22T01:17:35.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81240c7ac0 con 0x7f81240d4490 2024-04-22T01:17:35.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.897+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8124068210 con 0x7f81240d4490 2024-04-22T01:17:35.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.901+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f811c0c0070 con 0x7f81240d4490 2024-04-22T01:17:35.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.901+0000 7f81217fa640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81100678c0 0x7f8110069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:35.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.901+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7f811c11f7d0 con 0x7f81240d4490 2024-04-22T01:17:35.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.901+0000 7f8128a38640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81100678c0 0x7f8110069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:35.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.901+0000 7f8128a38640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81100678c0 0x7f8110069d80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8118004850 tx=0x7f8118009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:35.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:35.905+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f811c016990 con 0x7f81240d4490 2024-04-22T01:17:36.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.053+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f81240d1af0 con 0x7f81240d4490 2024-04-22T01:17:36.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f81217fa640 1 -- 172.21.15.70:0/2359112097 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f811c0effe0 con 0x7f81240d4490 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81100678c0 msgr2=0x7f8110069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81100678c0 0x7f8110069d80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8118004850 tx=0x7f8118009040 comp rx=0 tx=0).stop 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 msgr2=0x7f81240c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 0x7f81240c7fe0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f811c0b0040 tx=0x7f811c009750 comp rx=0 tx=0).stop 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 shutdown_connections 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81100678c0 0x7f8110069d80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81240d4490 0x7f81240c7fe0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81240ccfd0 0x7f81240d3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 --2- 172.21.15.70:0/2359112097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81241585a0 0x7f81240d2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 >> 172.21.15.70:0/2359112097 conn(0x7f8124072d20 msgr2=0x7f81240b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:36.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 shutdown_connections 2024-04-22T01:17:36.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.057+0000 7f812ac9c640 1 -- 172.21.15.70:0/2359112097 wait complete. 2024-04-22T01:17:36.078 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:17:36.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:17:36.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:17:36.163 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:17:36.163 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:17:36.163 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:36.168 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:17:36.169 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:36.169 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:36.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:17:36.210 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:17:36.210 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:36.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:17:36.220 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:36.220 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:36.262 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:17:36.262 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-22T01:17:36.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:36.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 Processor -- start 2024-04-22T01:17:36.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 -- start start 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcef81455b0 0x7fcef80cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80cc150 0x7fcef8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef80d0250 con 0x7fcef8072060 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcef80724d0 con 0x7fcef81455b0 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcefd738640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcef80cd7b0 con 0x7fcef80cc150 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcef81455b0 0x7fcef80cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcef81455b0 0x7fcef80cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43854/0 (socket says 172.21.15.70:43854) 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 -- 172.21.15.70:0/48292195 learned_addr learned my addr 172.21.15.70:0/48292195 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef67fc640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80cc150 0x7fcef8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 -- 172.21.15.70:0/48292195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80cc150 msgr2=0x7fcef8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80cc150 0x7fcef8071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 -- 172.21.15.70:0/48292195 >> v1:172.21.15.98:6789/0 conn(0x7fcef8072060 legacy=0x7fcef815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:36.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 -- 172.21.15.70:0/48292195 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef80cc710 con 0x7fcef81455b0 2024-04-22T01:17:36.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef67fc640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80cc150 0x7fcef8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:36.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.385+0000 7fcef6ffd640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef81455b0 0x7fcef80cfd10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009870 tx=0x7fcee801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0939e4874ae514e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:36.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef5ffb640 1 -- 172.21.15.70:0/48292195 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b0070 con 0x7fcef81455b0 2024-04-22T01:17:36.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef5ffb640 1 -- 172.21.15.70:0/48292195 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee80ab600 con 0x7fcef81455b0 2024-04-22T01:17:36.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef5ffb640 1 -- 172.21.15.70:0/48292195 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80ab9d0 con 0x7fcef81455b0 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/48292195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef81455b0 msgr2=0x7fcef80cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef81455b0 0x7fcef80cfd10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009870 tx=0x7fcee801c400 comp rx=0 tx=0).stop 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/48292195 shutdown_connections 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80cc150 0x7fcef8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 --2- 172.21.15.70:0/48292195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef81455b0 0x7fcef80cfd10 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/48292195 >> 172.21.15.70:0/48292195 conn(0x7fcef806b1f0 msgr2=0x7fcef806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/48292195 shutdown_connections 2024-04-22T01:17:36.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/48292195 wait complete. 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 Processor -- start 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- start start 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 0x7fcef81594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef80cc150 0x7fcef8160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef81455b0 0x7fcef8160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcef8168b40 con 0x7fcef8072060 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcef806f210 con 0x7fcef80cc150 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcef8168cc0 con 0x7fcef81455b0 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 0x7fcef81594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 0x7fcef81594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43860/0 (socket says 172.21.15.70:43860) 2024-04-22T01:17:36.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 -- 172.21.15.70:0/3994273316 learned_addr learned my addr 172.21.15.70:0/3994273316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef67fc640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef80cc150 0x7fcef8160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 -- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef81455b0 msgr2=0x7fcef8160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef81455b0 0x7fcef8160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 -- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef80cc150 msgr2=0x7fcef8160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef80cc150 0x7fcef8160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 -- 172.21.15.70:0/3994273316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef815dfb0 con 0x7fcef8072060 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcef6ffd640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 0x7fcef81594f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fcee801aab0 tx=0x7fcee801ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80ba020 con 0x7fcef8072060 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee80b6640 con 0x7fcef8072060 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef813eb60 con 0x7fcef8072060 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b0040 con 0x7fcef8072060 2024-04-22T01:17:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.389+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef813f120 con 0x7fcef8072060 2024-04-22T01:17:36.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.393+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef8068210 con 0x7fcef8072060 2024-04-22T01:17:36.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.393+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcee80bd020 con 0x7fcef8072060 2024-04-22T01:17:36.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.393+0000 7fcedf7fe640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcee40678c0 0x7fcee4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.393+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5335+0+0 (secure 0 0 0) 0x7fcee811fe90 con 0x7fcef8072060 2024-04-22T01:17:36.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.393+0000 7fcef67fc640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcee40678c0 0x7fcee4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.393+0000 7fcef67fc640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcee40678c0 0x7fcee4069d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fceec006fd0 tx=0x7fceec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:36.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.397+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcee80c4030 con 0x7fcef8072060 2024-04-22T01:17:36.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcef8141260 con 0x7fcef8072060 2024-04-22T01:17:36.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcedf7fe640 1 -- 172.21.15.70:0/3994273316 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+3866 (secure 0 0 0) 0x7fcee8016110 con 0x7fcef8072060 2024-04-22T01:17:36.594 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:36.594 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5012,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/771746880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":771746880},{"type":"v1","addr":"172.21.15.98:6835","nonce":771746880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5365,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1808497822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1808497822},{"type":"v1","addr":"172.21.15.98:6833","nonce":1808497822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2537204442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2537204442},{"type":"v1","addr":"172.21.15.70:6835","nonce":2537204442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6660,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3123527601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3123527601},{"type":"v1","addr":"172.21.15.70:6837","nonce":3123527601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[]} 2024-04-22T01:17:36.594 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 33 2024-04-22T01:17:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcee40678c0 msgr2=0x7fcee4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcee40678c0 0x7fcee4069d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fceec006fd0 tx=0x7fceec008040 comp rx=0 tx=0).stop 2024-04-22T01:17:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 msgr2=0x7fcef81594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 0x7fcef81594f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fcee801aab0 tx=0x7fcee801ac40 comp rx=0 tx=0).stop 2024-04-22T01:17:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 shutdown_connections 2024-04-22T01:17:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcee40678c0 0x7fcee4069d80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef81455b0 0x7fcef8160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef80cc150 0x7fcef8160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 --2- 172.21.15.70:0/3994273316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcef8072060 0x7fcef81594f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 >> 172.21.15.70:0/3994273316 conn(0x7fcef806b1f0 msgr2=0x7fcef815ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 shutdown_connections 2024-04-22T01:17:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.593+0000 7fcefd738640 1 -- 172.21.15.70:0/3994273316 wait complete. 2024-04-22T01:17:36.613 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:17:36.613 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-22T01:17:36.614 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:17:36.614 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:36.614 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:17:36.614 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-22T01:17:36.614 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:17:36.614 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:36.615 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:17:36.615 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-22T01:17:36.615 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:17:36.615 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:36.615 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:17:36.615 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-22T01:17:36.615 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:17:36.615 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:17:36.626 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:17:36.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:17:36.627 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:17:36.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:17:36.632 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:17:36.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:17:36.635 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:17:36.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:17:36.670 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:17:36.671 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:36.671 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:17:36.671 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:36.678 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:17:36.678 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:36.679 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:17:36.679 DEBUG:teuthology.parallel:result is None 2024-04-22T01:17:36.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-22T01:17:36.736 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:17:36.737 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:17:36.747 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:17:36.749 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:17:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.801+0000 7f05bee2b640 1 Processor -- start 2024-04-22T01:17:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.801+0000 7f05bee2b640 1 -- start start 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05b806b900 0x7f05b80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05b80d03a0 con 0x7f05b815dec0 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05b80cb0e0 con 0x7f05b80cfd30 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f05b80cb3c0 con 0x7f05b806b900 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bcbc7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f05b815dec0 0x7f05b80ccd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37812/0 (socket says 172.21.15.70:37812) 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bcbc7640 1 -- 172.21.15.70:0/623883666 learned_addr learned my addr 172.21.15.70:0/623883666 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 563321470 0 0) 0x7f05b80d03a0 con 0x7f05b815dec0 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05a0003530 con 0x7f05b815dec0 2024-04-22T01:17:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/623883666 >> v2:172.21.15.98:3300/0 conn(0x7f05b806b900 0x7f05b80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2709086430 0 0) 0x7f05a0003530 con 0x7f05b815dec0 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 >> v2:172.21.15.98:3300/0 conn(0x7f05b806b900 msgr2=0x7f05b80b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 --2- 172.21.15.70:0/623883666 >> v2:172.21.15.98:3300/0 conn(0x7f05b806b900 0x7f05b80b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 >> v1:172.21.15.98:6789/0 conn(0x7f05b80cfd30 legacy=0x7f05b806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b80cb6a0 con 0x7f05b815dec0 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f05a8001680 con 0x7f05b815dec0 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 >> v1:172.21.15.70:6789/0 conn(0x7f05b815dec0 legacy=0x7f05b80ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a0003a30 0x7f05a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a0004810 0x7f05a0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a00051d0 0x7f05a000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05b80cb3c0 con 0x7f05a00051d0 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05a000c100 con 0x7f05a0004810 2024-04-22T01:17:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05a000c300 con 0x7f05a0003a30 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a00051d0 0x7f05a000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b7fff640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a0004810 0x7f05a0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bcbc7640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a0003a30 0x7f05a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 -- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a0003a30 msgr2=0x7f05a0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05a0003a30 0x7f05a0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 -- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a0004810 msgr2=0x7f05a0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a0004810 0x7f05a0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 -- 172.21.15.70:0/623883666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a0003530 con 0x7f05a00051d0 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 -- 172.21.15.70:0/623883666 reap_dead start 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a00051d0 0x7f05a000b9d0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009f90 tx=0x7f05b001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685513dab1c23fec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0009c10 con 0x7f05a00051d0 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b00a5c90 con 0x7f05a00051d0 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05b77fe640 1 -- 172.21.15.70:0/623883666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0003be0 con 0x7f05a00051d0 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 -- 172.21.15.70:0/623883666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a00051d0 msgr2=0x7f05a000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a00051d0 0x7f05a000b9d0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009f90 tx=0x7f05b001cc80 comp rx=0 tx=0).stop 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 -- 172.21.15.70:0/623883666 shutdown_connections 2024-04-22T01:17:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 --2- 172.21.15.70:0/623883666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05a00051d0 0x7f05a000b9d0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.805+0000 7f05bee2b640 1 -- 172.21.15.70:0/623883666 >> 172.21.15.70:0/623883666 conn(0x7f05b8099a60 msgr2=0x7f05b8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- 172.21.15.70:0/623883666 shutdown_connections 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- 172.21.15.70:0/623883666 wait complete. 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 Processor -- start 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- start start 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a00051d0 0x7f05b80c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05b80c8000 0x7f05b80c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 0x7f05b80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05b815e420 con 0x7f05b80c6170 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05b815e2a0 con 0x7f05a00051d0 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05b80b3820 con 0x7f05b80c8000 2024-04-22T01:17:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 0x7f05b80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 0x7f05b80c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43908/0 (socket says 172.21.15.70:43908) 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 -- 172.21.15.70:0/1941529820 learned_addr learned my addr 172.21.15.70:0/1941529820 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05b7fff640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05b80c8000 0x7f05b80c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bcbc7640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a00051d0 0x7f05b80c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 -- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05b80c8000 msgr2=0x7f05b80c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05b80c8000 0x7f05b80c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 -- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a00051d0 msgr2=0x7f05b80c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a00051d0 0x7f05b80c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 -- 172.21.15.70:0/1941529820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b80c6b30 con 0x7f05b80c6170 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bcbc7640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a00051d0 0x7f05b80c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bd3c8640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 0x7f05b80c65f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f05b001cab0 tx=0x7f05b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0009c10 con 0x7f05b80c6170 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b00be070 con 0x7f05b80c6170 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b80c85a0 con 0x7f05b80c6170 2024-04-22T01:17:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0004450 con 0x7f05b80c6170 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b80c8b60 con 0x7f05b80c6170 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.809+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b804f510 con 0x7f05b80c6170 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.813+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05b00045f0 con 0x7f05b80c6170 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.813+0000 7f05b57fa640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05a40678f0 0x7f05a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.813+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5335+0+0 (secure 0 0 0) 0x7f05b0121040 con 0x7f05b80c6170 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.813+0000 7f05bcbc7640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05a40678f0 0x7f05a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.813+0000 7f05bcbc7640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05a40678f0 0x7f05a4069db0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f05a8001000 tx=0x7f05a8000bd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:36.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.817+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f05b0016990 con 0x7f05b80c6170 2024-04-22T01:17:36.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:36.985+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f05b806b1f0 con 0x7f05b80c6170 2024-04-22T01:17:37.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.781+0000 7f05b57fa640 1 -- 172.21.15.70:0/1941529820 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v74) ==== 97+0+0 (secure 0 0 0) 0x7f05b00b53a0 con 0x7f05b80c6170 2024-04-22T01:17:37.784 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-22T01:17:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05a40678f0 msgr2=0x7f05a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05a40678f0 0x7f05a4069db0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f05a8001000 tx=0x7f05a8000bd0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 msgr2=0x7f05b80c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 0x7f05b80c65f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f05b001cab0 tx=0x7f05b0009750 comp rx=0 tx=0).stop 2024-04-22T01:17:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 shutdown_connections 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05a40678f0 0x7f05a4069db0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b80c6170 0x7f05b80c65f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05b80c8000 0x7f05b80c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 --2- 172.21.15.70:0/1941529820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a00051d0 0x7f05b80c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 >> 172.21.15.70:0/1941529820 conn(0x7f05b8099a60 msgr2=0x7f05b806f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 shutdown_connections 2024-04-22T01:17:37.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.785+0000 7f05bee2b640 1 -- 172.21.15.70:0/1941529820 wait complete. 2024-04-22T01:17:37.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-22T01:17:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 Processor -- start 2024-04-22T01:17:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 -- start start 2024-04-22T01:17:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff41455b0 0x7fdff40cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdff4072230 0x7fdff406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff40d0250 con 0x7fdff40cc150 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdff40cd7b0 con 0x7fdff4072230 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff9e83640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdff40cc710 con 0x7fdff41455b0 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdff4072230 0x7fdff406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff2ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdff40cc150 0x7fdff4071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37828/0 (socket says 172.21.15.70:37828) 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff2ffd640 1 -- 172.21.15.70:0/2502684170 learned_addr learned my addr 172.21.15.70:0/2502684170 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff37fe640 1 --2- 172.21.15.70:0/2502684170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff41455b0 0x7fdff40cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff27fc640 1 -- 172.21.15.70:0/2502684170 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823470072 0 0) 0x7fdff40d0250 con 0x7fdff40cc150 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff27fc640 1 -- 172.21.15.70:0/2502684170 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfdc003530 con 0x7fdff40cc150 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff3fff640 1 -- 172.21.15.70:0/2502684170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff41455b0 msgr2=0x7fdff40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff3fff640 1 --2- 172.21.15.70:0/2502684170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff41455b0 0x7fdff40cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff3fff640 1 -- 172.21.15.70:0/2502684170 >> v1:172.21.15.70:6789/0 conn(0x7fdff40cc150 legacy=0x7fdff4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.929+0000 7fdff3fff640 1 -- 172.21.15.70:0/2502684170 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff40cc9f0 con 0x7fdff4072230 2024-04-22T01:17:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff3fff640 1 --2- 172.21.15.70:0/2502684170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff4072230 0x7fdff406e200 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009d90 tx=0x7fdfec01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30330e7c0f2c5895 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff27fc640 1 -- 172.21.15.70:0/2502684170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0b2070 con 0x7fdff4072230 2024-04-22T01:17:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff27fc640 1 -- 172.21.15.70:0/2502684170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfec0a5710 con 0x7fdff4072230 2024-04-22T01:17:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff27fc640 1 -- 172.21.15.70:0/2502684170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0a5ac0 con 0x7fdff4072230 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- 172.21.15.70:0/2502684170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff4072230 msgr2=0x7fdff406e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2502684170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff4072230 0x7fdff406e200 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009d90 tx=0x7fdfec01c850 comp rx=0 tx=0).stop 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- 172.21.15.70:0/2502684170 shutdown_connections 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2502684170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff4072230 0x7fdff406e200 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2502684170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff41455b0 0x7fdff40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- 172.21.15.70:0/2502684170 >> 172.21.15.70:0/2502684170 conn(0x7fdff406b1f0 msgr2=0x7fdff406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- 172.21.15.70:0/2502684170 shutdown_connections 2024-04-22T01:17:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- 172.21.15.70:0/2502684170 wait complete. 2024-04-22T01:17:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 Processor -- start 2024-04-22T01:17:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- start start 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff4072230 0x7fdff413b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 0x7fdff415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff41455b0 0x7fdff415b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdff406f2e0 con 0x7fdff40cc150 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff406f160 con 0x7fdff41455b0 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff9e83640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdff4168b40 con 0x7fdff4072230 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 0x7fdff415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 0x7fdff415ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43928/0 (socket says 172.21.15.70:43928) 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 -- 172.21.15.70:0/2511405353 learned_addr learned my addr 172.21.15.70:0/2511405353 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff37fe640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff4072230 0x7fdff413b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff3fff640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff41455b0 0x7fdff415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 -- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff4072230 msgr2=0x7fdff413b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff4072230 0x7fdff413b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 -- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff41455b0 msgr2=0x7fdff415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff41455b0 0x7fdff415b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 -- 172.21.15.70:0/2511405353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff40af7c0 con 0x7fdff40cc150 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff37fe640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff4072230 0x7fdff413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdff2ffd640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 0x7fdff415ae50 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800c3c0 tx=0x7fdfe800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8017070 con 0x7fdff40cc150 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe8005cf0 con 0x7fdff40cc150 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.933+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe80072f0 con 0x7fdff40cc150 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff40af500 con 0x7fdff40cc150 2024-04-22T01:17:37.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff4140630 con 0x7fdff40cc150 2024-04-22T01:17:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff40b0e40 con 0x7fdff40cc150 2024-04-22T01:17:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdfe801a070 con 0x7fdff40cc150 2024-04-22T01:17:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdfdbfff640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdfe00678c0 0x7fdfe0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:37.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdff37fe640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdfe00678c0 0x7fdfe0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:37.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5249+0+0 (secure 0 0 0) 0x7fdfe8084df0 con 0x7fdff40cc150 2024-04-22T01:17:37.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.937+0000 7fdff37fe640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdfe00678c0 0x7fdfe0069d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009840 tx=0x7fdfe4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:37.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:37.941+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdff40b0e40 con 0x7fdff40cc150 2024-04-22T01:17:38.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.153+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fdff4068210 con 0x7fdff40cc150 2024-04-22T01:17:38.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.153+0000 7fdfdbfff640 1 -- 172.21.15.70:0/2511405353 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7fdfe8055b80 con 0x7fdff40cc150 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw==", 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.155 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w==", 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-22T01:17:38.156 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ==", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA==", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.157 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA==", 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA==", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg==", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.159 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg==", 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-22T01:17:38.160 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong==", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ==", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA==", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ==", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:17:38.162 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw==", 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow r", 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-22T01:17:38.163 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ==", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA==", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:17:38.164 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw==", 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.165 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm2O5GBRAA/JxaogwD4Hjmtr9eQvMY0g==", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw==", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:17:38.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg==", 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-22T01:17:38.167 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:17:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdfe00678c0 msgr2=0x7fdfe0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdfe00678c0 0x7fdfe0069d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009840 tx=0x7fdfe4008040 comp rx=0 tx=0).stop 2024-04-22T01:17:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 msgr2=0x7fdff415ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 0x7fdff415ae50 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800c3c0 tx=0x7fdfe800c880 comp rx=0 tx=0).stop 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 shutdown_connections 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdfe00678c0 0x7fdfe0069d80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff41455b0 0x7fdff415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff40cc150 0x7fdff415ae50 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 --2- 172.21.15.70:0/2511405353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff4072230 0x7fdff413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 >> 172.21.15.70:0/2511405353 conn(0x7fdff406b1f0 msgr2=0x7fdff40cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 shutdown_connections 2024-04-22T01:17:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.157+0000 7fdff9e83640 1 -- 172.21.15.70:0/2511405353 wait complete. 2024-04-22T01:17:38.173 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:38.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.289+0000 7f9d522b3640 1 Processor -- start 2024-04-22T01:17:38.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.289+0000 7f9d522b3640 1 -- start start 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d522b3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9d4c15aa30 0x7f9d4c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d522b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c15bb50 0x7f9d4c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d522b3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d4c06dbd0 con 0x7f9d4c0aec70 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d522b3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4c0af0a0 con 0x7f9d4c15aa30 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d522b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4c06de70 con 0x7f9d4c15bb50 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9d4c15aa30 0x7f9d4c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9d4c15aa30 0x7f9d4c0b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43930/0 (socket says 172.21.15.70:43930) 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 -- 172.21.15.70:0/1192528629 learned_addr learned my addr 172.21.15.70:0/1192528629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4b7fe640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c15bb50 0x7f9d4c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 -- 172.21.15.70:0/1192528629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c15bb50 msgr2=0x7f9d4c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c15bb50 0x7f9d4c0ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 -- 172.21.15.70:0/1192528629 >> v1:172.21.15.98:6789/0 conn(0x7f9d4c0aec70 legacy=0x7f9d4c15e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 -- 172.21.15.70:0/1192528629 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c06e150 con 0x7f9d4c15aa30 2024-04-22T01:17:38.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4b7fe640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c15bb50 0x7f9d4c0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:38.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4bfff640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c0b4720 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009870 tx=0x7f9d3c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a50ce4d54fa699 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4affd640 1 -- 172.21.15.70:0/1192528629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0b1070 con 0x7f9d4c15aa30 2024-04-22T01:17:38.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4affd640 1 -- 172.21.15.70:0/1192528629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d3c0ac640 con 0x7f9d4c15aa30 2024-04-22T01:17:38.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.293+0000 7f9d4affd640 1 -- 172.21.15.70:0/1192528629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0aca10 con 0x7f9d4c15aa30 2024-04-22T01:17:38.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/1192528629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 msgr2=0x7f9d4c0b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c0b4720 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009870 tx=0x7f9d3c01c520 comp rx=0 tx=0).stop 2024-04-22T01:17:38.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/1192528629 shutdown_connections 2024-04-22T01:17:38.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c15bb50 0x7f9d4c0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 --2- 172.21.15.70:0/1192528629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c0b4720 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/1192528629 >> 172.21.15.70:0/1192528629 conn(0x7f9d4c06b1f0 msgr2=0x7f9d4c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:38.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/1192528629 shutdown_connections 2024-04-22T01:17:38.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/1192528629 wait complete. 2024-04-22T01:17:38.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 Processor -- start 2024-04-22T01:17:38.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- start start 2024-04-22T01:17:38.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c0aec70 0x7f9d4c14a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c14e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d4c15bb50 0x7f9d4c159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c15af90 con 0x7f9d4c15aa30 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c15ae10 con 0x7f9d4c15bb50 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4c06e950 con 0x7f9d4c0aec70 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c14e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c14e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43940/0 (socket says 172.21.15.70:43940) 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 -- 172.21.15.70:0/2404058686 learned_addr learned my addr 172.21.15.70:0/2404058686 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4bfff640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c0aec70 0x7f9d4c14a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d50850640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d4c15bb50 0x7f9d4c159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 -- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c0aec70 msgr2=0x7f9d4c14a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c0aec70 0x7f9d4c14a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 -- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d4c15bb50 msgr2=0x7f9d4c159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d4c15bb50 0x7f9d4c159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 -- 172.21.15.70:0/2404058686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c14a9e0 con 0x7f9d4c15aa30 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d50850640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d4c15bb50 0x7f9d4c159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d4b7fe640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c14e670 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000c7f0 tx=0x7f9d4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40015070 con 0x7f9d4c15aa30 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d400077c0 con 0x7f9d4c15aa30 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c14acc0 con 0x7f9d4c15aa30 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40007bb0 con 0x7f9d4c15aa30 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.297+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c14b280 con 0x7f9d4c15aa30 2024-04-22T01:17:38.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.301+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c068210 con 0x7f9d4c15aa30 2024-04-22T01:17:38.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.301+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9d40018070 con 0x7f9d4c15aa30 2024-04-22T01:17:38.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.301+0000 7f9d48ff9640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9d380678c0 0x7f9d38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.301+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5249+0+0 (secure 0 0 0) 0x7f9d40085b80 con 0x7f9d4c15aa30 2024-04-22T01:17:38.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.301+0000 7f9d4bfff640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9d380678c0 0x7f9d38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.301+0000 7f9d4bfff640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9d380678c0 0x7f9d38069d80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0ae000 tx=0x7f9d3c0a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.305+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9d40056910 con 0x7f9d4c15aa30 2024-04-22T01:17:38.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.465+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d4c06f190 con 0x7f9d4c15aa30 2024-04-22T01:17:38.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.469+0000 7f9d48ff9640 1 -- 172.21.15.70:0/2404058686 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+13129 (secure 0 0 0) 0x7f9d40056210 con 0x7f9d4c15aa30 2024-04-22T01:17:38.470 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:38.470 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":74,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:37.784096+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9d380678c0 msgr2=0x7f9d38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9d380678c0 0x7f9d38069d80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0ae000 tx=0x7f9d3c0a6070 comp rx=0 tx=0).stop 2024-04-22T01:17:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 msgr2=0x7f9d4c14e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c14e670 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000c7f0 tx=0x7f9d4000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 shutdown_connections 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9d380678c0 0x7f9d38069d80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d4c15bb50 0x7f9d4c159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d4c15aa30 0x7f9d4c14e670 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 --2- 172.21.15.70:0/2404058686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4c0aec70 0x7f9d4c14a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 >> 172.21.15.70:0/2404058686 conn(0x7f9d4c06b1f0 msgr2=0x7f9d4c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 shutdown_connections 2024-04-22T01:17:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.473+0000 7f9d522b3640 1 -- 172.21.15.70:0/2404058686 wait complete. 2024-04-22T01:17:38.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:38.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:38.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 Processor -- start 2024-04-22T01:17:38.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 -- start start 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faea015de20 0x7faea00b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 0x7faea00d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faea00b75e0 con 0x7faea00cea10 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faea00b78c0 con 0x7faea015de20 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea6f73640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faea00b7ba0 con 0x7faea00b6db0 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea5510640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7faea00cea10 0x7faea00ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37844/0 (socket says 172.21.15.70:37844) 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea5510640 1 -- 172.21.15.70:0/705372351 learned_addr learned my addr 172.21.15.70:0/705372351 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9ffff640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 0x7faea00d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 --2- 172.21.15.70:0/705372351 >> v2:172.21.15.98:3300/0 conn(0x7faea015de20 0x7faea00b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515616303 0 0) 0x7faea00b75e0 con 0x7faea00cea10 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae88003530 con 0x7faea00cea10 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 209606691 0 0) 0x7fae88003530 con 0x7faea00cea10 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 msgr2=0x7faea00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 0x7faea00d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 >> v2:172.21.15.98:3300/0 conn(0x7faea015de20 msgr2=0x7faea00b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 --2- 172.21.15.70:0/705372351 >> v2:172.21.15.98:3300/0 conn(0x7faea015de20 0x7faea00b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea00b7e80 con 0x7faea00cea10 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fae98001a00 con 0x7faea00cea10 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 >> v1:172.21.15.70:6789/0 conn(0x7faea00cea10 legacy=0x7faea00ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae880039c0 0x7fae88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae88004810 0x7fae88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 0x7fae880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faea00b78c0 con 0x7fae880039c0 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faea00b7ba0 con 0x7fae88004810 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fae8800b9d0 con 0x7faea00b6db0 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae880039c0 0x7fae88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7fae9ffff640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae88004810 0x7fae88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea5510640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 0x7fae880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 -- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 msgr2=0x7fae880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00b6db0 0x7fae880051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 -- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae88004810 msgr2=0x7fae88004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae88004810 0x7fae88004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.613+0000 7faea4d0f640 1 -- 172.21.15.70:0/705372351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae88003530 con 0x7fae880039c0 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea4d0f640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae880039c0 0x7fae88003de0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ae0 tx=0x7fae9001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a4833c5ab754df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae900b2070 con 0x7fae880039c0 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae900a55c0 con 0x7fae880039c0 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7fae9f7fe640 1 -- 172.21.15.70:0/705372351 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae900096b0 con 0x7fae880039c0 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- 172.21.15.70:0/705372351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae880039c0 msgr2=0x7fae88003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 --2- 172.21.15.70:0/705372351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae880039c0 0x7fae88003de0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ae0 tx=0x7fae9001cab0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea5510640 1 -- 172.21.15.70:0/705372351 reap_dead start 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- 172.21.15.70:0/705372351 shutdown_connections 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- 172.21.15.70:0/705372351 >> 172.21.15.70:0/705372351 conn(0x7faea0072d20 msgr2=0x7faea0073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- 172.21.15.70:0/705372351 shutdown_connections 2024-04-22T01:17:38.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- 172.21.15.70:0/705372351 wait complete. 2024-04-22T01:17:38.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 Processor -- start 2024-04-22T01:17:38.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- start start 2024-04-22T01:17:38.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae880039c0 0x7faea0152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00c73d0 0x7faea00c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 0x7faea00ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faea00b71f0 con 0x7faea00cc740 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faea015e200 con 0x7fae880039c0 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea6f73640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faea00b7370 con 0x7faea00c73d0 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.617+0000 7faea5510640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 0x7faea00ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 0x7faea00ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43954/0 (socket says 172.21.15.70:43954) 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 -- 172.21.15.70:0/4222701383 learned_addr learned my addr 172.21.15.70:0/4222701383 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea4d0f640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae880039c0 0x7faea0152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9ffff640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00c73d0 0x7faea00c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 -- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00c73d0 msgr2=0x7faea00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00c73d0 0x7faea00c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 -- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae880039c0 msgr2=0x7faea0152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae880039c0 0x7faea0152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 -- 172.21.15.70:0/4222701383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea00c7e90 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea5510640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 0x7faea00ccbc0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7faea0161900 tx=0x7fae98005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9801a070 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae98011040 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea00c8170 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9800ed80 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea0160e80 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea0068210 con 0x7faea00cc740 2024-04-22T01:17:38.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fae9801d070 con 0x7faea00cc740 2024-04-22T01:17:38.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9d7fa640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fae8c0678f0 0x7fae8c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5249+0+0 (secure 0 0 0) 0x7fae98088260 con 0x7faea00cc740 2024-04-22T01:17:38.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea4d0f640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fae8c0678f0 0x7fae8c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.621+0000 7faea4d0f640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fae8c0678f0 0x7fae8c069db0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fae900af000 tx=0x7fae900046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.625+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fae98058f60 con 0x7faea00cc740 2024-04-22T01:17:38.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.773+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faea00d1af0 con 0x7faea00cc740 2024-04-22T01:17:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.773+0000 7fae9d7fa640 1 -- 172.21.15.70:0/4222701383 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7fae98088080 con 0x7faea00cc740 2024-04-22T01:17:38.776 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fae8c0678f0 msgr2=0x7fae8c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fae8c0678f0 0x7fae8c069db0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fae900af000 tx=0x7fae900046d0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 msgr2=0x7faea00ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 0x7faea00ccbc0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7faea0161900 tx=0x7fae98005ac0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 shutdown_connections 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fae8c0678f0 0x7fae8c069db0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faea00cc740 0x7faea00ccbc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea00c73d0 0x7faea00c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 --2- 172.21.15.70:0/4222701383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae880039c0 0x7faea0152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 >> 172.21.15.70:0/4222701383 conn(0x7faea0072d20 msgr2=0x7faea00ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:38.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 shutdown_connections 2024-04-22T01:17:38.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.777+0000 7faea6f73640 1 -- 172.21.15.70:0/4222701383 wait complete. 2024-04-22T01:17:38.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:17:38.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.909+0000 7f6a42c41640 1 Processor -- start 2024-04-22T01:17:38.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.909+0000 7f6a42c41640 1 -- start start 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a42c41640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a3c0cea80 0x7f6a3c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a42c41640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c0d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a42c41640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c15e390 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a42c41640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6a3c0cf420 con 0x7f6a3c0cea80 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a42c41640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3c161e00 con 0x7f6a3c0d2110 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a3c0cea80 0x7f6a3c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a3c0cea80 0x7f6a3c0ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43962/0 (socket says 172.21.15.70:43962) 2024-04-22T01:17:38.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 -- 172.21.15.70:0/458076944 learned_addr learned my addr 172.21.15.70:0/458076944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 -- 172.21.15.70:0/458076944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 msgr2=0x7f6a3c0d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a411de640 1 --2- 172.21.15.70:0/458076944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c0d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 --2- 172.21.15.70:0/458076944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c0d25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 -- 172.21.15.70:0/458076944 >> v1:172.21.15.98:6789/0 conn(0x7f6a3c15dfb0 legacy=0x7f6a3c0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:38.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 -- 172.21.15.70:0/458076944 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c1620e0 con 0x7f6a3c0cea80 2024-04-22T01:17:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3bfff640 1 --2- 172.21.15.70:0/458076944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0ceee0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009870 tx=0x7f6a3001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fc87599e42a812 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3b7fe640 1 -- 172.21.15.70:0/458076944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300ab5f0 con 0x7f6a3c0cea80 2024-04-22T01:17:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3b7fe640 1 -- 172.21.15.70:0/458076944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a300b0070 con 0x7f6a3c0cea80 2024-04-22T01:17:38.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.913+0000 7f6a3b7fe640 1 -- 172.21.15.70:0/458076944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300ab9b0 con 0x7f6a3c0cea80 2024-04-22T01:17:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/458076944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c0cea80 msgr2=0x7f6a3c0ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 --2- 172.21.15.70:0/458076944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0ceee0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009870 tx=0x7f6a3001c520 comp rx=0 tx=0).stop 2024-04-22T01:17:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/458076944 shutdown_connections 2024-04-22T01:17:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 --2- 172.21.15.70:0/458076944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c0d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 --2- 172.21.15.70:0/458076944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0ceee0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/458076944 >> 172.21.15.70:0/458076944 conn(0x7f6a3c06c9b0 msgr2=0x7f6a3c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/458076944 shutdown_connections 2024-04-22T01:17:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/458076944 wait complete. 2024-04-22T01:17:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 Processor -- start 2024-04-22T01:17:38.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- start start 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 0x7f6a3c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c16af50 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c16add0 con 0x7f6a3c0cea80 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3c16b0d0 con 0x7f6a3c0d2110 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 0x7f6a3c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 0x7f6a3c156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43976/0 (socket says 172.21.15.70:43976) 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 -- 172.21.15.70:0/2802859090 learned_addr learned my addr 172.21.15.70:0/2802859090 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a3bfff640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a409dd640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 -- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 msgr2=0x7f6a3c143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 -- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a3c0cea80 msgr2=0x7f6a3c0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 -- 172.21.15.70:0/2802859090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c1550e0 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a411de640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 0x7f6a3c156f30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400cc60 tx=0x7f6a3400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340039a0 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a34003b40 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c1facf0 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34004020 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.917+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c1fb2b0 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.921+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3c068210 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.921+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6a340041c0 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.921+0000 7f6a397fa640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a280678c0 0x7f6a28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:38.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.921+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5249+0+0 (secure 0 0 0) 0x7f6a34085f00 con 0x7f6a3c15dfb0 2024-04-22T01:17:38.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.921+0000 7f6a409dd640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a280678c0 0x7f6a28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:38.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.921+0000 7f6a409dd640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a280678c0 0x7f6a28069d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00b810 tx=0x7f6a2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:38.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:38.925+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6a34014030 con 0x7f6a3c15dfb0 2024-04-22T01:17:39.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6a3c0b2f30 con 0x7f6a3c15dfb0 2024-04-22T01:17:39.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a397fa640 1 -- 172.21.15.70:0/2802859090 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) ==== 84+0+65 (secure 0 0 0) 0x7f6a34056c00 con 0x7f6a3c15dfb0 2024-04-22T01:17:39.090 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:39.090 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:17:39.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:17:39.091 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:17:39.091 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:17:39.091 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:39.091 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:17:39.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a280678c0 msgr2=0x7f6a28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a280678c0 0x7f6a28069d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00b810 tx=0x7f6a2c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 msgr2=0x7f6a3c156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 0x7f6a3c156f30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400cc60 tx=0x7f6a3400a5c0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 shutdown_connections 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a280678c0 0x7f6a28069d80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c15dfb0 0x7f6a3c156f30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a3c0d2110 0x7f6a3c143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 --2- 172.21.15.70:0/2802859090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a3c0cea80 0x7f6a3c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 >> 172.21.15.70:0/2802859090 conn(0x7f6a3c06c9b0 msgr2=0x7f6a3c0b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 shutdown_connections 2024-04-22T01:17:39.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.089+0000 7f6a42c41640 1 -- 172.21.15.70:0/2802859090 wait complete. 2024-04-22T01:17:39.109 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:17:39.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:17:39.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 Processor -- start 2024-04-22T01:17:39.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 -- start start 2024-04-22T01:17:39.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15dfc0 0x7f887c0b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f887c15e3a0 0x7f887c0b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f887c0d29c0 con 0x7f887c0d3a60 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f887c0b7f90 con 0x7f887c15e3a0 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f88819fd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f887c0d2ca0 con 0x7f887c15dfc0 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f887c15e3a0 0x7f887c0b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f887c15e3a0 0x7f887c0b58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43990/0 (socket says 172.21.15.70:43990) 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 -- 172.21.15.70:0/2782712264 learned_addr learned my addr 172.21.15.70:0/2782712264 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887affd640 1 --2- 172.21.15.70:0/2782712264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15dfc0 0x7f887c0b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 -- 172.21.15.70:0/2782712264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15dfc0 msgr2=0x7f887c0b9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 --2- 172.21.15.70:0/2782712264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15dfc0 0x7f887c0b9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 -- 172.21.15.70:0/2782712264 >> v1:172.21.15.70:6789/0 conn(0x7f887c0d3a60 legacy=0x7f887c0d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f887b7fe640 1 -- 172.21.15.70:0/2782712264 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c0d2f80 con 0x7f887c15e3a0 2024-04-22T01:17:39.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.233+0000 7f8879ffb640 1 -- 172.21.15.70:0/2782712264 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127097675 0 0) 0x7f887c0d29c0 con 0x7f887c0d3a60 2024-04-22T01:17:39.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f887b7fe640 1 --2- 172.21.15.70:0/2782712264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15e3a0 0x7f887c0b58d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f88740098a0 tx=0x7f887401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa35ec05eb6994f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:39.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f8879ffb640 1 -- 172.21.15.70:0/2782712264 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88740ab620 con 0x7f887c15e3a0 2024-04-22T01:17:39.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f8879ffb640 1 -- 172.21.15.70:0/2782712264 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88740b0070 con 0x7f887c15e3a0 2024-04-22T01:17:39.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f8879ffb640 1 -- 172.21.15.70:0/2782712264 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88740ab9e0 con 0x7f887c15e3a0 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- 172.21.15.70:0/2782712264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15e3a0 msgr2=0x7f887c0b58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 --2- 172.21.15.70:0/2782712264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15e3a0 0x7f887c0b58d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f88740098a0 tx=0x7f887401c510 comp rx=0 tx=0).stop 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- 172.21.15.70:0/2782712264 shutdown_connections 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 --2- 172.21.15.70:0/2782712264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15e3a0 0x7f887c0b58d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 --2- 172.21.15.70:0/2782712264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15dfc0 0x7f887c0b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- 172.21.15.70:0/2782712264 >> 172.21.15.70:0/2782712264 conn(0x7f887c0d08d0 msgr2=0x7f887c0d0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- 172.21.15.70:0/2782712264 shutdown_connections 2024-04-22T01:17:39.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- 172.21.15.70:0/2782712264 wait complete. 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 Processor -- start 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- start start 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 0x7f887c1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15e3a0 0x7f887c155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f887c0ccc40 0x7f887c0d3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f887c0b7b20 con 0x7f887c15dfc0 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f887c0b79a0 con 0x7f887c0ccc40 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f88819fd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f887c0b7ca0 con 0x7f887c15e3a0 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f887affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 0x7f887c1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f887affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 0x7f887c1571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44002/0 (socket says 172.21.15.70:44002) 2024-04-22T01:17:39.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.237+0000 7f887affd640 1 -- 172.21.15.70:0/654079967 learned_addr learned my addr 172.21.15.70:0/654079967 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887affd640 1 -- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15e3a0 msgr2=0x7f887c155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887affd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15e3a0 0x7f887c155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887affd640 1 -- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f887c0ccc40 msgr2=0x7f887c0d3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887b7fe640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f887c0ccc40 0x7f887c0d3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887affd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f887c0ccc40 0x7f887c0d3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887affd640 1 -- 172.21.15.70:0/654079967 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c0cd170 con 0x7f887c15dfc0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887affd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 0x7f887c1571b0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f886c007f20 tx=0x7f886c00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c014620 con 0x7f887c15dfc0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f886c019070 con 0x7f887c15dfc0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887c156910 con 0x7f887c15dfc0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c014a00 con 0x7f887c15dfc0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887c156ed0 con 0x7f887c15dfc0 2024-04-22T01:17:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887c068210 con 0x7f887c15dfc0 2024-04-22T01:17:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f886c014ba0 con 0x7f887c15dfc0 2024-04-22T01:17:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f8863fff640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88680678c0 0x7f8868069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5249+0+0 (secure 0 0 0) 0x7f886c086450 con 0x7f887c15dfc0 2024-04-22T01:17:39.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887a7fc640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88680678c0 0x7f8868069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.241+0000 7f887a7fc640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88680678c0 0x7f8868069d80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f887c0c7dc0 tx=0x7f8870007870 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:39.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.245+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f886c057150 con 0x7f887c15dfc0 2024-04-22T01:17:39.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.393+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f887c155a50 con 0x7f887c15dfc0 2024-04-22T01:17:39.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.793+0000 7f8863fff640 1 -- 172.21.15.70:0/654079967 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v75) ==== 138+0+0 (secure 0 0 0) 0x7f886c056a50 con 0x7f887c15dfc0 2024-04-22T01:17:39.797 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88680678c0 msgr2=0x7f8868069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88680678c0 0x7f8868069d80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f887c0c7dc0 tx=0x7f8870007870 comp rx=0 tx=0).stop 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 msgr2=0x7f887c1571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 0x7f887c1571b0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f886c007f20 tx=0x7f886c00e570 comp rx=0 tx=0).stop 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 shutdown_connections 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f88680678c0 0x7f8868069d80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f887c0ccc40 0x7f887c0d3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f887c15e3a0 0x7f887c155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 --2- 172.21.15.70:0/654079967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f887c15dfc0 0x7f887c1571b0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 >> 172.21.15.70:0/654079967 conn(0x7f887c0d08d0 msgr2=0x7f887c15d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 shutdown_connections 2024-04-22T01:17:39.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.797+0000 7f88819fd640 1 -- 172.21.15.70:0/654079967 wait complete. 2024-04-22T01:17:39.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:17:39.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 Processor -- start 2024-04-22T01:17:39.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 -- start start 2024-04-22T01:17:39.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f038c15e6e0 0x7f038c15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 0x7f038c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f038c15f000 con 0x7f038c0ceda0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f038c0b3800 con 0x7f038c15e6e0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.953+0000 7f0392c4d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f038c0b3ae0 con 0x7f038c0ce920 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f03911ea640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f038c0ceda0 0x7f038c06d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37852/0 (socket says 172.21.15.70:37852) 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f03911ea640 1 -- 172.21.15.70:0/111681784 learned_addr learned my addr 172.21.15.70:0/111681784 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 0x7f038c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f03909e9640 1 --2- 172.21.15.70:0/111681784 >> v2:172.21.15.98:3300/0 conn(0x7f038c15e6e0 0x7f038c15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841866788 0 0) 0x7f038c15f000 con 0x7f038c0ceda0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0374003530 con 0x7f038c0ceda0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1926245794 0 0) 0x7f0374003530 con 0x7f038c0ceda0 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 msgr2=0x7f038c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 0x7f038c154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 >> v2:172.21.15.98:3300/0 conn(0x7f038c15e6e0 msgr2=0x7f038c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 --2- 172.21.15.70:0/111681784 >> v2:172.21.15.98:3300/0 conn(0x7f038c15e6e0 0x7f038c15eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c0cde40 con 0x7f038c0ceda0 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f0384001a00 con 0x7f038c0ceda0 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 >> v1:172.21.15.70:6789/0 conn(0x7f038c0ceda0 legacy=0x7f038c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03740039c0 0x7f0374003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0374004810 0x7f0374004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 0x7f03740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0b3800 con 0x7f0374004810 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0b3ae0 con 0x7f03740039c0 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f037400b9d0 con 0x7f038c0ce920 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0374004810 0x7f0374004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f03909e9640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03740039c0 0x7f0374003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f03911ea640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 0x7f03740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 -- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 msgr2=0x7f03740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c0ce920 0x7f03740051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 -- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03740039c0 msgr2=0x7f0374003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:39.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03740039c0 0x7f0374003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 -- 172.21.15.70:0/111681784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0374003530 con 0x7f0374004810 2024-04-22T01:17:39.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f03909e9640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03740039c0 0x7f0374003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038bfff640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0374004810 0x7f0374004c90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f0380009670 tx=0x7f038001eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff26ae508c928ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03800b4070 con 0x7f0374004810 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03800a76b0 con 0x7f0374004810 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.957+0000 7f038b7fe640 1 -- 172.21.15.70:0/111681784 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03800096b0 con 0x7f0374004810 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/111681784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0374004810 msgr2=0x7f0374004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 --2- 172.21.15.70:0/111681784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0374004810 0x7f0374004c90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f0380009670 tx=0x7f038001eca0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 -- 172.21.15.70:0/111681784 reap_dead start 2024-04-22T01:17:39.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/111681784 shutdown_connections 2024-04-22T01:17:39.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/111681784 >> 172.21.15.70:0/111681784 conn(0x7f038c0b1d40 msgr2=0x7f038c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:39.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/111681784 shutdown_connections 2024-04-22T01:17:39.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/111681784 wait complete. 2024-04-22T01:17:39.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 Processor -- start 2024-04-22T01:17:39.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- start start 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0374004810 0x7f038c14da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c150040 0x7f038c14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 0x7f038c1541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f038c06f0e0 con 0x7f038c153d00 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0ce400 con 0x7f0374004810 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f038c06f260 con 0x7f038c150040 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 0x7f038c1541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 0x7f038c1541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44014/0 (socket says 172.21.15.70:44014) 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 -- 172.21.15.70:0/1250239665 learned_addr learned my addr 172.21.15.70:0/1250239665 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03909e9640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0374004810 0x7f038c14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f038bfff640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c150040 0x7f038c14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 -- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c150040 msgr2=0x7f038c14c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c150040 0x7f038c14c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 -- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0374004810 msgr2=0x7f038c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0374004810 0x7f038c14da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 -- 172.21.15.70:0/1250239665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c153350 con 0x7f038c153d00 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03911ea640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 0x7f038c1541b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f038c072e20 tx=0x7f0384011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384010040 con 0x7f038c153d00 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f038401a070 con 0x7f038c153d00 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038c153630 con 0x7f038c153d00 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038400faa0 con 0x7f038c153d00 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.961+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f038c14d2e0 con 0x7f038c153d00 2024-04-22T01:17:39.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.965+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f038c068210 con 0x7f038c153d00 2024-04-22T01:17:39.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.965+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f038400fc40 con 0x7f038c153d00 2024-04-22T01:17:39.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.965+0000 7f03897fa640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03780678f0 0x7f0378069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:39.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.965+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5619+0+0 (secure 0 0 0) 0x7f0384089420 con 0x7f038c153d00 2024-04-22T01:17:39.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.965+0000 7f03909e9640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03780678f0 0x7f0378069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:39.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.965+0000 7f03909e9640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03780678f0 0x7f0378069db0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f038c156890 tx=0x7f037c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:39.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:39.969+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f038405a120 con 0x7f038c153d00 2024-04-22T01:17:40.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.129+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f038c1565e0 con 0x7f038c153d00 2024-04-22T01:17:40.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.797+0000 7f03897fa640 1 -- 172.21.15.70:0/1250239665 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f0384059a20 con 0x7f038c153d00 2024-04-22T01:17:40.801 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:17:40.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03780678f0 msgr2=0x7f0378069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:40.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03780678f0 0x7f0378069db0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f038c156890 tx=0x7f037c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:40.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 msgr2=0x7f038c1541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:40.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 0x7f038c1541b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f038c072e20 tx=0x7f0384011c20 comp rx=0 tx=0).stop 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 shutdown_connections 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f03780678f0 0x7f0378069db0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f038c153d00 0x7f038c1541b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f038c150040 0x7f038c14c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 --2- 172.21.15.70:0/1250239665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0374004810 0x7f038c14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.801+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 >> 172.21.15.70:0/1250239665 conn(0x7f038c0b1d40 msgr2=0x7f038c15eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.805+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 shutdown_connections 2024-04-22T01:17:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.805+0000 7f0392c4d640 1 -- 172.21.15.70:0/1250239665 wait complete. 2024-04-22T01:17:40.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:17:40.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.965+0000 7fd1e6edc640 1 Processor -- start 2024-04-22T01:17:40.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.965+0000 7fd1e6edc640 1 -- start start 2024-04-22T01:17:40.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1e00ce920 0x7fd1e0154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:40.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1e00cedd0 0x7fd1e01551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:40.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00b3800 con 0x7fd1e015e6e0 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e00b3ae0 con 0x7fd1e00ce920 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e00cde40 con 0x7fd1e00cedd0 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1e00ce920 0x7fd1e0154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1e00ce920 0x7fd1e0154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44016/0 (socket says 172.21.15.70:44016) 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 -- 172.21.15.70:0/4081339114 learned_addr learned my addr 172.21.15.70:0/4081339114 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 -- 172.21.15.70:0/4081339114 >> v2:172.21.15.98:3300/0 conn(0x7fd1e00cedd0 msgr2=0x7fd1e01551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e5479640 1 --2- 172.21.15.70:0/4081339114 >> v2:172.21.15.98:3300/0 conn(0x7fd1e00cedd0 0x7fd1e01551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 --2- 172.21.15.70:0/4081339114 >> v2:172.21.15.98:3300/0 conn(0x7fd1e00cedd0 0x7fd1e01551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 -- 172.21.15.70:0/4081339114 >> v1:172.21.15.98:6789/0 conn(0x7fd1e015e6e0 legacy=0x7fd1e015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:40.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 -- 172.21.15.70:0/4081339114 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00ce120 con 0x7fd1e00ce920 2024-04-22T01:17:40.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1dffff640 1 --2- 172.21.15.70:0/4081339114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00ce920 0x7fd1e0154c60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0158cb0 tx=0x7fd1d401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f5930d453ccfea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:40.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1df7fe640 1 -- 172.21.15.70:0/4081339114 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40aa430 con 0x7fd1e00ce920 2024-04-22T01:17:40.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1df7fe640 1 -- 172.21.15.70:0/4081339114 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d40af070 con 0x7fd1e00ce920 2024-04-22T01:17:40.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1df7fe640 1 -- 172.21.15.70:0/4081339114 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40aa7f0 con 0x7fd1e00ce920 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 -- 172.21.15.70:0/4081339114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00ce920 msgr2=0x7fd1e0154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/4081339114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00ce920 0x7fd1e0154c60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0158cb0 tx=0x7fd1d401c4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 -- 172.21.15.70:0/4081339114 shutdown_connections 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/4081339114 >> v2:172.21.15.98:3300/0 conn(0x7fd1e00cedd0 0x7fd1e01551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/4081339114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00ce920 0x7fd1e0154c60 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.969+0000 7fd1e6edc640 1 -- 172.21.15.70:0/4081339114 >> 172.21.15.70:0/4081339114 conn(0x7fd1e00b1d40 msgr2=0x7fd1e00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- 172.21.15.70:0/4081339114 shutdown_connections 2024-04-22T01:17:40.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- 172.21.15.70:0/4081339114 wait complete. 2024-04-22T01:17:40.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 Processor -- start 2024-04-22T01:17:40.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- start start 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e00ce920 0x7fd1e00b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 0x7fd1e01569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e015e6e0 0x7fd1e013dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b2e80 con 0x7fd1e00cedd0 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b2d00 con 0x7fd1e015e6e0 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00b3000 con 0x7fd1e00ce920 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 0x7fd1e01569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 0x7fd1e01569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44024/0 (socket says 172.21.15.70:44024) 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 -- 172.21.15.70:0/1904993273 learned_addr learned my addr 172.21.15.70:0/1904993273 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e4c78640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e00ce920 0x7fd1e00b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:40.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e5479640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e015e6e0 0x7fd1e013dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 -- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e00ce920 msgr2=0x7fd1e00b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e00ce920 0x7fd1e00b0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 -- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e015e6e0 msgr2=0x7fd1e013dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e015e6e0 0x7fd1e013dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 -- 172.21.15.70:0/1904993273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00b0a50 con 0x7fd1e00cedd0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e5479640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e015e6e0 0x7fd1e013dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dffff640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 0x7fd1e01569e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40ac040 tx=0x7fd1d401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40b3800 con 0x7fd1e00cedd0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d40b39c0 con 0x7fd1e00cedd0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e00b0d30 con 0x7fd1e00cedd0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40b3db0 con 0x7fd1e00cedd0 2024-04-22T01:17:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e00b12f0 con 0x7fd1e00cedd0 2024-04-22T01:17:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.973+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e0068210 con 0x7fd1e00cedd0 2024-04-22T01:17:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.977+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd1d40c6390 con 0x7fd1e00cedd0 2024-04-22T01:17:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.977+0000 7fd1dd7fa640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1cc0678c0 0x7fd1cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:40.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.977+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6001+0+0 (secure 0 0 0) 0x7fd1d4120340 con 0x7fd1e00cedd0 2024-04-22T01:17:40.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.977+0000 7fd1e4c78640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1cc0678c0 0x7fd1cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:40.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.977+0000 7fd1e4c78640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1cc0678c0 0x7fd1cc069d80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00b1660 tx=0x7fd1d0009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:40.981+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd1d40c3060 con 0x7fd1e00cedd0 2024-04-22T01:17:41.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.169+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd1e00d08e0 con 0x7fd1e00cedd0 2024-04-22T01:17:41.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.869+0000 7fd1dd7fa640 1 -- 172.21.15.70:0/1904993273 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v35) ==== 174+0+0 (secure 0 0 0) 0x7fd1d40160d0 con 0x7fd1e00cedd0 2024-04-22T01:17:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 11 and data pool 12 2024-04-22T01:17:41.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1cc0678c0 msgr2=0x7fd1cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:41.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1cc0678c0 0x7fd1cc069d80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00b1660 tx=0x7fd1d0009060 comp rx=0 tx=0).stop 2024-04-22T01:17:41.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 msgr2=0x7fd1e01569e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:41.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 0x7fd1e01569e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40ac040 tx=0x7fd1d401acf0 comp rx=0 tx=0).stop 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 shutdown_connections 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd1cc0678c0 0x7fd1cc069d80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e015e6e0 0x7fd1e013dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1e00cedd0 0x7fd1e01569e0 secure :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40ac040 tx=0x7fd1d401acf0 comp rx=0 tx=0).stop 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 --2- 172.21.15.70:0/1904993273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1e00ce920 0x7fd1e00b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.881+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 >> 172.21.15.70:0/1904993273 conn(0x7fd1e00b1d40 msgr2=0x7fd1e00b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.885+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 shutdown_connections 2024-04-22T01:17:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:41.885+0000 7fd1e6edc640 1 -- 172.21.15.70:0/1904993273 wait complete. 2024-04-22T01:17:41.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:42.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 Processor -- start 2024-04-22T01:17:42.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 -- start start 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788415acc0 0x7f788415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f788406bd70 0x7f788406c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f788406c710 0x7f788406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f788415b5e0 con 0x7f788406c710 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f788406cb70 con 0x7f788406bd70 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788bb08640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f78840af6e0 con 0x7f788415acc0 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f788406c710 0x7f788406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f788406c710 0x7f788406ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44036/0 (socket says 172.21.15.70:44036) 2024-04-22T01:17:42.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 -- 172.21.15.70:0/2275907421 learned_addr learned my addr 172.21.15.70:0/2275907421 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:42.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 -- 172.21.15.70:0/2275907421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788415acc0 msgr2=0x7f788415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:42.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 --2- 172.21.15.70:0/2275907421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788415acc0 0x7f788415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 -- 172.21.15.70:0/2275907421 >> v2:172.21.15.98:3300/0 conn(0x7f788406bd70 msgr2=0x7f788406c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:42.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 --2- 172.21.15.70:0/2275907421 >> v2:172.21.15.98:3300/0 conn(0x7f788406bd70 0x7f788406c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.029+0000 7f788a0a5640 1 -- 172.21.15.70:0/2275907421 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78840af9c0 con 0x7f788406c710 2024-04-22T01:17:42.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788a0a5640 1 --2- 172.21.15.70:0/2275907421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f788406ef50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f788406dc80 tx=0x7f787c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc766cc5b3fc261 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78888a2640 1 -- 172.21.15.70:0/2275907421 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c0a5840 con 0x7f788406c710 2024-04-22T01:17:42.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78888a2640 1 -- 172.21.15.70:0/2275907421 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f787c0a59e0 con 0x7f788406c710 2024-04-22T01:17:42.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78888a2640 1 -- 172.21.15.70:0/2275907421 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c0a5db0 con 0x7f788406c710 2024-04-22T01:17:42.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- 172.21.15.70:0/2275907421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 msgr2=0x7f788406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- 172.21.15.70:0/2275907421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f788406ef50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f788406dc80 tx=0x7f787c01c9c0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- 172.21.15.70:0/2275907421 shutdown_connections 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- 172.21.15.70:0/2275907421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f788406ef50 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- 172.21.15.70:0/2275907421 >> v2:172.21.15.98:3300/0 conn(0x7f788406bd70 0x7f788406c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- 172.21.15.70:0/2275907421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788415acc0 0x7f788415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- 172.21.15.70:0/2275907421 >> 172.21.15.70:0/2275907421 conn(0x7f78840b3d90 msgr2=0x7f78840b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- 172.21.15.70:0/2275907421 shutdown_connections 2024-04-22T01:17:42.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- 172.21.15.70:0/2275907421 wait complete. 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 Processor -- start 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- start start 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788406bd70 0x7f788414c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f7884150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f788415acc0 0x7f788414e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f788406fe20 con 0x7f788406c710 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f788406fca0 con 0x7f788415acc0 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f788bb08640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f788406ffa0 con 0x7f788406bd70 2024-04-22T01:17:42.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f7884150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f7884150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44042/0 (socket says 172.21.15.70:44042) 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 -- 172.21.15.70:0/3954666047 learned_addr learned my addr 172.21.15.70:0/3954666047 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 -- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788406bd70 msgr2=0x7f788414c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78898a4640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788406bd70 0x7f788414c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788406bd70 0x7f788414c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 -- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f788415acc0 msgr2=0x7f788414e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f788415acc0 0x7f788414e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 -- 172.21.15.70:0/3954666047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788414ca40 con 0x7f788406c710 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.033+0000 7f78890a3640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f7884150ee0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f787400d990 tx=0x7f787400de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787401b070 con 0x7f788406c710 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f787400edc0 con 0x7f788406c710 2024-04-22T01:17:42.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788414cd20 con 0x7f788406c710 2024-04-22T01:17:42.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874016680 con 0x7f788406c710 2024-04-22T01:17:42.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788414d2e0 con 0x7f788406c710 2024-04-22T01:17:42.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788414d620 con 0x7f788406c710 2024-04-22T01:17:42.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f787401e070 con 0x7f788406c710 2024-04-22T01:17:42.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f787a7fc640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f78680678c0 0x7f7868069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f78898a4640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f78680678c0 0x7f7868069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6101+0+0 (secure 0 0 0) 0x7f7874087c80 con 0x7f788406c710 2024-04-22T01:17:42.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.037+0000 7f78898a4640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f78680678c0 0x7f7868069d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f788000a790 tx=0x7f7880008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.041+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f788414d620 con 0x7f788406c710 2024-04-22T01:17:42.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.201+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7884068920 con 0x7f788406c710 2024-04-22T01:17:42.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.205+0000 7f787a7fc640 1 -- 172.21.15.70:0/3954666047 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+16922 (secure 0 0 0) 0x7f7874058630 con 0x7f788406c710 2024-04-22T01:17:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:42.207 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":77,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:41.173616+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:42.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f78680678c0 msgr2=0x7f7868069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f78680678c0 0x7f7868069d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f788000a790 tx=0x7f7880008040 comp rx=0 tx=0).stop 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 msgr2=0x7f7884150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f7884150ee0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f787400d990 tx=0x7f787400de50 comp rx=0 tx=0).stop 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 shutdown_connections 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f78680678c0 0x7f7868069d80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f788415acc0 0x7f788414e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f788406c710 0x7f7884150ee0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 --2- 172.21.15.70:0/3954666047 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f788406bd70 0x7f788414c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 >> 172.21.15.70:0/3954666047 conn(0x7f78840b3d90 msgr2=0x7f7884159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 shutdown_connections 2024-04-22T01:17:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.209+0000 7f788bb08640 1 -- 172.21.15.70:0/3954666047 wait complete. 2024-04-22T01:17:42.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 Processor -- start 2024-04-22T01:17:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 -- start start 2024-04-22T01:17:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f706815dfb0 0x7f70680b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f70680ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706815e390 con 0x7f70680b5730 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f70680cf420 con 0x7f706815dfb0 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.345+0000 7f706f85d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7068161e00 con 0x7f70680cea80 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f706815dfb0 0x7f70680b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f706815dfb0 0x7f70680b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44056/0 (socket says 172.21.15.70:44056) 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 -- 172.21.15.70:0/1041065589 learned_addr learned my addr 172.21.15.70:0/1041065589 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706cdf8640 1 --2- 172.21.15.70:0/1041065589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f70680ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 -- 172.21.15.70:0/1041065589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 msgr2=0x7f70680ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 --2- 172.21.15.70:0/1041065589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f70680ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 -- 172.21.15.70:0/1041065589 >> v1:172.21.15.98:6789/0 conn(0x7f70680b5730 legacy=0x7f70680b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 -- 172.21.15.70:0/1041065589 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70681620e0 con 0x7f706815dfb0 2024-04-22T01:17:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706d5f9640 1 --2- 172.21.15.70:0/1041065589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f706815dfb0 0x7f70680b95d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7064009870 tx=0x7f706401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158a86ba9cfcb928 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f705ffff640 1 -- 172.21.15.70:0/1041065589 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70640ab5f0 con 0x7f706815dfb0 2024-04-22T01:17:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f705ffff640 1 -- 172.21.15.70:0/1041065589 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70640b0070 con 0x7f706815dfb0 2024-04-22T01:17:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f705ffff640 1 -- 172.21.15.70:0/1041065589 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70640ab9b0 con 0x7f706815dfb0 2024-04-22T01:17:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 -- 172.21.15.70:0/1041065589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f706815dfb0 msgr2=0x7f70680b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 --2- 172.21.15.70:0/1041065589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f706815dfb0 0x7f70680b95d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7064009870 tx=0x7f706401c520 comp rx=0 tx=0).stop 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 -- 172.21.15.70:0/1041065589 shutdown_connections 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 --2- 172.21.15.70:0/1041065589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f70680ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 --2- 172.21.15.70:0/1041065589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f706815dfb0 0x7f70680b95d0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 -- 172.21.15.70:0/1041065589 >> 172.21.15.70:0/1041065589 conn(0x7f706806c9b0 msgr2=0x7f706806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 -- 172.21.15.70:0/1041065589 shutdown_connections 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.349+0000 7f706f85d640 1 -- 172.21.15.70:0/1041065589 wait complete. 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 Processor -- start 2024-04-22T01:17:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- start start 2024-04-22T01:17:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 0x7f706815ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f7068156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706815dfb0 0x7f70681551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f706816b5e0 con 0x7f70680b5730 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7068161a90 con 0x7f706815dfb0 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f706816b760 con 0x7f70680cea80 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 0x7f706815ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 0x7f706815ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44070/0 (socket says 172.21.15.70:44070) 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 -- 172.21.15.70:0/625554255 learned_addr learned my addr 172.21.15.70:0/625554255 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 -- 172.21.15.70:0/625554255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 msgr2=0x7f7068156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f7068156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 -- 172.21.15.70:0/625554255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706815dfb0 msgr2=0x7f70681551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706815dfb0 0x7f70681551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 -- 172.21.15.70:0/625554255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7068157290 con 0x7f70680b5730 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706d5f9640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 0x7f706815ea50 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f70640ad040 tx=0x7f706401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70640be5f0 con 0x7f70680b5730 2024-04-22T01:17:42.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70640be7d0 con 0x7f70680b5730 2024-04-22T01:17:42.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70680cad80 con 0x7f70680b5730 2024-04-22T01:17:42.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70640bebc0 con 0x7f70680b5730 2024-04-22T01:17:42.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706815f410 con 0x7f70680b5730 2024-04-22T01:17:42.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7068068210 con 0x7f70680b5730 2024-04-22T01:17:42.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.353+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70640b43f0 con 0x7f70680b5730 2024-04-22T01:17:42.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.357+0000 7f705dffb640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f704c0678c0 0x7f704c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.357+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6101+0+0 (secure 0 0 0) 0x7f70641208e0 con 0x7f70680b5730 2024-04-22T01:17:42.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.357+0000 7f706cdf8640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f704c0678c0 0x7f704c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.357+0000 7f706cdf8640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f704c0678c0 0x7f704c069d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f70681618e0 tx=0x7f7058008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.361+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f70640c4060 con 0x7f70680b5730 2024-04-22T01:17:42.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.517+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7068068210 con 0x7f70680b5730 2024-04-22T01:17:42.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f705dffb640 1 -- 172.21.15.70:0/625554255 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+16922 (secure 0 0 0) 0x7f7064016110 con 0x7f70680b5730 2024-04-22T01:17:42.523 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:42.523 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":77,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:41.173616+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f704c0678c0 msgr2=0x7f704c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f704c0678c0 0x7f704c069d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f70681618e0 tx=0x7f7058008040 comp rx=0 tx=0).stop 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 msgr2=0x7f706815ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 0x7f706815ea50 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f70640ad040 tx=0x7f706401ad60 comp rx=0 tx=0).stop 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 shutdown_connections 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f704c0678c0 0x7f704c069d80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706815dfb0 0x7f70681551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680cea80 0x7f7068156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 --2- 172.21.15.70:0/625554255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70680b5730 0x7f706815ea50 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.521+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 >> 172.21.15.70:0/625554255 conn(0x7f706806c9b0 msgr2=0x7f70680b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:42.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.525+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 shutdown_connections 2024-04-22T01:17:42.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.525+0000 7f706f85d640 1 -- 172.21.15.70:0/625554255 wait complete. 2024-04-22T01:17:42.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:17:42.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 Processor -- start 2024-04-22T01:17:42.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 -- start start 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff0200d0060 0x7ff020099da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff02009a2e0 0x7ff02009a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff02015d1d0 con 0x7ff0200cfc80 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff0200cd790 con 0x7ff0200d0060 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff0263ce640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0200ccb80 con 0x7ff02009a2e0 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff01ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff0200cfc80 0x7ff020160340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37864/0 (socket says 172.21.15.70:37864) 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff01ffff640 1 -- 172.21.15.70:0/2067488630 learned_addr learned my addr 172.21.15.70:0/2067488630 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff02496b640 1 --2- 172.21.15.70:0/2067488630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff02009a2e0 0x7ff02009a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff017fff640 1 --2- 172.21.15.70:0/2067488630 >> v2:172.21.15.98:3300/0 conn(0x7ff0200d0060 0x7ff020099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff01f7fe640 1 -- 172.21.15.70:0/2067488630 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377706593 0 0) 0x7ff02015d1d0 con 0x7ff0200cfc80 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff01f7fe640 1 -- 172.21.15.70:0/2067488630 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff00c003530 con 0x7ff0200cfc80 2024-04-22T01:17:42.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff01f7fe640 1 -- 172.21.15.70:0/2067488630 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2983495550 0 0) 0x7ff00c003530 con 0x7ff0200cfc80 2024-04-22T01:17:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff02496b640 1 -- 172.21.15.70:0/2067488630 >> v2:172.21.15.98:3300/0 conn(0x7ff0200d0060 msgr2=0x7ff020099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff02496b640 1 --2- 172.21.15.70:0/2067488630 >> v2:172.21.15.98:3300/0 conn(0x7ff0200d0060 0x7ff020099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff02496b640 1 -- 172.21.15.70:0/2067488630 >> v1:172.21.15.70:6789/0 conn(0x7ff0200cfc80 legacy=0x7ff020160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.653+0000 7ff02496b640 1 -- 172.21.15.70:0/2067488630 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0200cce60 con 0x7ff02009a2e0 2024-04-22T01:17:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 --2- 172.21.15.70:0/2067488630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff02009a2e0 0x7ff02009a770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff018009d90 tx=0x7ff01801c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=312c36d356d6c960 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01f7fe640 1 -- 172.21.15.70:0/2067488630 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180b2070 con 0x7ff02009a2e0 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01f7fe640 1 -- 172.21.15.70:0/2067488630 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0180a55d0 con 0x7ff02009a2e0 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01f7fe640 1 -- 172.21.15.70:0/2067488630 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010001680 con 0x7ff02009a2e0 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/2067488630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff02009a2e0 msgr2=0x7ff02009a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 --2- 172.21.15.70:0/2067488630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff02009a2e0 0x7ff02009a770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff018009d90 tx=0x7ff01801c7d0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/2067488630 shutdown_connections 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 --2- 172.21.15.70:0/2067488630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff02009a2e0 0x7ff02009a770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 --2- 172.21.15.70:0/2067488630 >> v2:172.21.15.98:3300/0 conn(0x7ff0200d0060 0x7ff020099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/2067488630 >> 172.21.15.70:0/2067488630 conn(0x7ff0200b36f0 msgr2=0x7ff0200b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/2067488630 shutdown_connections 2024-04-22T01:17:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/2067488630 wait complete. 2024-04-22T01:17:42.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 Processor -- start 2024-04-22T01:17:42.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- start start 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff02009a2e0 0x7ff0201577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0200cfc80 0x7ff020153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 0x7ff0201520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff0200d05f0 con 0x7ff0200d0060 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0200d0470 con 0x7ff02009a2e0 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0200cc710 con 0x7ff0200cfc80 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 0x7ff0201520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 0x7ff0201520e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44084/0 (socket says 172.21.15.70:44084) 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 -- 172.21.15.70:0/298289514 learned_addr learned my addr 172.21.15.70:0/298289514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff017fff640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0200cfc80 0x7ff020153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01ffff640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff02009a2e0 0x7ff0201577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 -- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0200cfc80 msgr2=0x7ff020153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0200cfc80 0x7ff020153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 -- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff02009a2e0 msgr2=0x7ff0201577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff02009a2e0 0x7ff0201577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 -- 172.21.15.70:0/298289514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff020152620 con 0x7ff0200d0060 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01ffff640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff02009a2e0 0x7ff0201577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff02496b640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 0x7ff0201520e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff01801ae80 tx=0x7ff018019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180bb020 con 0x7ff0200d0060 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0180a5980 con 0x7ff0200d0060 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0200b53a0 con 0x7ff0200d0060 2024-04-22T01:17:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180b2040 con 0x7ff0200d0060 2024-04-22T01:17:42.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.657+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff020153350 con 0x7ff0200d0060 2024-04-22T01:17:42.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.661+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0200b5810 con 0x7ff0200d0060 2024-04-22T01:17:42.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.661+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff0180be020 con 0x7ff0200d0060 2024-04-22T01:17:42.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.661+0000 7ff01d7fa640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0000678c0 0x7ff000069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:42.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.661+0000 7ff01ffff640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0000678c0 0x7ff000069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:42.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.661+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6101+0+0 (secure 0 0 0) 0x7ff018120430 con 0x7ff0200d0060 2024-04-22T01:17:42.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.661+0000 7ff01ffff640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0000678c0 0x7ff000069d80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff02015ec20 tx=0x7ff010009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:42.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.665+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff0180c5030 con 0x7ff0200d0060 2024-04-22T01:17:42.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.861+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff020068920 con 0x7ff0200d0060 2024-04-22T01:17:42.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.873+0000 7ff01d7fa640 1 -- 172.21.15.70:0/298289514 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) ==== 118+0+0 (secure 0 0 0) 0x7ff018016480 con 0x7ff0200d0060 2024-04-22T01:17:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0000678c0 msgr2=0x7ff000069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0000678c0 0x7ff000069d80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff02015ec20 tx=0x7ff010009040 comp rx=0 tx=0).stop 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 msgr2=0x7ff0201520e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 0x7ff0201520e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff01801ae80 tx=0x7ff018019570 comp rx=0 tx=0).stop 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 shutdown_connections 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff0000678c0 0x7ff000069d80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0200d0060 0x7ff0201520e0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0200cfc80 0x7ff020153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 --2- 172.21.15.70:0/298289514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff02009a2e0 0x7ff0201577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 >> 172.21.15.70:0/298289514 conn(0x7ff0200b36f0 msgr2=0x7ff0200b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 shutdown_connections 2024-04-22T01:17:42.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:42.877+0000 7ff0263ce640 1 -- 172.21.15.70:0/298289514 wait complete. 2024-04-22T01:17:42.896 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-22T01:17:42.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:17:42.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:17:42.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:17:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 Processor -- start 2024-04-22T01:17:43.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 -- start start 2024-04-22T01:17:43.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab540b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fab540cea10 0x7fab540ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab540cf3e0 con 0x7fab540b6db0 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fab540b75e0 con 0x7fab540cea10 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5bcec640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fab540b78c0 con 0x7fab5415de20 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fab540cea10 0x7fab540ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fab540cea10 0x7fab540ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44090/0 (socket says 172.21.15.70:44090) 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 -- 172.21.15.70:0/2938087587 learned_addr learned my addr 172.21.15.70:0/2938087587 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab59a88640 1 --2- 172.21.15.70:0/2938087587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab540b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 -- 172.21.15.70:0/2938087587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 msgr2=0x7fab540b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 --2- 172.21.15.70:0/2938087587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab540b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 -- 172.21.15.70:0/2938087587 >> v1:172.21.15.70:6789/0 conn(0x7fab540b6db0 legacy=0x7fab540d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab58a86640 1 -- 172.21.15.70:0/2938087587 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757211106 0 0) 0x7fab540cf3e0 con 0x7fab540b6db0 2024-04-22T01:17:43.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 -- 172.21.15.70:0/2938087587 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab540b7ba0 con 0x7fab540cea10 2024-04-22T01:17:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.021+0000 7fab5a289640 1 --2- 172.21.15.70:0/2938087587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540cea10 0x7fab540ceea0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009e80 tx=0x7fab4c01f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bfb410a28f38b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab58a86640 1 -- 172.21.15.70:0/2938087587 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0b4070 con 0x7fab540cea10 2024-04-22T01:17:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab58a86640 1 -- 172.21.15.70:0/2938087587 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4c0af400 con 0x7fab540cea10 2024-04-22T01:17:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab58a86640 1 -- 172.21.15.70:0/2938087587 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0af7d0 con 0x7fab540cea10 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/2938087587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540cea10 msgr2=0x7fab540ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 --2- 172.21.15.70:0/2938087587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540cea10 0x7fab540ceea0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009e80 tx=0x7fab4c01f200 comp rx=0 tx=0).stop 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/2938087587 shutdown_connections 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 --2- 172.21.15.70:0/2938087587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540cea10 0x7fab540ceea0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 --2- 172.21.15.70:0/2938087587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab540b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/2938087587 >> 172.21.15.70:0/2938087587 conn(0x7fab54072d20 msgr2=0x7fab54073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/2938087587 shutdown_connections 2024-04-22T01:17:43.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/2938087587 wait complete. 2024-04-22T01:17:43.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 Processor -- start 2024-04-22T01:17:43.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- start start 2024-04-22T01:17:43.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 0x7fab54151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540cea10 0x7fab540d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab5415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab5415e200 con 0x7fab540b6db0 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab540b8160 con 0x7fab540cea10 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fab540b71f0 con 0x7fab5415de20 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 0x7fab54151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 0x7fab54151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44094/0 (socket says 172.21.15.70:44094) 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 -- 172.21.15.70:0/3222737679 learned_addr learned my addr 172.21.15.70:0/3222737679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 -- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 msgr2=0x7fab5415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5a289640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab5415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59287640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540cea10 0x7fab540d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab5415c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 -- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540cea10 msgr2=0x7fab540d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:43.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540cea10 0x7fab540d2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 -- 172.21.15.70:0/3222737679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab54154930 con 0x7fab540b6db0 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5a289640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab5415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab59a88640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 0x7fab54151f80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fab5000d5f0 tx=0x7fab5000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab500125f0 con 0x7fab540b6db0 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab50019070 con 0x7fab540b6db0 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab54154bd0 con 0x7fab540b6db0 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab500129b0 con 0x7fab540b6db0 2024-04-22T01:17:43.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.025+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab54155190 con 0x7fab540b6db0 2024-04-22T01:17:43.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.029+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab54068210 con 0x7fab540b6db0 2024-04-22T01:17:43.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.029+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab50012b50 con 0x7fab540b6db0 2024-04-22T01:17:43.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.029+0000 7fab4a7fc640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab380678c0 0x7fab38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:43.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.029+0000 7fab59287640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab380678c0 0x7fab38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:43.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.029+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fab50086260 con 0x7fab540b6db0 2024-04-22T01:17:43.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.029+0000 7fab59287640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab380678c0 0x7fab38069d80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab540d4cd0 tx=0x7fab44009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:43.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.033+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fab50056c10 con 0x7fab540b6db0 2024-04-22T01:17:43.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.229+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fab540ce7c0 con 0x7fab540b6db0 2024-04-22T01:17:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.877+0000 7fab4a7fc640 1 -- 172.21.15.70:0/3222737679 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) ==== 122+0+0 (secure 0 0 0) 0x7fab50056510 con 0x7fab540b6db0 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.877+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab380678c0 msgr2=0x7fab38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.877+0000 7fab5bcec640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab380678c0 0x7fab38069d80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab540d4cd0 tx=0x7fab44009040 comp rx=0 tx=0).stop 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.877+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 msgr2=0x7fab54151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 0x7fab54151f80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fab5000d5f0 tx=0x7fab5000dab0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 shutdown_connections 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab380678c0 0x7fab38069d80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5415de20 0x7fab5415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540cea10 0x7fab540d2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 --2- 172.21.15.70:0/3222737679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab540b6db0 0x7fab54151f80 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:43.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 >> 172.21.15.70:0/3222737679 conn(0x7fab54072d20 msgr2=0x7fab540c9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:43.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 shutdown_connections 2024-04-22T01:17:43.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:43.881+0000 7fab5bcec640 1 -- 172.21.15.70:0/3222737679 wait complete. 2024-04-22T01:17:43.902 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:17:43.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 Processor -- start 2024-04-22T01:17:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 -- start start 2024-04-22T01:17:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c15f180 0x7f8c3c06b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8c3c06b730 0x7f8c3c0af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c3c06efc0 con 0x7f8c3c15e690 2024-04-22T01:17:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8c3c06fbd0 con 0x7f8c3c06b730 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c41557640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3c06f2a0 con 0x7f8c3c15f180 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3affd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8c3c15e690 0x7f8c3c15ea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37882/0 (socket says 172.21.15.70:37882) 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3affd640 1 -- 172.21.15.70:0/2386593298 learned_addr learned my addr 172.21.15.70:0/2386593298 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/2386593298 >> v2:172.21.15.70:3300/0 conn(0x7f8c3c06b730 0x7f8c3c0af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3b7fe640 1 -- 172.21.15.70:0/2386593298 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c15f180 msgr2=0x7f8c3c06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/2386593298 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c15f180 0x7f8c3c06b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3b7fe640 1 -- 172.21.15.70:0/2386593298 >> v1:172.21.15.70:6789/0 conn(0x7f8c3c15e690 legacy=0x7f8c3c15ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.065+0000 7f8c3b7fe640 1 -- 172.21.15.70:0/2386593298 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c06f580 con 0x7f8c3c06b730 2024-04-22T01:17:44.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/2386593298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c06b730 0x7f8c3c0af7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009d90 tx=0x7f8c3401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b6e307f50d7bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c39ffb640 1 -- 172.21.15.70:0/2386593298 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c340b2070 con 0x7f8c3c06b730 2024-04-22T01:17:44.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c39ffb640 1 -- 172.21.15.70:0/2386593298 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c340a5830 con 0x7f8c3c06b730 2024-04-22T01:17:44.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c39ffb640 1 -- 172.21.15.70:0/2386593298 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c340a5c00 con 0x7f8c3c06b730 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- 172.21.15.70:0/2386593298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c06b730 msgr2=0x7f8c3c0af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 --2- 172.21.15.70:0/2386593298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c06b730 0x7f8c3c0af7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009d90 tx=0x7f8c3401c970 comp rx=0 tx=0).stop 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- 172.21.15.70:0/2386593298 shutdown_connections 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 --2- 172.21.15.70:0/2386593298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c06b730 0x7f8c3c0af7a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 --2- 172.21.15.70:0/2386593298 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c15f180 0x7f8c3c06b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- 172.21.15.70:0/2386593298 >> 172.21.15.70:0/2386593298 conn(0x7f8c3c099a60 msgr2=0x7f8c3c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- 172.21.15.70:0/2386593298 shutdown_connections 2024-04-22T01:17:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- 172.21.15.70:0/2386593298 wait complete. 2024-04-22T01:17:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 Processor -- start 2024-04-22T01:17:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- start start 2024-04-22T01:17:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c06b730 0x7f8c3c0c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 0x7f8c3c15f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c15f180 0x7f8c3c15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c0b0270 con 0x7f8c3c15e690 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c0b00f0 con 0x7f8c3c15f180 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c41557640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3c0d0390 con 0x7f8c3c06b730 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 0x7f8c3c15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 0x7f8c3c15f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44118/0 (socket says 172.21.15.70:44118) 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 -- 172.21.15.70:0/655674946 learned_addr learned my addr 172.21.15.70:0/655674946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3affd640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c06b730 0x7f8c3c0c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c15f180 0x7f8c3c15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 -- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c06b730 msgr2=0x7f8c3c0c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c06b730 0x7f8c3c0c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 -- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c15f180 msgr2=0x7f8c3c15faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.069+0000 7f8c3a7fc640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c15f180 0x7f8c3c15faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c3a7fc640 1 -- 172.21.15.70:0/655674946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c0c5c30 con 0x7f8c3c15e690 2024-04-22T01:17:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c3a7fc640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 0x7f8c3c15f560 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000b440 tx=0x7f8c3000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30015020 con 0x7f8c3c15e690 2024-04-22T01:17:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c300118b0 con 0x7f8c3c15e690 2024-04-22T01:17:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3c0c5f10 con 0x7f8c3c15e690 2024-04-22T01:17:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30011ca0 con 0x7f8c3c15e690 2024-04-22T01:17:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c3c0c64d0 con 0x7f8c3c15e690 2024-04-22T01:17:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3c068920 con 0x7f8c3c15e690 2024-04-22T01:17:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8c30018020 con 0x7f8c3c15e690 2024-04-22T01:17:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c237fe640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c280678c0 0x7f8c28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f8c30085fb0 con 0x7f8c3c15e690 2024-04-22T01:17:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c3affd640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c280678c0 0x7f8c28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.073+0000 7f8c3affd640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c280678c0 0x7f8c28069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c000f70 tx=0x7f8c2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.077+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8c3001f020 con 0x7f8c3c15e690 2024-04-22T01:17:44.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c3c0c42b0 con 0x7f8c3c15e690 2024-04-22T01:17:44.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c237fe640 1 -- 172.21.15.70:0/655674946 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5174 (secure 0 0 0) 0x7f8c3000d040 con 0x7f8c3c15e690 2024-04-22T01:17:44.278 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:44.278 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:41.174335+0000","modified":"2024-04-22T01:17:43.233600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6759":{"gid":6759,"name":"d","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6833/258015518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":258015518},{"type":"v1","addr":"172.21.15.98:6833","nonce":258015518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6759,"qdb_cluster":[6759]},"id":5}]} 2024-04-22T01:17:44.279 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 37 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c280678c0 msgr2=0x7f8c28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c280678c0 0x7f8c28069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c000f70 tx=0x7f8c2c009040 comp rx=0 tx=0).stop 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 msgr2=0x7f8c3c15f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 0x7f8c3c15f560 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000b440 tx=0x7f8c3000b900 comp rx=0 tx=0).stop 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 shutdown_connections 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8c280678c0 0x7f8c28069d80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c3c15f180 0x7f8c3c15faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c3c15e690 0x7f8c3c15f560 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 --2- 172.21.15.70:0/655674946 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c3c06b730 0x7f8c3c0c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 >> 172.21.15.70:0/655674946 conn(0x7f8c3c099a60 msgr2=0x7f8c3c160e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 shutdown_connections 2024-04-22T01:17:44.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.277+0000 7f8c41557640 1 -- 172.21.15.70:0/655674946 wait complete. 2024-04-22T01:17:44.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.413+0000 7faef27a9640 1 Processor -- start 2024-04-22T01:17:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.413+0000 7faef27a9640 1 -- start start 2024-04-22T01:17:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef27a9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 0x7faeec0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef27a9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faeec0d0250 con 0x7faeec072230 2024-04-22T01:17:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef27a9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faeec0cd7b0 con 0x7faeec0cc150 2024-04-22T01:17:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef27a9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faeec0cc710 con 0x7faeec1455b0 2024-04-22T01:17:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef0d46640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7faeec072230 0x7faeec06e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37886/0 (socket says 172.21.15.70:37886) 2024-04-22T01:17:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef0d46640 1 -- 172.21.15.70:0/1704375941 learned_addr learned my addr 172.21.15.70:0/1704375941 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeebfff640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 0x7faeec0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110376060 0 0) 0x7faeec0d0250 con 0x7faeec072230 2024-04-22T01:17:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faed4003530 con 0x7faeec072230 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2143677400 0 0) 0x7faed4003530 con 0x7faeec072230 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 msgr2=0x7faeec0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 0x7faeec0cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 >> v1:172.21.15.98:6789/0 conn(0x7faeec0cc150 legacy=0x7faeec071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeec0cc9f0 con 0x7faeec072230 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7faee4001a00 con 0x7faeec072230 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 >> v1:172.21.15.70:6789/0 conn(0x7faeec072230 legacy=0x7faeec06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed40039f0 0x7faed4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faed4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 0x7faed40052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faeec0cc710 con 0x7faed4004810 2024-04-22T01:17:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faed400bb90 con 0x7faed40039f0 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faed400bd90 con 0x7faeec1455b0 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faed4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeebfff640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed40039f0 0x7faed4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faef0d46640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 0x7faed40052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeb7fe640 1 -- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 msgr2=0x7faed40052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec1455b0 0x7faed40052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeb7fe640 1 -- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed40039f0 msgr2=0x7faed4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed40039f0 0x7faed4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.417+0000 7faeeb7fe640 1 -- 172.21.15.70:0/1704375941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed4003530 con 0x7faed4004810 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faed4004c90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7faee0001780 tx=0x7faee001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efcf97f88cd10da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00013a0 con 0x7faed4004810 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee00052f0 con 0x7faed4004810 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeeaffd640 1 -- 172.21.15.70:0/1704375941 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0003840 con 0x7faed4004810 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/1704375941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 msgr2=0x7faed4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 --2- 172.21.15.70:0/1704375941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faed4004c90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7faee0001780 tx=0x7faee001c420 comp rx=0 tx=0).stop 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef0d46640 1 -- 172.21.15.70:0/1704375941 reap_dead start 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/1704375941 shutdown_connections 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/1704375941 >> 172.21.15.70:0/1704375941 conn(0x7faeec06b1f0 msgr2=0x7faeec06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/1704375941 shutdown_connections 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/1704375941 wait complete. 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 Processor -- start 2024-04-22T01:17:44.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- start start 2024-04-22T01:17:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faeec133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec163b60 0x7faeec141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeec163fe0 0x7faeec142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faeec167ed0 con 0x7faed4004810 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faeec167d50 con 0x7faeec163fe0 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faeec168050 con 0x7faeec163b60 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faeec133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faeec133780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44138/0 (socket says 172.21.15.70:44138) 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 -- 172.21.15.70:0/3284699295 learned_addr learned my addr 172.21.15.70:0/3284699295 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec163b60 0x7faeec141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef0d46640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeec163fe0 0x7faeec142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 -- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec163b60 msgr2=0x7faeec141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec163b60 0x7faeec141d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 -- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeec163fe0 msgr2=0x7faeec142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeec163fe0 0x7faeec142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 -- 172.21.15.70:0/3284699295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeec1645b0 con 0x7faed4004810 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faeebfff640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faeec133780 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7faedc011610 tx=0x7faedc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faedc00f8f0 con 0x7faed4004810 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faedc00fa90 con 0x7faed4004810 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeec164890 con 0x7faed4004810 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faedc00fe40 con 0x7faed4004810 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.421+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeec165650 con 0x7faed4004810 2024-04-22T01:17:44.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.425+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeec068210 con 0x7faed4004810 2024-04-22T01:17:44.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.425+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faedc015460 con 0x7faed4004810 2024-04-22T01:17:44.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.425+0000 7faee8ff9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faed80678f0 0x7faed8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.425+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7faedc086850 con 0x7faed4004810 2024-04-22T01:17:44.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.425+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faed80678f0 0x7faed8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.425+0000 7faeeb7fe640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faed80678f0 0x7faed8069db0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faee0001750 tx=0x7faee00011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.429+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7faedc019030 con 0x7faed4004810 2024-04-22T01:17:44.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.589+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faeec0cbb00 con 0x7faed4004810 2024-04-22T01:17:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.593+0000 7faee8ff9640 1 -- 172.21.15.70:0/3284699295 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7faedc05d030 con 0x7faed4004810 2024-04-22T01:17:44.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:44.596 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faed80678f0 msgr2=0x7faed8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faed80678f0 0x7faed8069db0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faee0001750 tx=0x7faee00011a0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 msgr2=0x7faeec133780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faeec133780 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7faedc011610 tx=0x7faedc011ad0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 shutdown_connections 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faed80678f0 0x7faed8069db0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeec163fe0 0x7faeec142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faeec163b60 0x7faeec141d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 --2- 172.21.15.70:0/3284699295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed4004810 0x7faeec133780 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 >> 172.21.15.70:0/3284699295 conn(0x7faeec06b1f0 msgr2=0x7faeec15e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 shutdown_connections 2024-04-22T01:17:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.597+0000 7faef27a9640 1 -- 172.21.15.70:0/3284699295 wait complete. 2024-04-22T01:17:44.614 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:17:44.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:17:44.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.725+0000 7f1c043cd640 1 Processor -- start 2024-04-22T01:17:44.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.725+0000 7f1c043cd640 1 -- start start 2024-04-22T01:17:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c043cd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bfc15dfb0 0x7f1bfc0b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c043cd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 0x7f1bfc0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c043cd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bfc15e390 con 0x7f1bfc0cea80 2024-04-22T01:17:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c043cd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1bfc0cf420 con 0x7f1bfc15dfb0 2024-04-22T01:17:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c043cd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1bfc161e00 con 0x7f1bfc0d3120 2024-04-22T01:17:44.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01968640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1bfc0cea80 0x7f1bfc0b5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37894/0 (socket says 172.21.15.70:37894) 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01968640 1 -- 172.21.15.70:0/759842102 learned_addr learned my addr 172.21.15.70:0/759842102 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236686271 0 0) 0x7f1bfc15e390 con 0x7f1bfc0cea80 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be8003530 con 0x7f1bfc0cea80 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c0296a640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 0x7f1bfc0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 --2- 172.21.15.70:0/759842102 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc15dfb0 0x7f1bfc0b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2544278645 0 0) 0x7f1be8003530 con 0x7f1bfc0cea80 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 msgr2=0x7f1bfc0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 0x7f1bfc0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc15dfb0 msgr2=0x7f1bfc0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 --2- 172.21.15.70:0/759842102 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc15dfb0 0x7f1bfc0b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc1620e0 con 0x7f1bfc0cea80 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1bf80016a0 con 0x7f1bfc0cea80 2024-04-22T01:17:44.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 --2- 172.21.15.70:0/759842102 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc15dfb0 0x7f1bfc0b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 >> v1:172.21.15.70:6789/0 conn(0x7f1bfc0cea80 legacy=0x7f1bfc0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1be8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 0x7f1be8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be8004d50 0x7f1be80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc161e00 con 0x7f1be80039c0 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1be800ba80 con 0x7f1be8004d50 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1be800bc30 con 0x7f1bfc0d3120 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1be8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c0296a640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be8004d50 0x7f1be80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 -- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 msgr2=0x7f1be8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0d3120 0x7f1be8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 -- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be8004d50 msgr2=0x7f1be80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be8004d50 0x7f1be80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 -- 172.21.15.70:0/759842102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be8003530 con 0x7f1be80039c0 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c0296a640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be8004d50 0x7f1be80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c02169640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1be8003de0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40073e0 tx=0x7f1bf401cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979d2918bf78134a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b2070 con 0x7f1be80039c0 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf40a5d30 con 0x7f1be80039c0 2024-04-22T01:17:44.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.729+0000 7f1c01167640 1 -- 172.21.15.70:0/759842102 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4009c80 con 0x7f1be80039c0 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/759842102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 msgr2=0x7f1be8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 --2- 172.21.15.70:0/759842102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1be8003de0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40073e0 tx=0x7f1bf401cee0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c0296a640 1 -- 172.21.15.70:0/759842102 reap_dead start 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/759842102 shutdown_connections 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/759842102 >> 172.21.15.70:0/759842102 conn(0x7f1bfc06c9b0 msgr2=0x7f1bfc06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/759842102 shutdown_connections 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/759842102 wait complete. 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 Processor -- start 2024-04-22T01:17:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- start start 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1bfc1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0b7380 0x7f1bfc0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0cad80 0x7f1bfc0cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc161a90 con 0x7f1be80039c0 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc168dc0 con 0x7f1bfc0cad80 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1bfc168f40 con 0x7f1bfc0b7380 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1bfc1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1bfc1542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44152/0 (socket says 172.21.15.70:44152) 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 -- 172.21.15.70:0/3791230702 learned_addr learned my addr 172.21.15.70:0/3791230702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c01968640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0b7380 0x7f1bfc0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c0296a640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0cad80 0x7f1bfc0cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 -- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0b7380 msgr2=0x7f1bfc0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0b7380 0x7f1bfc0b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 -- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0cad80 msgr2=0x7f1bfc0cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0cad80 0x7f1bfc0cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 -- 172.21.15.70:0/3791230702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc0cb770 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c0296a640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0cad80 0x7f1bfc0cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c02169640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1bfc1542d0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40b0040 tx=0x7f1be8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b2040 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf40ada60 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc0cba00 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40ade30 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc15f190 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.733+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bfc068210 con 0x7f1be80039c0 2024-04-22T01:17:44.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.737+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1bf40039a0 con 0x7f1be80039c0 2024-04-22T01:17:44.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.737+0000 7f1beeffd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1be00678f0 0x7f1be0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:44.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.737+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f1bf411fc90 con 0x7f1be80039c0 2024-04-22T01:17:44.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.737+0000 7f1c01968640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1be00678f0 0x7f1be0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:44.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.737+0000 7f1c01968640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1be00678f0 0x7f1be0069db0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8001ec0 tx=0x7f1bf8001630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:44.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.741+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1bf40bd060 con 0x7f1be80039c0 2024-04-22T01:17:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.893+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1bfc154850 con 0x7f1be00678f0 2024-04-22T01:17:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.905+0000 7f1beeffd640 1 -- 172.21.15.70:0/3791230702 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252060 (secure 0 0 0) 0x7f1bfc154850 con 0x7f1be00678f0 2024-04-22T01:17:44.909 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:44.911 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1be00678f0 msgr2=0x7f1be0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1be00678f0 0x7f1be0069db0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8001ec0 tx=0x7f1bf8001630 comp rx=0 tx=0).stop 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 msgr2=0x7f1bfc1542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1bfc1542d0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40b0040 tx=0x7f1be8005840 comp rx=0 tx=0).stop 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 shutdown_connections 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1be00678f0 0x7f1be0069db0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0cad80 0x7f1bfc0cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc0b7380 0x7f1bfc0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 --2- 172.21.15.70:0/3791230702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be80039c0 0x7f1bfc1542d0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 >> 172.21.15.70:0/3791230702 conn(0x7f1bfc06c9b0 msgr2=0x7f1bfc0b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 shutdown_connections 2024-04-22T01:17:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:44.909+0000 7f1c043cd640 1 -- 172.21.15.70:0/3791230702 wait complete. 2024-04-22T01:17:44.926 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-22T01:17:44.640283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262260,"kb_used_data":8632,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712460,"statfs":{"total":773094113280,"available":772825559040,"internally_reserved":0,"allocated":8839168,"data_stored":4645636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.840015"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892972+0000","last_change":"2024-04-22T01:17:41.889762+0000","last_active":"2024-04-22T01:17:41.892972+0000","last_peered":"2024-04-22T01:17:41.892972+0000","last_clean":"2024-04-22T01:17:41.892972+0000","last_became_active":"2024-04-22T01:17:41.889429+0000","last_became_peered":"2024-04-22T01:17:41.889429+0000","last_unstale":"2024-04-22T01:17:41.892972+0000","last_undegraded":"2024-04-22T01:17:41.892972+0000","last_fullsized":"2024-04-22T01:17:41.892972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:34:39.563352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886738+0000","last_change":"2024-04-22T01:17:41.893467+0000","last_active":"2024-04-22T01:17:42.886738+0000","last_peered":"2024-04-22T01:17:42.886738+0000","last_clean":"2024-04-22T01:17:42.886738+0000","last_became_active":"2024-04-22T01:17:41.893013+0000","last_became_peered":"2024-04-22T01:17:41.893013+0000","last_unstale":"2024-04-22T01:17:42.886738+0000","last_undegraded":"2024-04-22T01:17:42.886738+0000","last_fullsized":"2024-04-22T01:17:42.886738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:59:24.841737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.885863+0000","last_change":"2024-04-22T01:17:41.878734+0000","last_active":"2024-04-22T01:17:41.885863+0000","last_peered":"2024-04-22T01:17:41.885863+0000","last_clean":"2024-04-22T01:17:41.885863+0000","last_became_active":"2024-04-22T01:17:41.878237+0000","last_became_peered":"2024-04-22T01:17:41.878237+0000","last_unstale":"2024-04-22T01:17:41.885863+0000","last_undegraded":"2024-04-22T01:17:41.885863+0000","last_fullsized":"2024-04-22T01:17:41.885863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:58.237820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849447+0000","last_change":"2024-04-22T01:17:41.867063+0000","last_active":"2024-04-22T01:17:42.849447+0000","last_peered":"2024-04-22T01:17:42.849447+0000","last_clean":"2024-04-22T01:17:42.849447+0000","last_became_active":"2024-04-22T01:17:41.866349+0000","last_became_peered":"2024-04-22T01:17:41.866349+0000","last_unstale":"2024-04-22T01:17:42.849447+0000","last_undegraded":"2024-04-22T01:17:42.849447+0000","last_fullsized":"2024-04-22T01:17:42.849447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:05.400978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847908+0000","last_change":"2024-04-22T01:17:41.876173+0000","last_active":"2024-04-22T01:17:42.847908+0000","last_peered":"2024-04-22T01:17:42.847908+0000","last_clean":"2024-04-22T01:17:42.847908+0000","last_became_active":"2024-04-22T01:17:41.875573+0000","last_became_peered":"2024-04-22T01:17:41.875573+0000","last_unstale":"2024-04-22T01:17:42.847908+0000","last_undegraded":"2024-04-22T01:17:42.847908+0000","last_fullsized":"2024-04-22T01:17:42.847908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:34.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892004+0000","last_change":"2024-04-22T01:17:41.891143+0000","last_active":"2024-04-22T01:17:41.892004+0000","last_peered":"2024-04-22T01:17:41.892004+0000","last_clean":"2024-04-22T01:17:41.892004+0000","last_became_active":"2024-04-22T01:17:41.890146+0000","last_became_peered":"2024-04-22T01:17:41.890146+0000","last_unstale":"2024-04-22T01:17:41.892004+0000","last_undegraded":"2024-04-22T01:17:41.892004+0000","last_fullsized":"2024-04-22T01:17:41.892004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:34:39.563352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850355+0000","last_change":"2024-04-22T01:17:41.895449+0000","last_active":"2024-04-22T01:17:42.850355+0000","last_peered":"2024-04-22T01:17:42.850355+0000","last_clean":"2024-04-22T01:17:42.850355+0000","last_became_active":"2024-04-22T01:17:41.893334+0000","last_became_peered":"2024-04-22T01:17:41.893334+0000","last_unstale":"2024-04-22T01:17:42.850355+0000","last_undegraded":"2024-04-22T01:17:42.850355+0000","last_fullsized":"2024-04-22T01:17:42.850355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:58:52.699678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848084+0000","last_change":"2024-04-22T01:17:41.877758+0000","last_active":"2024-04-22T01:17:42.848084+0000","last_peered":"2024-04-22T01:17:42.848084+0000","last_clean":"2024-04-22T01:17:42.848084+0000","last_became_active":"2024-04-22T01:17:41.877271+0000","last_became_peered":"2024-04-22T01:17:41.877271+0000","last_unstale":"2024-04-22T01:17:42.848084+0000","last_undegraded":"2024-04-22T01:17:42.848084+0000","last_fullsized":"2024-04-22T01:17:42.848084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:32:52.914497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889091+0000","last_change":"2024-04-22T01:17:41.872477+0000","last_active":"2024-04-22T01:17:41.889091+0000","last_peered":"2024-04-22T01:17:41.889091+0000","last_clean":"2024-04-22T01:17:41.889091+0000","last_became_active":"2024-04-22T01:17:41.871772+0000","last_became_peered":"2024-04-22T01:17:41.871772+0000","last_unstale":"2024-04-22T01:17:41.889091+0000","last_undegraded":"2024-04-22T01:17:41.889091+0000","last_fullsized":"2024-04-22T01:17:41.889091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:45.951103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849420+0000","last_change":"2024-04-22T01:17:41.882229+0000","last_active":"2024-04-22T01:17:42.849420+0000","last_peered":"2024-04-22T01:17:42.849420+0000","last_clean":"2024-04-22T01:17:42.849420+0000","last_became_active":"2024-04-22T01:17:41.881057+0000","last_became_peered":"2024-04-22T01:17:41.881057+0000","last_unstale":"2024-04-22T01:17:42.849420+0000","last_undegraded":"2024-04-22T01:17:42.849420+0000","last_fullsized":"2024-04-22T01:17:42.849420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:43.570838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886598+0000","last_change":"2024-04-22T01:17:41.893467+0000","last_active":"2024-04-22T01:17:42.886598+0000","last_peered":"2024-04-22T01:17:42.886598+0000","last_clean":"2024-04-22T01:17:42.886598+0000","last_became_active":"2024-04-22T01:17:41.884806+0000","last_became_peered":"2024-04-22T01:17:41.884806+0000","last_unstale":"2024-04-22T01:17:42.886598+0000","last_undegraded":"2024-04-22T01:17:42.886598+0000","last_fullsized":"2024-04-22T01:17:42.886598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:54:42.629423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849840+0000","last_change":"2024-04-22T01:17:41.878986+0000","last_active":"2024-04-22T01:17:42.849840+0000","last_peered":"2024-04-22T01:17:42.849840+0000","last_clean":"2024-04-22T01:17:42.849840+0000","last_became_active":"2024-04-22T01:17:41.877965+0000","last_became_peered":"2024-04-22T01:17:41.877965+0000","last_unstale":"2024-04-22T01:17:42.849840+0000","last_undegraded":"2024-04-22T01:17:42.849840+0000","last_fullsized":"2024-04-22T01:17:42.849840+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:27.182121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848195+0000","last_change":"2024-04-22T01:17:41.875657+0000","last_active":"2024-04-22T01:17:42.848195+0000","last_peered":"2024-04-22T01:17:42.848195+0000","last_clean":"2024-04-22T01:17:42.848195+0000","last_became_active":"2024-04-22T01:17:41.874951+0000","last_became_peered":"2024-04-22T01:17:41.874951+0000","last_unstale":"2024-04-22T01:17:42.848195+0000","last_undegraded":"2024-04-22T01:17:42.848195+0000","last_fullsized":"2024-04-22T01:17:42.848195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:15:50.183294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.887798+0000","last_change":"2024-04-22T01:17:41.883265+0000","last_active":"2024-04-22T01:17:41.887798+0000","last_peered":"2024-04-22T01:17:41.887798+0000","last_clean":"2024-04-22T01:17:41.887798+0000","last_became_active":"2024-04-22T01:17:41.882708+0000","last_became_peered":"2024-04-22T01:17:41.882708+0000","last_unstale":"2024-04-22T01:17:41.887798+0000","last_undegraded":"2024-04-22T01:17:41.887798+0000","last_fullsized":"2024-04-22T01:17:41.887798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:27.182121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886248+0000","last_change":"2024-04-22T01:17:41.875070+0000","last_active":"2024-04-22T01:17:42.886248+0000","last_peered":"2024-04-22T01:17:42.886248+0000","last_clean":"2024-04-22T01:17:42.886248+0000","last_became_active":"2024-04-22T01:17:41.873429+0000","last_became_peered":"2024-04-22T01:17:41.873429+0000","last_unstale":"2024-04-22T01:17:42.886248+0000","last_undegraded":"2024-04-22T01:17:42.886248+0000","last_fullsized":"2024-04-22T01:17:42.886248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:38.710984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.875774+0000","last_change":"2024-04-22T01:17:41.868397+0000","last_active":"2024-04-22T01:17:41.875774+0000","last_peered":"2024-04-22T01:17:41.875774+0000","last_clean":"2024-04-22T01:17:41.875774+0000","last_became_active":"2024-04-22T01:17:41.867596+0000","last_became_peered":"2024-04-22T01:17:41.867596+0000","last_unstale":"2024-04-22T01:17:41.875774+0000","last_undegraded":"2024-04-22T01:17:41.875774+0000","last_fullsized":"2024-04-22T01:17:41.875774+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:05.400978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848489+0000","last_change":"2024-04-22T01:17:41.889444+0000","last_active":"2024-04-22T01:17:42.848489+0000","last_peered":"2024-04-22T01:17:42.848489+0000","last_clean":"2024-04-22T01:17:42.848489+0000","last_became_active":"2024-04-22T01:17:41.889000+0000","last_became_peered":"2024-04-22T01:17:41.889000+0000","last_unstale":"2024-04-22T01:17:42.848489+0000","last_undegraded":"2024-04-22T01:17:42.848489+0000","last_fullsized":"2024-04-22T01:17:42.848489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:41.038244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.868983+0000","last_change":"2024-04-22T01:17:41.861362+0000","last_active":"2024-04-22T01:17:41.868983+0000","last_peered":"2024-04-22T01:17:41.868983+0000","last_clean":"2024-04-22T01:17:41.868983+0000","last_became_active":"2024-04-22T01:17:41.860570+0000","last_became_peered":"2024-04-22T01:17:41.860570+0000","last_unstale":"2024-04-22T01:17:41.868983+0000","last_undegraded":"2024-04-22T01:17:41.868983+0000","last_fullsized":"2024-04-22T01:17:41.868983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:27:09.461717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.871761+0000","last_change":"2024-04-22T01:17:41.865860+0000","last_active":"2024-04-22T01:17:41.871761+0000","last_peered":"2024-04-22T01:17:41.871761+0000","last_clean":"2024-04-22T01:17:41.871761+0000","last_became_active":"2024-04-22T01:17:41.863593+0000","last_became_peered":"2024-04-22T01:17:41.863593+0000","last_unstale":"2024-04-22T01:17:41.871761+0000","last_undegraded":"2024-04-22T01:17:41.871761+0000","last_fullsized":"2024-04-22T01:17:41.871761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:05.493690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847903+0000","last_change":"2024-04-22T01:17:41.868773+0000","last_active":"2024-04-22T01:17:42.847903+0000","last_peered":"2024-04-22T01:17:42.847903+0000","last_clean":"2024-04-22T01:17:42.847903+0000","last_became_active":"2024-04-22T01:17:41.867463+0000","last_became_peered":"2024-04-22T01:17:41.867463+0000","last_unstale":"2024-04-22T01:17:42.847903+0000","last_undegraded":"2024-04-22T01:17:42.847903+0000","last_fullsized":"2024-04-22T01:17:42.847903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:06.830084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889123+0000","last_change":"2024-04-22T01:17:41.872432+0000","last_active":"2024-04-22T01:17:41.889123+0000","last_peered":"2024-04-22T01:17:41.889123+0000","last_clean":"2024-04-22T01:17:41.889123+0000","last_became_active":"2024-04-22T01:17:41.870157+0000","last_became_peered":"2024-04-22T01:17:41.870157+0000","last_unstale":"2024-04-22T01:17:41.889123+0000","last_undegraded":"2024-04-22T01:17:41.889123+0000","last_fullsized":"2024-04-22T01:17:41.889123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:19:05.042181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847972+0000","last_change":"2024-04-22T01:17:41.874790+0000","last_active":"2024-04-22T01:17:42.847972+0000","last_peered":"2024-04-22T01:17:42.847972+0000","last_clean":"2024-04-22T01:17:42.847972+0000","last_became_active":"2024-04-22T01:17:41.874200+0000","last_became_peered":"2024-04-22T01:17:41.874200+0000","last_unstale":"2024-04-22T01:17:42.847972+0000","last_undegraded":"2024-04-22T01:17:42.847972+0000","last_fullsized":"2024-04-22T01:17:42.847972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:48.541919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.888772+0000","last_change":"2024-04-22T01:17:41.858324+0000","last_active":"2024-04-22T01:17:41.888772+0000","last_peered":"2024-04-22T01:17:41.888772+0000","last_clean":"2024-04-22T01:17:41.888772+0000","last_became_active":"2024-04-22T01:17:41.857554+0000","last_became_peered":"2024-04-22T01:17:41.857554+0000","last_unstale":"2024-04-22T01:17:41.888772+0000","last_undegraded":"2024-04-22T01:17:41.888772+0000","last_fullsized":"2024-04-22T01:17:41.888772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:37.885728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886115+0000","last_change":"2024-04-22T01:17:41.883449+0000","last_active":"2024-04-22T01:17:42.886115+0000","last_peered":"2024-04-22T01:17:42.886115+0000","last_clean":"2024-04-22T01:17:42.886115+0000","last_became_active":"2024-04-22T01:17:41.883143+0000","last_became_peered":"2024-04-22T01:17:41.883143+0000","last_unstale":"2024-04-22T01:17:42.886115+0000","last_undegraded":"2024-04-22T01:17:42.886115+0000","last_fullsized":"2024-04-22T01:17:42.886115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:52.040748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886947+0000","last_change":"2024-04-22T01:17:41.867495+0000","last_active":"2024-04-22T01:17:42.886947+0000","last_peered":"2024-04-22T01:17:42.886947+0000","last_clean":"2024-04-22T01:17:42.886947+0000","last_became_active":"2024-04-22T01:17:41.866764+0000","last_became_peered":"2024-04-22T01:17:41.866764+0000","last_unstale":"2024-04-22T01:17:42.886947+0000","last_undegraded":"2024-04-22T01:17:42.886947+0000","last_fullsized":"2024-04-22T01:17:42.886947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:27:25.202727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849815+0000","last_change":"2024-04-22T01:17:41.881695+0000","last_active":"2024-04-22T01:17:42.849815+0000","last_peered":"2024-04-22T01:17:42.849815+0000","last_clean":"2024-04-22T01:17:42.849815+0000","last_became_active":"2024-04-22T01:17:41.880183+0000","last_became_peered":"2024-04-22T01:17:41.880183+0000","last_unstale":"2024-04-22T01:17:42.849815+0000","last_undegraded":"2024-04-22T01:17:42.849815+0000","last_fullsized":"2024-04-22T01:17:42.849815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:15.283639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.882617+0000","last_change":"2024-04-22T01:17:41.876203+0000","last_active":"2024-04-22T01:17:41.882617+0000","last_peered":"2024-04-22T01:17:41.882617+0000","last_clean":"2024-04-22T01:17:41.882617+0000","last_became_active":"2024-04-22T01:17:41.875465+0000","last_became_peered":"2024-04-22T01:17:41.875465+0000","last_unstale":"2024-04-22T01:17:41.882617+0000","last_undegraded":"2024-04-22T01:17:41.882617+0000","last_fullsized":"2024-04-22T01:17:41.882617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:27.182121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847917+0000","last_change":"2024-04-22T01:17:41.874913+0000","last_active":"2024-04-22T01:17:42.847917+0000","last_peered":"2024-04-22T01:17:42.847917+0000","last_clean":"2024-04-22T01:17:42.847917+0000","last_became_active":"2024-04-22T01:17:41.874304+0000","last_became_peered":"2024-04-22T01:17:41.874304+0000","last_unstale":"2024-04-22T01:17:42.847917+0000","last_undegraded":"2024-04-22T01:17:42.847917+0000","last_fullsized":"2024-04-22T01:17:42.847917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:51.174276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886474+0000","last_change":"2024-04-22T01:17:41.884069+0000","last_active":"2024-04-22T01:17:42.886474+0000","last_peered":"2024-04-22T01:17:42.886474+0000","last_clean":"2024-04-22T01:17:42.886474+0000","last_became_active":"2024-04-22T01:17:41.883403+0000","last_became_peered":"2024-04-22T01:17:41.883403+0000","last_unstale":"2024-04-22T01:17:42.886474+0000","last_undegraded":"2024-04-22T01:17:42.886474+0000","last_fullsized":"2024-04-22T01:17:42.886474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:09.479666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.898355+0000","last_change":"2024-04-22T01:17:41.894670+0000","last_active":"2024-04-22T01:17:41.898355+0000","last_peered":"2024-04-22T01:17:41.898355+0000","last_clean":"2024-04-22T01:17:41.898355+0000","last_became_active":"2024-04-22T01:17:41.894437+0000","last_became_peered":"2024-04-22T01:17:41.894437+0000","last_unstale":"2024-04-22T01:17:41.898355+0000","last_undegraded":"2024-04-22T01:17:41.898355+0000","last_fullsized":"2024-04-22T01:17:41.898355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:00.723573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.890175+0000","last_change":"2024-04-22T01:17:41.887569+0000","last_active":"2024-04-22T01:17:41.890175+0000","last_peered":"2024-04-22T01:17:41.890175+0000","last_clean":"2024-04-22T01:17:41.890175+0000","last_became_active":"2024-04-22T01:17:41.887318+0000","last_became_peered":"2024-04-22T01:17:41.887318+0000","last_unstale":"2024-04-22T01:17:41.890175+0000","last_undegraded":"2024-04-22T01:17:41.890175+0000","last_fullsized":"2024-04-22T01:17:41.890175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:37.106102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892998+0000","last_change":"2024-04-22T01:17:41.889792+0000","last_active":"2024-04-22T01:17:41.892998+0000","last_peered":"2024-04-22T01:17:41.892998+0000","last_clean":"2024-04-22T01:17:41.892998+0000","last_became_active":"2024-04-22T01:17:41.889422+0000","last_became_peered":"2024-04-22T01:17:41.889422+0000","last_unstale":"2024-04-22T01:17:41.892998+0000","last_undegraded":"2024-04-22T01:17:41.892998+0000","last_fullsized":"2024-04-22T01:17:41.892998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:01.996527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.870078+0000","last_change":"2024-04-22T01:17:41.861470+0000","last_active":"2024-04-22T01:17:41.870078+0000","last_peered":"2024-04-22T01:17:41.870078+0000","last_clean":"2024-04-22T01:17:41.870078+0000","last_became_active":"2024-04-22T01:17:41.860680+0000","last_became_peered":"2024-04-22T01:17:41.860680+0000","last_unstale":"2024-04-22T01:17:41.870078+0000","last_undegraded":"2024-04-22T01:17:41.870078+0000","last_fullsized":"2024-04-22T01:17:41.870078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:59:32.206639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"77'4","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848641+0000","last_change":"2024-04-22T01:17:40.828053+0000","last_active":"2024-04-22T01:17:42.848641+0000","last_peered":"2024-04-22T01:17:42.848641+0000","last_clean":"2024-04-22T01:17:42.848641+0000","last_became_active":"2024-04-22T01:17:40.827707+0000","last_became_peered":"2024-04-22T01:17:40.827707+0000","last_unstale":"2024-04-22T01:17:42.848641+0000","last_undegraded":"2024-04-22T01:17:42.848641+0000","last_fullsized":"2024-04-22T01:17:42.848641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:53.437069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886420+0000","last_change":"2024-04-22T01:17:41.875686+0000","last_active":"2024-04-22T01:17:42.886420+0000","last_peered":"2024-04-22T01:17:42.886420+0000","last_clean":"2024-04-22T01:17:42.886420+0000","last_became_active":"2024-04-22T01:17:41.874372+0000","last_became_peered":"2024-04-22T01:17:41.874372+0000","last_unstale":"2024-04-22T01:17:42.886420+0000","last_undegraded":"2024-04-22T01:17:42.886420+0000","last_fullsized":"2024-04-22T01:17:42.886420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:20.111786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.888795+0000","last_change":"2024-04-22T01:17:40.848569+0000","last_active":"2024-04-22T01:17:41.888795+0000","last_peered":"2024-04-22T01:17:41.888795+0000","last_clean":"2024-04-22T01:17:41.888795+0000","last_became_active":"2024-04-22T01:17:40.846769+0000","last_became_peered":"2024-04-22T01:17:40.846769+0000","last_unstale":"2024-04-22T01:17:41.888795+0000","last_undegraded":"2024-04-22T01:17:41.888795+0000","last_fullsized":"2024-04-22T01:17:41.888795+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:40.246816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892205+0000","last_change":"2024-04-22T01:17:41.880848+0000","last_active":"2024-04-22T01:17:41.892205+0000","last_peered":"2024-04-22T01:17:41.892205+0000","last_clean":"2024-04-22T01:17:41.892205+0000","last_became_active":"2024-04-22T01:17:41.872876+0000","last_became_peered":"2024-04-22T01:17:41.872876+0000","last_unstale":"2024-04-22T01:17:41.892205+0000","last_undegraded":"2024-04-22T01:17:41.892205+0000","last_fullsized":"2024-04-22T01:17:41.892205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:22:14.961296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.899641+0000","last_change":"2024-04-22T01:17:40.840101+0000","last_active":"2024-04-22T01:17:41.899641+0000","last_peered":"2024-04-22T01:17:41.899641+0000","last_clean":"2024-04-22T01:17:41.899641+0000","last_became_active":"2024-04-22T01:17:40.839293+0000","last_became_peered":"2024-04-22T01:17:40.839293+0000","last_unstale":"2024-04-22T01:17:41.899641+0000","last_undegraded":"2024-04-22T01:17:41.899641+0000","last_fullsized":"2024-04-22T01:17:41.899641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:46:37.415042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889150+0000","last_change":"2024-04-22T01:17:41.872870+0000","last_active":"2024-04-22T01:17:41.889150+0000","last_peered":"2024-04-22T01:17:41.889150+0000","last_clean":"2024-04-22T01:17:41.889150+0000","last_became_active":"2024-04-22T01:17:41.872114+0000","last_became_peered":"2024-04-22T01:17:41.872114+0000","last_unstale":"2024-04-22T01:17:41.889150+0000","last_undegraded":"2024-04-22T01:17:41.889150+0000","last_fullsized":"2024-04-22T01:17:41.889150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:33.652052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849112+0000","last_change":"2024-04-22T01:17:40.845275+0000","last_active":"2024-04-22T01:17:42.849112+0000","last_peered":"2024-04-22T01:17:42.849112+0000","last_clean":"2024-04-22T01:17:42.849112+0000","last_became_active":"2024-04-22T01:17:40.844181+0000","last_became_peered":"2024-04-22T01:17:40.844181+0000","last_unstale":"2024-04-22T01:17:42.849112+0000","last_undegraded":"2024-04-22T01:17:42.849112+0000","last_fullsized":"2024-04-22T01:17:42.849112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:34:50.449570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886760+0000","last_change":"2024-04-22T01:17:41.884627+0000","last_active":"2024-04-22T01:17:42.886760+0000","last_peered":"2024-04-22T01:17:42.886760+0000","last_clean":"2024-04-22T01:17:42.886760+0000","last_became_active":"2024-04-22T01:17:41.884021+0000","last_became_peered":"2024-04-22T01:17:41.884021+0000","last_unstale":"2024-04-22T01:17:42.886760+0000","last_undegraded":"2024-04-22T01:17:42.886760+0000","last_fullsized":"2024-04-22T01:17:42.886760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:14.492576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892415+0000","last_change":"2024-04-22T01:17:40.831323+0000","last_active":"2024-04-22T01:17:41.892415+0000","last_peered":"2024-04-22T01:17:41.892415+0000","last_clean":"2024-04-22T01:17:41.892415+0000","last_became_active":"2024-04-22T01:17:40.830501+0000","last_became_peered":"2024-04-22T01:17:40.830501+0000","last_unstale":"2024-04-22T01:17:41.892415+0000","last_undegraded":"2024-04-22T01:17:41.892415+0000","last_fullsized":"2024-04-22T01:17:41.892415+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:51.683494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886696+0000","last_change":"2024-04-22T01:17:41.885506+0000","last_active":"2024-04-22T01:17:42.886696+0000","last_peered":"2024-04-22T01:17:42.886696+0000","last_clean":"2024-04-22T01:17:42.886696+0000","last_became_active":"2024-04-22T01:17:41.884009+0000","last_became_peered":"2024-04-22T01:17:41.884009+0000","last_unstale":"2024-04-22T01:17:42.886696+0000","last_undegraded":"2024-04-22T01:17:42.886696+0000","last_fullsized":"2024-04-22T01:17:42.886696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:16.418965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848531+0000","last_change":"2024-04-22T01:17:40.827513+0000","last_active":"2024-04-22T01:17:42.848531+0000","last_peered":"2024-04-22T01:17:42.848531+0000","last_clean":"2024-04-22T01:17:42.848531+0000","last_became_active":"2024-04-22T01:17:40.826741+0000","last_became_peered":"2024-04-22T01:17:40.826741+0000","last_unstale":"2024-04-22T01:17:42.848531+0000","last_undegraded":"2024-04-22T01:17:42.848531+0000","last_fullsized":"2024-04-22T01:17:42.848531+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:47:48.218080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892203+0000","last_change":"2024-04-22T01:17:41.878923+0000","last_active":"2024-04-22T01:17:41.892203+0000","last_peered":"2024-04-22T01:17:41.892203+0000","last_clean":"2024-04-22T01:17:41.892203+0000","last_became_active":"2024-04-22T01:17:41.878045+0000","last_became_peered":"2024-04-22T01:17:41.878045+0000","last_unstale":"2024-04-22T01:17:41.892203+0000","last_undegraded":"2024-04-22T01:17:41.892203+0000","last_fullsized":"2024-04-22T01:17:41.892203+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:24.410186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886728+0000","last_change":"2024-04-22T01:17:40.834316+0000","last_active":"2024-04-22T01:17:42.886728+0000","last_peered":"2024-04-22T01:17:42.886728+0000","last_clean":"2024-04-22T01:17:42.886728+0000","last_became_active":"2024-04-22T01:17:40.834033+0000","last_became_peered":"2024-04-22T01:17:40.834033+0000","last_unstale":"2024-04-22T01:17:42.886728+0000","last_undegraded":"2024-04-22T01:17:42.886728+0000","last_fullsized":"2024-04-22T01:17:42.886728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:14.457408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850361+0000","last_change":"2024-04-22T01:17:41.887477+0000","last_active":"2024-04-22T01:17:42.850361+0000","last_peered":"2024-04-22T01:17:42.850361+0000","last_clean":"2024-04-22T01:17:42.850361+0000","last_became_active":"2024-04-22T01:17:41.887147+0000","last_became_peered":"2024-04-22T01:17:41.887147+0000","last_unstale":"2024-04-22T01:17:42.850361+0000","last_undegraded":"2024-04-22T01:17:42.850361+0000","last_fullsized":"2024-04-22T01:17:42.850361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:45:41.159892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847485+0000","last_change":"2024-04-22T01:17:40.833502+0000","last_active":"2024-04-22T01:17:42.847485+0000","last_peered":"2024-04-22T01:17:42.847485+0000","last_clean":"2024-04-22T01:17:42.847485+0000","last_became_active":"2024-04-22T01:17:40.833162+0000","last_became_peered":"2024-04-22T01:17:40.833162+0000","last_unstale":"2024-04-22T01:17:42.847485+0000","last_undegraded":"2024-04-22T01:17:42.847485+0000","last_fullsized":"2024-04-22T01:17:42.847485+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:21.104552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886117+0000","last_change":"2024-04-22T01:17:41.874080+0000","last_active":"2024-04-22T01:17:42.886117+0000","last_peered":"2024-04-22T01:17:42.886117+0000","last_clean":"2024-04-22T01:17:42.886117+0000","last_became_active":"2024-04-22T01:17:41.873605+0000","last_became_peered":"2024-04-22T01:17:41.873605+0000","last_unstale":"2024-04-22T01:17:42.886117+0000","last_undegraded":"2024-04-22T01:17:42.886117+0000","last_fullsized":"2024-04-22T01:17:42.886117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:21:07.002244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.860297+0000","last_change":"2024-04-22T01:17:40.847444+0000","last_active":"2024-04-22T01:17:41.860297+0000","last_peered":"2024-04-22T01:17:41.860297+0000","last_clean":"2024-04-22T01:17:41.860297+0000","last_became_active":"2024-04-22T01:17:40.846762+0000","last_became_peered":"2024-04-22T01:17:40.846762+0000","last_unstale":"2024-04-22T01:17:41.860297+0000","last_undegraded":"2024-04-22T01:17:41.860297+0000","last_fullsized":"2024-04-22T01:17:41.860297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:11.544234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848947+0000","last_change":"2024-04-22T01:17:41.866658+0000","last_active":"2024-04-22T01:17:42.848947+0000","last_peered":"2024-04-22T01:17:42.848947+0000","last_clean":"2024-04-22T01:17:42.848947+0000","last_became_active":"2024-04-22T01:17:41.865952+0000","last_became_peered":"2024-04-22T01:17:41.865952+0000","last_unstale":"2024-04-22T01:17:42.848947+0000","last_undegraded":"2024-04-22T01:17:42.848947+0000","last_fullsized":"2024-04-22T01:17:42.848947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:05.493690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850066+0000","last_change":"2024-04-22T01:17:40.841354+0000","last_active":"2024-04-22T01:17:42.850066+0000","last_peered":"2024-04-22T01:17:42.850066+0000","last_clean":"2024-04-22T01:17:42.850066+0000","last_became_active":"2024-04-22T01:17:40.840513+0000","last_became_peered":"2024-04-22T01:17:40.840513+0000","last_unstale":"2024-04-22T01:17:42.850066+0000","last_undegraded":"2024-04-22T01:17:42.850066+0000","last_fullsized":"2024-04-22T01:17:42.850066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:45:40.156277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.872515+0000","last_change":"2024-04-22T01:17:41.869513+0000","last_active":"2024-04-22T01:17:41.872515+0000","last_peered":"2024-04-22T01:17:41.872515+0000","last_clean":"2024-04-22T01:17:41.872515+0000","last_became_active":"2024-04-22T01:17:41.868739+0000","last_became_peered":"2024-04-22T01:17:41.868739+0000","last_unstale":"2024-04-22T01:17:41.872515+0000","last_undegraded":"2024-04-22T01:17:41.872515+0000","last_fullsized":"2024-04-22T01:17:41.872515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:05.400978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848430+0000","last_change":"2024-04-22T01:17:40.833422+0000","last_active":"2024-04-22T01:17:42.848430+0000","last_peered":"2024-04-22T01:17:42.848430+0000","last_clean":"2024-04-22T01:17:42.848430+0000","last_became_active":"2024-04-22T01:17:40.833036+0000","last_became_peered":"2024-04-22T01:17:40.833036+0000","last_unstale":"2024-04-22T01:17:42.848430+0000","last_undegraded":"2024-04-22T01:17:42.848430+0000","last_fullsized":"2024-04-22T01:17:42.848430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:33.059281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.885727+0000","last_change":"2024-04-22T01:17:41.877650+0000","last_active":"2024-04-22T01:17:41.885727+0000","last_peered":"2024-04-22T01:17:41.885727+0000","last_clean":"2024-04-22T01:17:41.885727+0000","last_became_active":"2024-04-22T01:17:41.877369+0000","last_became_peered":"2024-04-22T01:17:41.877369+0000","last_unstale":"2024-04-22T01:17:41.885727+0000","last_undegraded":"2024-04-22T01:17:41.885727+0000","last_fullsized":"2024-04-22T01:17:41.885727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:53.908425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"77'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849380+0000","last_change":"2024-04-22T01:17:40.836475+0000","last_active":"2024-04-22T01:17:42.849380+0000","last_peered":"2024-04-22T01:17:42.849380+0000","last_clean":"2024-04-22T01:17:42.849380+0000","last_became_active":"2024-04-22T01:17:40.835921+0000","last_became_peered":"2024-04-22T01:17:40.835921+0000","last_unstale":"2024-04-22T01:17:42.849380+0000","last_undegraded":"2024-04-22T01:17:42.849380+0000","last_fullsized":"2024-04-22T01:17:42.849380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:21:16.138190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850216+0000","last_change":"2024-04-22T01:17:41.868936+0000","last_active":"2024-04-22T01:17:42.850216+0000","last_peered":"2024-04-22T01:17:42.850216+0000","last_clean":"2024-04-22T01:17:42.850216+0000","last_became_active":"2024-04-22T01:17:41.868198+0000","last_became_peered":"2024-04-22T01:17:41.868198+0000","last_unstale":"2024-04-22T01:17:42.850216+0000","last_undegraded":"2024-04-22T01:17:42.850216+0000","last_fullsized":"2024-04-22T01:17:42.850216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:34:39.563352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848326+0000","last_change":"2024-04-22T01:17:40.842649+0000","last_active":"2024-04-22T01:17:42.848326+0000","last_peered":"2024-04-22T01:17:42.848326+0000","last_clean":"2024-04-22T01:17:42.848326+0000","last_became_active":"2024-04-22T01:17:40.842050+0000","last_became_peered":"2024-04-22T01:17:40.842050+0000","last_unstale":"2024-04-22T01:17:42.848326+0000","last_undegraded":"2024-04-22T01:17:42.848326+0000","last_fullsized":"2024-04-22T01:17:42.848326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:16.993968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850440+0000","last_change":"2024-04-22T01:17:41.881798+0000","last_active":"2024-04-22T01:17:42.850440+0000","last_peered":"2024-04-22T01:17:42.850440+0000","last_clean":"2024-04-22T01:17:42.850440+0000","last_became_active":"2024-04-22T01:17:41.881417+0000","last_became_peered":"2024-04-22T01:17:41.881417+0000","last_unstale":"2024-04-22T01:17:42.850440+0000","last_undegraded":"2024-04-22T01:17:42.850440+0000","last_fullsized":"2024-04-22T01:17:42.850440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:30.648935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.859254+0000","last_change":"2024-04-22T01:17:40.843316+0000","last_active":"2024-04-22T01:17:41.859254+0000","last_peered":"2024-04-22T01:17:41.859254+0000","last_clean":"2024-04-22T01:17:41.859254+0000","last_became_active":"2024-04-22T01:17:40.842543+0000","last_became_peered":"2024-04-22T01:17:40.842543+0000","last_unstale":"2024-04-22T01:17:41.859254+0000","last_undegraded":"2024-04-22T01:17:41.859254+0000","last_fullsized":"2024-04-22T01:17:41.859254+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:14.457408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850061+0000","last_change":"2024-04-22T01:17:41.894050+0000","last_active":"2024-04-22T01:17:42.850061+0000","last_peered":"2024-04-22T01:17:42.850061+0000","last_clean":"2024-04-22T01:17:42.850061+0000","last_became_active":"2024-04-22T01:17:41.893759+0000","last_became_peered":"2024-04-22T01:17:41.893759+0000","last_unstale":"2024-04-22T01:17:42.850061+0000","last_undegraded":"2024-04-22T01:17:42.850061+0000","last_fullsized":"2024-04-22T01:17:42.850061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:08.613173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847743+0000","last_change":"2024-04-22T01:17:40.828400+0000","last_active":"2024-04-22T01:17:42.847743+0000","last_peered":"2024-04-22T01:17:42.847743+0000","last_clean":"2024-04-22T01:17:42.847743+0000","last_became_active":"2024-04-22T01:17:40.827685+0000","last_became_peered":"2024-04-22T01:17:40.827685+0000","last_unstale":"2024-04-22T01:17:42.847743+0000","last_undegraded":"2024-04-22T01:17:42.847743+0000","last_fullsized":"2024-04-22T01:17:42.847743+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:33:18.254370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850475+0000","last_change":"2024-04-22T01:17:41.885754+0000","last_active":"2024-04-22T01:17:42.850475+0000","last_peered":"2024-04-22T01:17:42.850475+0000","last_clean":"2024-04-22T01:17:42.850475+0000","last_became_active":"2024-04-22T01:17:41.885378+0000","last_became_peered":"2024-04-22T01:17:41.885378+0000","last_unstale":"2024-04-22T01:17:42.850475+0000","last_undegraded":"2024-04-22T01:17:42.850475+0000","last_fullsized":"2024-04-22T01:17:42.850475+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:26.530451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886239+0000","last_change":"2024-04-22T01:17:40.847699+0000","last_active":"2024-04-22T01:17:42.886239+0000","last_peered":"2024-04-22T01:17:42.886239+0000","last_clean":"2024-04-22T01:17:42.886239+0000","last_became_active":"2024-04-22T01:17:40.847393+0000","last_became_peered":"2024-04-22T01:17:40.847393+0000","last_unstale":"2024-04-22T01:17:42.886239+0000","last_undegraded":"2024-04-22T01:17:42.886239+0000","last_fullsized":"2024-04-22T01:17:42.886239+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:25.526836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.882630+0000","last_change":"2024-04-22T01:17:41.880855+0000","last_active":"2024-04-22T01:17:41.882630+0000","last_peered":"2024-04-22T01:17:41.882630+0000","last_clean":"2024-04-22T01:17:41.882630+0000","last_became_active":"2024-04-22T01:17:41.880209+0000","last_became_peered":"2024-04-22T01:17:41.880209+0000","last_unstale":"2024-04-22T01:17:41.882630+0000","last_undegraded":"2024-04-22T01:17:41.882630+0000","last_fullsized":"2024-04-22T01:17:41.882630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:15.283639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889866+0000","last_change":"2024-04-22T01:17:40.839274+0000","last_active":"2024-04-22T01:17:41.889866+0000","last_peered":"2024-04-22T01:17:41.889866+0000","last_clean":"2024-04-22T01:17:41.889866+0000","last_became_active":"2024-04-22T01:17:40.837914+0000","last_became_peered":"2024-04-22T01:17:40.837914+0000","last_unstale":"2024-04-22T01:17:41.889866+0000","last_undegraded":"2024-04-22T01:17:41.889866+0000","last_fullsized":"2024-04-22T01:17:41.889866+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:42.567223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847498+0000","last_change":"2024-04-22T01:17:41.888298+0000","last_active":"2024-04-22T01:17:42.847498+0000","last_peered":"2024-04-22T01:17:42.847498+0000","last_clean":"2024-04-22T01:17:42.847498+0000","last_became_active":"2024-04-22T01:17:41.887999+0000","last_became_peered":"2024-04-22T01:17:41.887999+0000","last_unstale":"2024-04-22T01:17:42.847498+0000","last_undegraded":"2024-04-22T01:17:42.847498+0000","last_fullsized":"2024-04-22T01:17:42.847498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:25.245611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.893947+0000","last_change":"2024-04-22T01:17:40.833472+0000","last_active":"2024-04-22T01:17:41.893947+0000","last_peered":"2024-04-22T01:17:41.893947+0000","last_clean":"2024-04-22T01:17:41.893947+0000","last_became_active":"2024-04-22T01:17:40.831399+0000","last_became_peered":"2024-04-22T01:17:40.831399+0000","last_unstale":"2024-04-22T01:17:41.893947+0000","last_undegraded":"2024-04-22T01:17:41.893947+0000","last_fullsized":"2024-04-22T01:17:41.893947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:50.396784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847890+0000","last_change":"2024-04-22T01:17:41.877959+0000","last_active":"2024-04-22T01:17:42.847890+0000","last_peered":"2024-04-22T01:17:42.847890+0000","last_clean":"2024-04-22T01:17:42.847890+0000","last_became_active":"2024-04-22T01:17:41.877408+0000","last_became_peered":"2024-04-22T01:17:41.877408+0000","last_unstale":"2024-04-22T01:17:42.847890+0000","last_undegraded":"2024-04-22T01:17:42.847890+0000","last_fullsized":"2024-04-22T01:17:42.847890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:23.996559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848263+0000","last_change":"2024-04-22T01:17:40.836219+0000","last_active":"2024-04-22T01:17:42.848263+0000","last_peered":"2024-04-22T01:17:42.848263+0000","last_clean":"2024-04-22T01:17:42.848263+0000","last_became_active":"2024-04-22T01:17:40.835727+0000","last_became_peered":"2024-04-22T01:17:40.835727+0000","last_unstale":"2024-04-22T01:17:42.848263+0000","last_undegraded":"2024-04-22T01:17:42.848263+0000","last_fullsized":"2024-04-22T01:17:42.848263+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:35.561611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.872127+0000","last_change":"2024-04-22T01:17:41.867855+0000","last_active":"2024-04-22T01:17:41.872127+0000","last_peered":"2024-04-22T01:17:41.872127+0000","last_clean":"2024-04-22T01:17:41.872127+0000","last_became_active":"2024-04-22T01:17:41.867055+0000","last_became_peered":"2024-04-22T01:17:41.867055+0000","last_unstale":"2024-04-22T01:17:41.872127+0000","last_undegraded":"2024-04-22T01:17:41.872127+0000","last_fullsized":"2024-04-22T01:17:41.872127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:43.570838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.839959+0000","last_change":"2024-04-22T01:17:40.839500+0000","last_active":"2024-04-22T01:17:41.839959+0000","last_peered":"2024-04-22T01:17:41.839959+0000","last_clean":"2024-04-22T01:17:41.839959+0000","last_became_active":"2024-04-22T01:17:40.839108+0000","last_became_peered":"2024-04-22T01:17:40.839108+0000","last_unstale":"2024-04-22T01:17:41.839959+0000","last_undegraded":"2024-04-22T01:17:41.839959+0000","last_fullsized":"2024-04-22T01:17:41.839959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:18.243611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886743+0000","last_change":"2024-04-22T01:17:41.873055+0000","last_active":"2024-04-22T01:17:42.886743+0000","last_peered":"2024-04-22T01:17:42.886743+0000","last_clean":"2024-04-22T01:17:42.886743+0000","last_became_active":"2024-04-22T01:17:41.867148+0000","last_became_peered":"2024-04-22T01:17:41.867148+0000","last_unstale":"2024-04-22T01:17:42.886743+0000","last_undegraded":"2024-04-22T01:17:42.886743+0000","last_fullsized":"2024-04-22T01:17:42.886743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:23:54.290857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.898219+0000","last_change":"2024-04-22T01:17:40.829490+0000","last_active":"2024-04-22T01:17:41.898219+0000","last_peered":"2024-04-22T01:17:41.898219+0000","last_clean":"2024-04-22T01:17:41.898219+0000","last_became_active":"2024-04-22T01:17:40.829131+0000","last_became_peered":"2024-04-22T01:17:40.829131+0000","last_unstale":"2024-04-22T01:17:41.898219+0000","last_undegraded":"2024-04-22T01:17:41.898219+0000","last_fullsized":"2024-04-22T01:17:41.898219+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:14.457408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886828+0000","last_change":"2024-04-22T01:17:41.898727+0000","last_active":"2024-04-22T01:17:42.886828+0000","last_peered":"2024-04-22T01:17:42.886828+0000","last_clean":"2024-04-22T01:17:42.886828+0000","last_became_active":"2024-04-22T01:17:41.894191+0000","last_became_peered":"2024-04-22T01:17:41.894191+0000","last_unstale":"2024-04-22T01:17:42.886828+0000","last_undegraded":"2024-04-22T01:17:42.886828+0000","last_fullsized":"2024-04-22T01:17:42.886828+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:06.191633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.848137+0000","last_change":"2024-04-22T01:17:40.841281+0000","last_active":"2024-04-22T01:17:41.848137+0000","last_peered":"2024-04-22T01:17:41.848137+0000","last_clean":"2024-04-22T01:17:41.848137+0000","last_became_active":"2024-04-22T01:17:40.840844+0000","last_became_peered":"2024-04-22T01:17:40.840844+0000","last_unstale":"2024-04-22T01:17:41.848137+0000","last_undegraded":"2024-04-22T01:17:41.848137+0000","last_fullsized":"2024-04-22T01:17:41.848137+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:46:37.415042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850914+0000","last_change":"2024-04-22T01:17:41.885013+0000","last_active":"2024-04-22T01:17:42.850914+0000","last_peered":"2024-04-22T01:17:42.850914+0000","last_clean":"2024-04-22T01:17:42.850914+0000","last_became_active":"2024-04-22T01:17:41.884629+0000","last_became_peered":"2024-04-22T01:17:41.884629+0000","last_unstale":"2024-04-22T01:17:42.850914+0000","last_undegraded":"2024-04-22T01:17:42.850914+0000","last_fullsized":"2024-04-22T01:17:42.850914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:15.461024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.848066+0000","last_change":"2024-04-22T01:17:40.840312+0000","last_active":"2024-04-22T01:17:41.848066+0000","last_peered":"2024-04-22T01:17:41.848066+0000","last_clean":"2024-04-22T01:17:41.848066+0000","last_became_active":"2024-04-22T01:17:40.840017+0000","last_became_peered":"2024-04-22T01:17:40.840017+0000","last_unstale":"2024-04-22T01:17:41.848066+0000","last_undegraded":"2024-04-22T01:17:41.848066+0000","last_fullsized":"2024-04-22T01:17:41.848066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:40.246816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892570+0000","last_change":"2024-04-22T01:17:41.887240+0000","last_active":"2024-04-22T01:17:41.892570+0000","last_peered":"2024-04-22T01:17:41.892570+0000","last_clean":"2024-04-22T01:17:41.892570+0000","last_became_active":"2024-04-22T01:17:41.886905+0000","last_became_peered":"2024-04-22T01:17:41.886905+0000","last_unstale":"2024-04-22T01:17:41.892570+0000","last_undegraded":"2024-04-22T01:17:41.892570+0000","last_fullsized":"2024-04-22T01:17:41.892570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:12:27.511810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848863+0000","last_change":"2024-04-22T01:17:40.829125+0000","last_active":"2024-04-22T01:17:42.848863+0000","last_peered":"2024-04-22T01:17:42.848863+0000","last_clean":"2024-04-22T01:17:42.848863+0000","last_became_active":"2024-04-22T01:17:40.828255+0000","last_became_peered":"2024-04-22T01:17:40.828255+0000","last_unstale":"2024-04-22T01:17:42.848863+0000","last_undegraded":"2024-04-22T01:17:42.848863+0000","last_fullsized":"2024-04-22T01:17:42.848863+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:58:51.696063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850139+0000","last_change":"2024-04-22T01:17:41.882276+0000","last_active":"2024-04-22T01:17:42.850139+0000","last_peered":"2024-04-22T01:17:42.850139+0000","last_clean":"2024-04-22T01:17:42.850139+0000","last_became_active":"2024-04-22T01:17:41.881733+0000","last_became_peered":"2024-04-22T01:17:41.881733+0000","last_unstale":"2024-04-22T01:17:42.850139+0000","last_undegraded":"2024-04-22T01:17:42.850139+0000","last_fullsized":"2024-04-22T01:17:42.850139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:52.460357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"78'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.888484+0000","last_change":"2024-04-22T01:17:40.828678+0000","last_active":"2024-04-22T01:17:42.888484+0000","last_peered":"2024-04-22T01:17:42.888484+0000","last_clean":"2024-04-22T01:17:42.888484+0000","last_became_active":"2024-04-22T01:17:40.828297+0000","last_became_peered":"2024-04-22T01:17:40.828297+0000","last_unstale":"2024-04-22T01:17:42.888484+0000","last_undegraded":"2024-04-22T01:17:42.888484+0000","last_fullsized":"2024-04-22T01:17:42.888484+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:40.246816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892154+0000","last_change":"2024-04-22T01:17:41.873726+0000","last_active":"2024-04-22T01:17:41.892154+0000","last_peered":"2024-04-22T01:17:41.892154+0000","last_clean":"2024-04-22T01:17:41.892154+0000","last_became_active":"2024-04-22T01:17:41.871866+0000","last_became_peered":"2024-04-22T01:17:41.871866+0000","last_unstale":"2024-04-22T01:17:41.892154+0000","last_undegraded":"2024-04-22T01:17:41.892154+0000","last_fullsized":"2024-04-22T01:17:41.892154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:50.588832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.860314+0000","last_change":"2024-04-22T01:17:40.848697+0000","last_active":"2024-04-22T01:17:41.860314+0000","last_peered":"2024-04-22T01:17:41.860314+0000","last_clean":"2024-04-22T01:17:41.860314+0000","last_became_active":"2024-04-22T01:17:40.848084+0000","last_became_peered":"2024-04-22T01:17:40.848084+0000","last_unstale":"2024-04-22T01:17:41.860314+0000","last_undegraded":"2024-04-22T01:17:41.860314+0000","last_fullsized":"2024-04-22T01:17:41.860314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:04.490075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886697+0000","last_change":"2024-04-22T01:17:41.873843+0000","last_active":"2024-04-22T01:17:42.886697+0000","last_peered":"2024-04-22T01:17:42.886697+0000","last_clean":"2024-04-22T01:17:42.886697+0000","last_became_active":"2024-04-22T01:17:41.873524+0000","last_became_peered":"2024-04-22T01:17:41.873524+0000","last_unstale":"2024-04-22T01:17:42.886697+0000","last_undegraded":"2024-04-22T01:17:42.886697+0000","last_fullsized":"2024-04-22T01:17:42.886697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:37.293915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.858819+0000","last_change":"2024-04-22T01:17:40.847822+0000","last_active":"2024-04-22T01:17:41.858819+0000","last_peered":"2024-04-22T01:17:41.858819+0000","last_clean":"2024-04-22T01:17:41.858819+0000","last_became_active":"2024-04-22T01:17:40.847176+0000","last_became_peered":"2024-04-22T01:17:40.847176+0000","last_unstale":"2024-04-22T01:17:41.858819+0000","last_undegraded":"2024-04-22T01:17:41.858819+0000","last_fullsized":"2024-04-22T01:17:41.858819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:25.526836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850237+0000","last_change":"2024-04-22T01:17:41.882199+0000","last_active":"2024-04-22T01:17:42.850237+0000","last_peered":"2024-04-22T01:17:42.850237+0000","last_clean":"2024-04-22T01:17:42.850237+0000","last_became_active":"2024-04-22T01:17:41.881614+0000","last_became_peered":"2024-04-22T01:17:41.881614+0000","last_unstale":"2024-04-22T01:17:42.850237+0000","last_undegraded":"2024-04-22T01:17:42.850237+0000","last_fullsized":"2024-04-22T01:17:42.850237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:17.997584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848731+0000","last_change":"2024-04-22T01:17:40.844114+0000","last_active":"2024-04-22T01:17:42.848731+0000","last_peered":"2024-04-22T01:17:42.848731+0000","last_clean":"2024-04-22T01:17:42.848731+0000","last_became_active":"2024-04-22T01:17:40.839294+0000","last_became_peered":"2024-04-22T01:17:40.839294+0000","last_unstale":"2024-04-22T01:17:42.848731+0000","last_undegraded":"2024-04-22T01:17:42.848731+0000","last_fullsized":"2024-04-22T01:17:42.848731+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:12:26.508195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850139+0000","last_change":"2024-04-22T01:17:41.892576+0000","last_active":"2024-04-22T01:17:42.850139+0000","last_peered":"2024-04-22T01:17:42.850139+0000","last_clean":"2024-04-22T01:17:42.850139+0000","last_became_active":"2024-04-22T01:17:41.892262+0000","last_became_peered":"2024-04-22T01:17:41.892262+0000","last_unstale":"2024-04-22T01:17:42.850139+0000","last_undegraded":"2024-04-22T01:17:42.850139+0000","last_fullsized":"2024-04-22T01:17:42.850139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:47:49.221695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848083+0000","last_change":"2024-04-22T01:17:40.830460+0000","last_active":"2024-04-22T01:17:42.848083+0000","last_peered":"2024-04-22T01:17:42.848083+0000","last_clean":"2024-04-22T01:17:42.848083+0000","last_became_active":"2024-04-22T01:17:40.829847+0000","last_became_peered":"2024-04-22T01:17:40.829847+0000","last_unstale":"2024-04-22T01:17:42.848083+0000","last_undegraded":"2024-04-22T01:17:42.848083+0000","last_fullsized":"2024-04-22T01:17:42.848083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:25.341958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.885995+0000","last_change":"2024-04-22T01:17:41.879127+0000","last_active":"2024-04-22T01:17:41.885995+0000","last_peered":"2024-04-22T01:17:41.885995+0000","last_clean":"2024-04-22T01:17:41.885995+0000","last_became_active":"2024-04-22T01:17:41.878595+0000","last_became_peered":"2024-04-22T01:17:41.878595+0000","last_unstale":"2024-04-22T01:17:41.885995+0000","last_undegraded":"2024-04-22T01:17:41.885995+0000","last_fullsized":"2024-04-22T01:17:41.885995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:44.717342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850606+0000","last_change":"2024-04-22T01:17:40.838793+0000","last_active":"2024-04-22T01:17:42.850606+0000","last_peered":"2024-04-22T01:17:42.850606+0000","last_clean":"2024-04-22T01:17:42.850606+0000","last_became_active":"2024-04-22T01:17:40.837776+0000","last_became_peered":"2024-04-22T01:17:40.837776+0000","last_unstale":"2024-04-22T01:17:42.850606+0000","last_undegraded":"2024-04-22T01:17:42.850606+0000","last_fullsized":"2024-04-22T01:17:42.850606+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:13.336740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.873352+0000","last_change":"2024-04-22T01:17:41.869252+0000","last_active":"2024-04-22T01:17:41.873352+0000","last_peered":"2024-04-22T01:17:41.873352+0000","last_clean":"2024-04-22T01:17:41.873352+0000","last_became_active":"2024-04-22T01:17:41.868563+0000","last_became_peered":"2024-04-22T01:17:41.868563+0000","last_unstale":"2024-04-22T01:17:41.873352+0000","last_undegraded":"2024-04-22T01:17:41.873352+0000","last_fullsized":"2024-04-22T01:17:41.873352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:50.843950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.839841+0000","last_change":"2024-04-22T01:17:40.832593+0000","last_active":"2024-04-22T01:17:41.839841+0000","last_peered":"2024-04-22T01:17:41.839841+0000","last_clean":"2024-04-22T01:17:41.839841+0000","last_became_active":"2024-04-22T01:17:40.830928+0000","last_became_peered":"2024-04-22T01:17:40.830928+0000","last_unstale":"2024-04-22T01:17:41.839841+0000","last_undegraded":"2024-04-22T01:17:41.839841+0000","last_fullsized":"2024-04-22T01:17:41.839841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:39.704746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.877088+0000","last_change":"2024-04-22T01:17:41.872198+0000","last_active":"2024-04-22T01:17:41.877088+0000","last_peered":"2024-04-22T01:17:41.877088+0000","last_clean":"2024-04-22T01:17:41.877088+0000","last_became_active":"2024-04-22T01:17:41.871402+0000","last_became_peered":"2024-04-22T01:17:41.871402+0000","last_unstale":"2024-04-22T01:17:41.877088+0000","last_undegraded":"2024-04-22T01:17:41.877088+0000","last_fullsized":"2024-04-22T01:17:41.877088+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:15.283639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.848860+0000","last_change":"2024-04-22T01:17:40.840871+0000","last_active":"2024-04-22T01:17:41.848860+0000","last_peered":"2024-04-22T01:17:41.848860+0000","last_clean":"2024-04-22T01:17:41.848860+0000","last_became_active":"2024-04-22T01:17:40.840540+0000","last_became_peered":"2024-04-22T01:17:40.840540+0000","last_unstale":"2024-04-22T01:17:41.848860+0000","last_undegraded":"2024-04-22T01:17:41.848860+0000","last_fullsized":"2024-04-22T01:17:41.848860+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:11.544234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":270,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.858827+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:17:41.858827+0000","last_peered":"2024-04-22T01:17:41.858827+0000","last_clean":"2024-04-22T01:17:41.858827+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:17:41.858827+0000","last_undegraded":"2024-04-22T01:17:41.858827+0000","last_fullsized":"2024-04-22T01:17:41.858827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1540096,"data_stored":1019554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":437468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":435890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":962560,"data_stored":430880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36268,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335572,"statfs":{"total":96636764160,"available":96599625728,"internally_reserved":0,"allocated":946176,"data_stored":429746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36292,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335548,"statfs":{"total":96636764160,"available":96599601152,"internally_reserved":0,"allocated":966656,"data_stored":436598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335568,"statfs":{"total":96636764160,"available":96599621632,"internally_reserved":0,"allocated":950272,"data_stored":429276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1548288,"data_stored":1026224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:17:44.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:17:45.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 Processor -- start 2024-04-22T01:17:45.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 -- start start 2024-04-22T01:17:45.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4a00cc150 0x7fc4a0071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a0072060 0x7fc4a006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a00cd7b0 con 0x7fc4a01455b0 2024-04-22T01:17:45.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a00724d0 con 0x7fc4a00cc150 2024-04-22T01:17:45.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a8446640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a00cc710 con 0x7fc4a0072060 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a59e1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4a00cc150 0x7fc4a0071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a59e1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4a00cc150 0x7fc4a0071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44164/0 (socket says 172.21.15.70:44164) 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.077+0000 7fc4a59e1640 1 -- 172.21.15.70:0/103449130 learned_addr learned my addr 172.21.15.70:0/103449130 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a59e1640 1 -- 172.21.15.70:0/103449130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a0072060 msgr2=0x7fc4a006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a69e3640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a0072060 0x7fc4a006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a59e1640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a0072060 0x7fc4a006e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a59e1640 1 -- 172.21.15.70:0/103449130 >> v1:172.21.15.98:6789/0 conn(0x7fc4a01455b0 legacy=0x7fc4a00cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a59e1640 1 -- 172.21.15.70:0/103449130 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a00cc9f0 con 0x7fc4a00cc150 2024-04-22T01:17:45.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a69e3640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a0072060 0x7fc4a006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:45.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a59e1640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a00cc150 0x7fc4a0071b20 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc494009870 tx=0x7fc49401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1adff63c5fe704b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a51e0640 1 -- 172.21.15.70:0/103449130 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940b0070 con 0x7fc4a00cc150 2024-04-22T01:17:45.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a51e0640 1 -- 172.21.15.70:0/103449130 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4940a4e50 con 0x7fc4a00cc150 2024-04-22T01:17:45.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a51e0640 1 -- 172.21.15.70:0/103449130 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940ab820 con 0x7fc4a00cc150 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 -- 172.21.15.70:0/103449130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a00cc150 msgr2=0x7fc4a0071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a00cc150 0x7fc4a0071b20 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc494009870 tx=0x7fc49401c400 comp rx=0 tx=0).stop 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 -- 172.21.15.70:0/103449130 shutdown_connections 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a0072060 0x7fc4a006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 --2- 172.21.15.70:0/103449130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a00cc150 0x7fc4a0071b20 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 -- 172.21.15.70:0/103449130 >> 172.21.15.70:0/103449130 conn(0x7fc4a006b1f0 msgr2=0x7fc4a006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:45.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 -- 172.21.15.70:0/103449130 shutdown_connections 2024-04-22T01:17:45.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.081+0000 7fc4a8446640 1 -- 172.21.15.70:0/103449130 wait complete. 2024-04-22T01:17:45.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 Processor -- start 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- start start 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 0x7fc4a0139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a00cc150 0x7fc4a015ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a01455b0 0x7fc4a013c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a006f320 con 0x7fc4a0072060 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a006f1a0 con 0x7fc4a00cc150 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a0168b40 con 0x7fc4a01455b0 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 0x7fc4a0139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 0x7fc4a0139560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44172/0 (socket says 172.21.15.70:44172) 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 -- 172.21.15.70:0/1488711340 learned_addr learned my addr 172.21.15.70:0/1488711340 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a59e1640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a00cc150 0x7fc4a015ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 -- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a01455b0 msgr2=0x7fc4a013c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a01455b0 0x7fc4a013c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 -- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a00cc150 msgr2=0x7fc4a015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a00cc150 0x7fc4a015ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 -- 172.21.15.70:0/1488711340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a0139aa0 con 0x7fc4a0072060 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a61e2640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 0x7fc4a0139560 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc49000ba80 tx=0x7fc49000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49000cc00 con 0x7fc4a0072060 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc49000cda0 con 0x7fc4a0072060 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490003bc0 con 0x7fc4a0072060 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a015d790 con 0x7fc4a0072060 2024-04-22T01:17:45.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a015dd50 con 0x7fc4a0072060 2024-04-22T01:17:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a0068210 con 0x7fc4a0072060 2024-04-22T01:17:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc490006d10 con 0x7fc4a0072060 2024-04-22T01:17:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc49effd640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc48c0678c0 0x7fc48c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fc490085aa0 con 0x7fc4a0072060 2024-04-22T01:17:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.085+0000 7fc4a59e1640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc48c0678c0 0x7fc48c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.089+0000 7fc4a59e1640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc48c0678c0 0x7fc48c069d80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ad000 tx=0x7fc494008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.089+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc490056450 con 0x7fc4a0072060 2024-04-22T01:17:45.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.245+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4a015f510 con 0x7fc48c0678c0 2024-04-22T01:17:45.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.253+0000 7fc49effd640 1 -- 172.21.15.70:0/1488711340 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252060 (secure 0 0 0) 0x7fc4a015f510 con 0x7fc48c0678c0 2024-04-22T01:17:45.257 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:45.258 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc48c0678c0 msgr2=0x7fc48c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc48c0678c0 0x7fc48c069d80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ad000 tx=0x7fc494008040 comp rx=0 tx=0).stop 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 msgr2=0x7fc4a0139560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 0x7fc4a0139560 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc49000ba80 tx=0x7fc49000bf40 comp rx=0 tx=0).stop 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 shutdown_connections 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc48c0678c0 0x7fc48c069d80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4a01455b0 0x7fc4a013c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a00cc150 0x7fc4a015ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 --2- 172.21.15.70:0/1488711340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a0072060 0x7fc4a0139560 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 >> 172.21.15.70:0/1488711340 conn(0x7fc4a006b1f0 msgr2=0x7fc4a013fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:45.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 shutdown_connections 2024-04-22T01:17:45.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.257+0000 7fc4a8446640 1 -- 172.21.15.70:0/1488711340 wait complete. 2024-04-22T01:17:45.274 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-22T01:17:44.640283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262260,"kb_used_data":8632,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712460,"statfs":{"total":773094113280,"available":772825559040,"internally_reserved":0,"allocated":8839168,"data_stored":4645636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.840015"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892972+0000","last_change":"2024-04-22T01:17:41.889762+0000","last_active":"2024-04-22T01:17:41.892972+0000","last_peered":"2024-04-22T01:17:41.892972+0000","last_clean":"2024-04-22T01:17:41.892972+0000","last_became_active":"2024-04-22T01:17:41.889429+0000","last_became_peered":"2024-04-22T01:17:41.889429+0000","last_unstale":"2024-04-22T01:17:41.892972+0000","last_undegraded":"2024-04-22T01:17:41.892972+0000","last_fullsized":"2024-04-22T01:17:41.892972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:34:39.563352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886738+0000","last_change":"2024-04-22T01:17:41.893467+0000","last_active":"2024-04-22T01:17:42.886738+0000","last_peered":"2024-04-22T01:17:42.886738+0000","last_clean":"2024-04-22T01:17:42.886738+0000","last_became_active":"2024-04-22T01:17:41.893013+0000","last_became_peered":"2024-04-22T01:17:41.893013+0000","last_unstale":"2024-04-22T01:17:42.886738+0000","last_undegraded":"2024-04-22T01:17:42.886738+0000","last_fullsized":"2024-04-22T01:17:42.886738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:59:24.841737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.885863+0000","last_change":"2024-04-22T01:17:41.878734+0000","last_active":"2024-04-22T01:17:41.885863+0000","last_peered":"2024-04-22T01:17:41.885863+0000","last_clean":"2024-04-22T01:17:41.885863+0000","last_became_active":"2024-04-22T01:17:41.878237+0000","last_became_peered":"2024-04-22T01:17:41.878237+0000","last_unstale":"2024-04-22T01:17:41.885863+0000","last_undegraded":"2024-04-22T01:17:41.885863+0000","last_fullsized":"2024-04-22T01:17:41.885863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:58.237820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849447+0000","last_change":"2024-04-22T01:17:41.867063+0000","last_active":"2024-04-22T01:17:42.849447+0000","last_peered":"2024-04-22T01:17:42.849447+0000","last_clean":"2024-04-22T01:17:42.849447+0000","last_became_active":"2024-04-22T01:17:41.866349+0000","last_became_peered":"2024-04-22T01:17:41.866349+0000","last_unstale":"2024-04-22T01:17:42.849447+0000","last_undegraded":"2024-04-22T01:17:42.849447+0000","last_fullsized":"2024-04-22T01:17:42.849447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:05.400978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847908+0000","last_change":"2024-04-22T01:17:41.876173+0000","last_active":"2024-04-22T01:17:42.847908+0000","last_peered":"2024-04-22T01:17:42.847908+0000","last_clean":"2024-04-22T01:17:42.847908+0000","last_became_active":"2024-04-22T01:17:41.875573+0000","last_became_peered":"2024-04-22T01:17:41.875573+0000","last_unstale":"2024-04-22T01:17:42.847908+0000","last_undegraded":"2024-04-22T01:17:42.847908+0000","last_fullsized":"2024-04-22T01:17:42.847908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:34.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892004+0000","last_change":"2024-04-22T01:17:41.891143+0000","last_active":"2024-04-22T01:17:41.892004+0000","last_peered":"2024-04-22T01:17:41.892004+0000","last_clean":"2024-04-22T01:17:41.892004+0000","last_became_active":"2024-04-22T01:17:41.890146+0000","last_became_peered":"2024-04-22T01:17:41.890146+0000","last_unstale":"2024-04-22T01:17:41.892004+0000","last_undegraded":"2024-04-22T01:17:41.892004+0000","last_fullsized":"2024-04-22T01:17:41.892004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:34:39.563352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850355+0000","last_change":"2024-04-22T01:17:41.895449+0000","last_active":"2024-04-22T01:17:42.850355+0000","last_peered":"2024-04-22T01:17:42.850355+0000","last_clean":"2024-04-22T01:17:42.850355+0000","last_became_active":"2024-04-22T01:17:41.893334+0000","last_became_peered":"2024-04-22T01:17:41.893334+0000","last_unstale":"2024-04-22T01:17:42.850355+0000","last_undegraded":"2024-04-22T01:17:42.850355+0000","last_fullsized":"2024-04-22T01:17:42.850355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:58:52.699678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848084+0000","last_change":"2024-04-22T01:17:41.877758+0000","last_active":"2024-04-22T01:17:42.848084+0000","last_peered":"2024-04-22T01:17:42.848084+0000","last_clean":"2024-04-22T01:17:42.848084+0000","last_became_active":"2024-04-22T01:17:41.877271+0000","last_became_peered":"2024-04-22T01:17:41.877271+0000","last_unstale":"2024-04-22T01:17:42.848084+0000","last_undegraded":"2024-04-22T01:17:42.848084+0000","last_fullsized":"2024-04-22T01:17:42.848084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:32:52.914497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889091+0000","last_change":"2024-04-22T01:17:41.872477+0000","last_active":"2024-04-22T01:17:41.889091+0000","last_peered":"2024-04-22T01:17:41.889091+0000","last_clean":"2024-04-22T01:17:41.889091+0000","last_became_active":"2024-04-22T01:17:41.871772+0000","last_became_peered":"2024-04-22T01:17:41.871772+0000","last_unstale":"2024-04-22T01:17:41.889091+0000","last_undegraded":"2024-04-22T01:17:41.889091+0000","last_fullsized":"2024-04-22T01:17:41.889091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:57:45.951103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849420+0000","last_change":"2024-04-22T01:17:41.882229+0000","last_active":"2024-04-22T01:17:42.849420+0000","last_peered":"2024-04-22T01:17:42.849420+0000","last_clean":"2024-04-22T01:17:42.849420+0000","last_became_active":"2024-04-22T01:17:41.881057+0000","last_became_peered":"2024-04-22T01:17:41.881057+0000","last_unstale":"2024-04-22T01:17:42.849420+0000","last_undegraded":"2024-04-22T01:17:42.849420+0000","last_fullsized":"2024-04-22T01:17:42.849420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:43.570838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886598+0000","last_change":"2024-04-22T01:17:41.893467+0000","last_active":"2024-04-22T01:17:42.886598+0000","last_peered":"2024-04-22T01:17:42.886598+0000","last_clean":"2024-04-22T01:17:42.886598+0000","last_became_active":"2024-04-22T01:17:41.884806+0000","last_became_peered":"2024-04-22T01:17:41.884806+0000","last_unstale":"2024-04-22T01:17:42.886598+0000","last_undegraded":"2024-04-22T01:17:42.886598+0000","last_fullsized":"2024-04-22T01:17:42.886598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:54:42.629423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849840+0000","last_change":"2024-04-22T01:17:41.878986+0000","last_active":"2024-04-22T01:17:42.849840+0000","last_peered":"2024-04-22T01:17:42.849840+0000","last_clean":"2024-04-22T01:17:42.849840+0000","last_became_active":"2024-04-22T01:17:41.877965+0000","last_became_peered":"2024-04-22T01:17:41.877965+0000","last_unstale":"2024-04-22T01:17:42.849840+0000","last_undegraded":"2024-04-22T01:17:42.849840+0000","last_fullsized":"2024-04-22T01:17:42.849840+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:27.182121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848195+0000","last_change":"2024-04-22T01:17:41.875657+0000","last_active":"2024-04-22T01:17:42.848195+0000","last_peered":"2024-04-22T01:17:42.848195+0000","last_clean":"2024-04-22T01:17:42.848195+0000","last_became_active":"2024-04-22T01:17:41.874951+0000","last_became_peered":"2024-04-22T01:17:41.874951+0000","last_unstale":"2024-04-22T01:17:42.848195+0000","last_undegraded":"2024-04-22T01:17:42.848195+0000","last_fullsized":"2024-04-22T01:17:42.848195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:15:50.183294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.887798+0000","last_change":"2024-04-22T01:17:41.883265+0000","last_active":"2024-04-22T01:17:41.887798+0000","last_peered":"2024-04-22T01:17:41.887798+0000","last_clean":"2024-04-22T01:17:41.887798+0000","last_became_active":"2024-04-22T01:17:41.882708+0000","last_became_peered":"2024-04-22T01:17:41.882708+0000","last_unstale":"2024-04-22T01:17:41.887798+0000","last_undegraded":"2024-04-22T01:17:41.887798+0000","last_fullsized":"2024-04-22T01:17:41.887798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:27.182121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886248+0000","last_change":"2024-04-22T01:17:41.875070+0000","last_active":"2024-04-22T01:17:42.886248+0000","last_peered":"2024-04-22T01:17:42.886248+0000","last_clean":"2024-04-22T01:17:42.886248+0000","last_became_active":"2024-04-22T01:17:41.873429+0000","last_became_peered":"2024-04-22T01:17:41.873429+0000","last_unstale":"2024-04-22T01:17:42.886248+0000","last_undegraded":"2024-04-22T01:17:42.886248+0000","last_fullsized":"2024-04-22T01:17:42.886248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:38.710984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.875774+0000","last_change":"2024-04-22T01:17:41.868397+0000","last_active":"2024-04-22T01:17:41.875774+0000","last_peered":"2024-04-22T01:17:41.875774+0000","last_clean":"2024-04-22T01:17:41.875774+0000","last_became_active":"2024-04-22T01:17:41.867596+0000","last_became_peered":"2024-04-22T01:17:41.867596+0000","last_unstale":"2024-04-22T01:17:41.875774+0000","last_undegraded":"2024-04-22T01:17:41.875774+0000","last_fullsized":"2024-04-22T01:17:41.875774+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:05.400978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848489+0000","last_change":"2024-04-22T01:17:41.889444+0000","last_active":"2024-04-22T01:17:42.848489+0000","last_peered":"2024-04-22T01:17:42.848489+0000","last_clean":"2024-04-22T01:17:42.848489+0000","last_became_active":"2024-04-22T01:17:41.889000+0000","last_became_peered":"2024-04-22T01:17:41.889000+0000","last_unstale":"2024-04-22T01:17:42.848489+0000","last_undegraded":"2024-04-22T01:17:42.848489+0000","last_fullsized":"2024-04-22T01:17:42.848489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:41.038244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.868983+0000","last_change":"2024-04-22T01:17:41.861362+0000","last_active":"2024-04-22T01:17:41.868983+0000","last_peered":"2024-04-22T01:17:41.868983+0000","last_clean":"2024-04-22T01:17:41.868983+0000","last_became_active":"2024-04-22T01:17:41.860570+0000","last_became_peered":"2024-04-22T01:17:41.860570+0000","last_unstale":"2024-04-22T01:17:41.868983+0000","last_undegraded":"2024-04-22T01:17:41.868983+0000","last_fullsized":"2024-04-22T01:17:41.868983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:27:09.461717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.871761+0000","last_change":"2024-04-22T01:17:41.865860+0000","last_active":"2024-04-22T01:17:41.871761+0000","last_peered":"2024-04-22T01:17:41.871761+0000","last_clean":"2024-04-22T01:17:41.871761+0000","last_became_active":"2024-04-22T01:17:41.863593+0000","last_became_peered":"2024-04-22T01:17:41.863593+0000","last_unstale":"2024-04-22T01:17:41.871761+0000","last_undegraded":"2024-04-22T01:17:41.871761+0000","last_fullsized":"2024-04-22T01:17:41.871761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:05.493690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847903+0000","last_change":"2024-04-22T01:17:41.868773+0000","last_active":"2024-04-22T01:17:42.847903+0000","last_peered":"2024-04-22T01:17:42.847903+0000","last_clean":"2024-04-22T01:17:42.847903+0000","last_became_active":"2024-04-22T01:17:41.867463+0000","last_became_peered":"2024-04-22T01:17:41.867463+0000","last_unstale":"2024-04-22T01:17:42.847903+0000","last_undegraded":"2024-04-22T01:17:42.847903+0000","last_fullsized":"2024-04-22T01:17:42.847903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:06.830084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889123+0000","last_change":"2024-04-22T01:17:41.872432+0000","last_active":"2024-04-22T01:17:41.889123+0000","last_peered":"2024-04-22T01:17:41.889123+0000","last_clean":"2024-04-22T01:17:41.889123+0000","last_became_active":"2024-04-22T01:17:41.870157+0000","last_became_peered":"2024-04-22T01:17:41.870157+0000","last_unstale":"2024-04-22T01:17:41.889123+0000","last_undegraded":"2024-04-22T01:17:41.889123+0000","last_fullsized":"2024-04-22T01:17:41.889123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:19:05.042181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847972+0000","last_change":"2024-04-22T01:17:41.874790+0000","last_active":"2024-04-22T01:17:42.847972+0000","last_peered":"2024-04-22T01:17:42.847972+0000","last_clean":"2024-04-22T01:17:42.847972+0000","last_became_active":"2024-04-22T01:17:41.874200+0000","last_became_peered":"2024-04-22T01:17:41.874200+0000","last_unstale":"2024-04-22T01:17:42.847972+0000","last_undegraded":"2024-04-22T01:17:42.847972+0000","last_fullsized":"2024-04-22T01:17:42.847972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:48.541919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.888772+0000","last_change":"2024-04-22T01:17:41.858324+0000","last_active":"2024-04-22T01:17:41.888772+0000","last_peered":"2024-04-22T01:17:41.888772+0000","last_clean":"2024-04-22T01:17:41.888772+0000","last_became_active":"2024-04-22T01:17:41.857554+0000","last_became_peered":"2024-04-22T01:17:41.857554+0000","last_unstale":"2024-04-22T01:17:41.888772+0000","last_undegraded":"2024-04-22T01:17:41.888772+0000","last_fullsized":"2024-04-22T01:17:41.888772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:53:37.885728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886115+0000","last_change":"2024-04-22T01:17:41.883449+0000","last_active":"2024-04-22T01:17:42.886115+0000","last_peered":"2024-04-22T01:17:42.886115+0000","last_clean":"2024-04-22T01:17:42.886115+0000","last_became_active":"2024-04-22T01:17:41.883143+0000","last_became_peered":"2024-04-22T01:17:41.883143+0000","last_unstale":"2024-04-22T01:17:42.886115+0000","last_undegraded":"2024-04-22T01:17:42.886115+0000","last_fullsized":"2024-04-22T01:17:42.886115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:52.040748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886947+0000","last_change":"2024-04-22T01:17:41.867495+0000","last_active":"2024-04-22T01:17:42.886947+0000","last_peered":"2024-04-22T01:17:42.886947+0000","last_clean":"2024-04-22T01:17:42.886947+0000","last_became_active":"2024-04-22T01:17:41.866764+0000","last_became_peered":"2024-04-22T01:17:41.866764+0000","last_unstale":"2024-04-22T01:17:42.886947+0000","last_undegraded":"2024-04-22T01:17:42.886947+0000","last_fullsized":"2024-04-22T01:17:42.886947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:27:25.202727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849815+0000","last_change":"2024-04-22T01:17:41.881695+0000","last_active":"2024-04-22T01:17:42.849815+0000","last_peered":"2024-04-22T01:17:42.849815+0000","last_clean":"2024-04-22T01:17:42.849815+0000","last_became_active":"2024-04-22T01:17:41.880183+0000","last_became_peered":"2024-04-22T01:17:41.880183+0000","last_unstale":"2024-04-22T01:17:42.849815+0000","last_undegraded":"2024-04-22T01:17:42.849815+0000","last_fullsized":"2024-04-22T01:17:42.849815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:15.283639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.882617+0000","last_change":"2024-04-22T01:17:41.876203+0000","last_active":"2024-04-22T01:17:41.882617+0000","last_peered":"2024-04-22T01:17:41.882617+0000","last_clean":"2024-04-22T01:17:41.882617+0000","last_became_active":"2024-04-22T01:17:41.875465+0000","last_became_peered":"2024-04-22T01:17:41.875465+0000","last_unstale":"2024-04-22T01:17:41.882617+0000","last_undegraded":"2024-04-22T01:17:41.882617+0000","last_fullsized":"2024-04-22T01:17:41.882617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:22:27.182121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847917+0000","last_change":"2024-04-22T01:17:41.874913+0000","last_active":"2024-04-22T01:17:42.847917+0000","last_peered":"2024-04-22T01:17:42.847917+0000","last_clean":"2024-04-22T01:17:42.847917+0000","last_became_active":"2024-04-22T01:17:41.874304+0000","last_became_peered":"2024-04-22T01:17:41.874304+0000","last_unstale":"2024-04-22T01:17:42.847917+0000","last_undegraded":"2024-04-22T01:17:42.847917+0000","last_fullsized":"2024-04-22T01:17:42.847917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:51.174276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886474+0000","last_change":"2024-04-22T01:17:41.884069+0000","last_active":"2024-04-22T01:17:42.886474+0000","last_peered":"2024-04-22T01:17:42.886474+0000","last_clean":"2024-04-22T01:17:42.886474+0000","last_became_active":"2024-04-22T01:17:41.883403+0000","last_became_peered":"2024-04-22T01:17:41.883403+0000","last_unstale":"2024-04-22T01:17:42.886474+0000","last_undegraded":"2024-04-22T01:17:42.886474+0000","last_fullsized":"2024-04-22T01:17:42.886474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:09.479666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.898355+0000","last_change":"2024-04-22T01:17:41.894670+0000","last_active":"2024-04-22T01:17:41.898355+0000","last_peered":"2024-04-22T01:17:41.898355+0000","last_clean":"2024-04-22T01:17:41.898355+0000","last_became_active":"2024-04-22T01:17:41.894437+0000","last_became_peered":"2024-04-22T01:17:41.894437+0000","last_unstale":"2024-04-22T01:17:41.898355+0000","last_undegraded":"2024-04-22T01:17:41.898355+0000","last_fullsized":"2024-04-22T01:17:41.898355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:00.723573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.890175+0000","last_change":"2024-04-22T01:17:41.887569+0000","last_active":"2024-04-22T01:17:41.890175+0000","last_peered":"2024-04-22T01:17:41.890175+0000","last_clean":"2024-04-22T01:17:41.890175+0000","last_became_active":"2024-04-22T01:17:41.887318+0000","last_became_peered":"2024-04-22T01:17:41.887318+0000","last_unstale":"2024-04-22T01:17:41.890175+0000","last_undegraded":"2024-04-22T01:17:41.890175+0000","last_fullsized":"2024-04-22T01:17:41.890175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:37.106102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892998+0000","last_change":"2024-04-22T01:17:41.889792+0000","last_active":"2024-04-22T01:17:41.892998+0000","last_peered":"2024-04-22T01:17:41.892998+0000","last_clean":"2024-04-22T01:17:41.892998+0000","last_became_active":"2024-04-22T01:17:41.889422+0000","last_became_peered":"2024-04-22T01:17:41.889422+0000","last_unstale":"2024-04-22T01:17:41.892998+0000","last_undegraded":"2024-04-22T01:17:41.892998+0000","last_fullsized":"2024-04-22T01:17:41.892998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:01.996527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.870078+0000","last_change":"2024-04-22T01:17:41.861470+0000","last_active":"2024-04-22T01:17:41.870078+0000","last_peered":"2024-04-22T01:17:41.870078+0000","last_clean":"2024-04-22T01:17:41.870078+0000","last_became_active":"2024-04-22T01:17:41.860680+0000","last_became_peered":"2024-04-22T01:17:41.860680+0000","last_unstale":"2024-04-22T01:17:41.870078+0000","last_undegraded":"2024-04-22T01:17:41.870078+0000","last_fullsized":"2024-04-22T01:17:41.870078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:59:32.206639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"77'4","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848641+0000","last_change":"2024-04-22T01:17:40.828053+0000","last_active":"2024-04-22T01:17:42.848641+0000","last_peered":"2024-04-22T01:17:42.848641+0000","last_clean":"2024-04-22T01:17:42.848641+0000","last_became_active":"2024-04-22T01:17:40.827707+0000","last_became_peered":"2024-04-22T01:17:40.827707+0000","last_unstale":"2024-04-22T01:17:42.848641+0000","last_undegraded":"2024-04-22T01:17:42.848641+0000","last_fullsized":"2024-04-22T01:17:42.848641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:53.437069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886420+0000","last_change":"2024-04-22T01:17:41.875686+0000","last_active":"2024-04-22T01:17:42.886420+0000","last_peered":"2024-04-22T01:17:42.886420+0000","last_clean":"2024-04-22T01:17:42.886420+0000","last_became_active":"2024-04-22T01:17:41.874372+0000","last_became_peered":"2024-04-22T01:17:41.874372+0000","last_unstale":"2024-04-22T01:17:42.886420+0000","last_undegraded":"2024-04-22T01:17:42.886420+0000","last_fullsized":"2024-04-22T01:17:42.886420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:20.111786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.888795+0000","last_change":"2024-04-22T01:17:40.848569+0000","last_active":"2024-04-22T01:17:41.888795+0000","last_peered":"2024-04-22T01:17:41.888795+0000","last_clean":"2024-04-22T01:17:41.888795+0000","last_became_active":"2024-04-22T01:17:40.846769+0000","last_became_peered":"2024-04-22T01:17:40.846769+0000","last_unstale":"2024-04-22T01:17:41.888795+0000","last_undegraded":"2024-04-22T01:17:41.888795+0000","last_fullsized":"2024-04-22T01:17:41.888795+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:40.246816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892205+0000","last_change":"2024-04-22T01:17:41.880848+0000","last_active":"2024-04-22T01:17:41.892205+0000","last_peered":"2024-04-22T01:17:41.892205+0000","last_clean":"2024-04-22T01:17:41.892205+0000","last_became_active":"2024-04-22T01:17:41.872876+0000","last_became_peered":"2024-04-22T01:17:41.872876+0000","last_unstale":"2024-04-22T01:17:41.892205+0000","last_undegraded":"2024-04-22T01:17:41.892205+0000","last_fullsized":"2024-04-22T01:17:41.892205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:22:14.961296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.899641+0000","last_change":"2024-04-22T01:17:40.840101+0000","last_active":"2024-04-22T01:17:41.899641+0000","last_peered":"2024-04-22T01:17:41.899641+0000","last_clean":"2024-04-22T01:17:41.899641+0000","last_became_active":"2024-04-22T01:17:40.839293+0000","last_became_peered":"2024-04-22T01:17:40.839293+0000","last_unstale":"2024-04-22T01:17:41.899641+0000","last_undegraded":"2024-04-22T01:17:41.899641+0000","last_fullsized":"2024-04-22T01:17:41.899641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:46:37.415042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889150+0000","last_change":"2024-04-22T01:17:41.872870+0000","last_active":"2024-04-22T01:17:41.889150+0000","last_peered":"2024-04-22T01:17:41.889150+0000","last_clean":"2024-04-22T01:17:41.889150+0000","last_became_active":"2024-04-22T01:17:41.872114+0000","last_became_peered":"2024-04-22T01:17:41.872114+0000","last_unstale":"2024-04-22T01:17:41.889150+0000","last_undegraded":"2024-04-22T01:17:41.889150+0000","last_fullsized":"2024-04-22T01:17:41.889150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:33.652052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849112+0000","last_change":"2024-04-22T01:17:40.845275+0000","last_active":"2024-04-22T01:17:42.849112+0000","last_peered":"2024-04-22T01:17:42.849112+0000","last_clean":"2024-04-22T01:17:42.849112+0000","last_became_active":"2024-04-22T01:17:40.844181+0000","last_became_peered":"2024-04-22T01:17:40.844181+0000","last_unstale":"2024-04-22T01:17:42.849112+0000","last_undegraded":"2024-04-22T01:17:42.849112+0000","last_fullsized":"2024-04-22T01:17:42.849112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:34:50.449570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886760+0000","last_change":"2024-04-22T01:17:41.884627+0000","last_active":"2024-04-22T01:17:42.886760+0000","last_peered":"2024-04-22T01:17:42.886760+0000","last_clean":"2024-04-22T01:17:42.886760+0000","last_became_active":"2024-04-22T01:17:41.884021+0000","last_became_peered":"2024-04-22T01:17:41.884021+0000","last_unstale":"2024-04-22T01:17:42.886760+0000","last_undegraded":"2024-04-22T01:17:42.886760+0000","last_fullsized":"2024-04-22T01:17:42.886760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:14.492576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892415+0000","last_change":"2024-04-22T01:17:40.831323+0000","last_active":"2024-04-22T01:17:41.892415+0000","last_peered":"2024-04-22T01:17:41.892415+0000","last_clean":"2024-04-22T01:17:41.892415+0000","last_became_active":"2024-04-22T01:17:40.830501+0000","last_became_peered":"2024-04-22T01:17:40.830501+0000","last_unstale":"2024-04-22T01:17:41.892415+0000","last_undegraded":"2024-04-22T01:17:41.892415+0000","last_fullsized":"2024-04-22T01:17:41.892415+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:51.683494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886696+0000","last_change":"2024-04-22T01:17:41.885506+0000","last_active":"2024-04-22T01:17:42.886696+0000","last_peered":"2024-04-22T01:17:42.886696+0000","last_clean":"2024-04-22T01:17:42.886696+0000","last_became_active":"2024-04-22T01:17:41.884009+0000","last_became_peered":"2024-04-22T01:17:41.884009+0000","last_unstale":"2024-04-22T01:17:42.886696+0000","last_undegraded":"2024-04-22T01:17:42.886696+0000","last_fullsized":"2024-04-22T01:17:42.886696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:16.418965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848531+0000","last_change":"2024-04-22T01:17:40.827513+0000","last_active":"2024-04-22T01:17:42.848531+0000","last_peered":"2024-04-22T01:17:42.848531+0000","last_clean":"2024-04-22T01:17:42.848531+0000","last_became_active":"2024-04-22T01:17:40.826741+0000","last_became_peered":"2024-04-22T01:17:40.826741+0000","last_unstale":"2024-04-22T01:17:42.848531+0000","last_undegraded":"2024-04-22T01:17:42.848531+0000","last_fullsized":"2024-04-22T01:17:42.848531+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:47:48.218080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892203+0000","last_change":"2024-04-22T01:17:41.878923+0000","last_active":"2024-04-22T01:17:41.892203+0000","last_peered":"2024-04-22T01:17:41.892203+0000","last_clean":"2024-04-22T01:17:41.892203+0000","last_became_active":"2024-04-22T01:17:41.878045+0000","last_became_peered":"2024-04-22T01:17:41.878045+0000","last_unstale":"2024-04-22T01:17:41.892203+0000","last_undegraded":"2024-04-22T01:17:41.892203+0000","last_fullsized":"2024-04-22T01:17:41.892203+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:24.410186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886728+0000","last_change":"2024-04-22T01:17:40.834316+0000","last_active":"2024-04-22T01:17:42.886728+0000","last_peered":"2024-04-22T01:17:42.886728+0000","last_clean":"2024-04-22T01:17:42.886728+0000","last_became_active":"2024-04-22T01:17:40.834033+0000","last_became_peered":"2024-04-22T01:17:40.834033+0000","last_unstale":"2024-04-22T01:17:42.886728+0000","last_undegraded":"2024-04-22T01:17:42.886728+0000","last_fullsized":"2024-04-22T01:17:42.886728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:14.457408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850361+0000","last_change":"2024-04-22T01:17:41.887477+0000","last_active":"2024-04-22T01:17:42.850361+0000","last_peered":"2024-04-22T01:17:42.850361+0000","last_clean":"2024-04-22T01:17:42.850361+0000","last_became_active":"2024-04-22T01:17:41.887147+0000","last_became_peered":"2024-04-22T01:17:41.887147+0000","last_unstale":"2024-04-22T01:17:42.850361+0000","last_undegraded":"2024-04-22T01:17:42.850361+0000","last_fullsized":"2024-04-22T01:17:42.850361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:45:41.159892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847485+0000","last_change":"2024-04-22T01:17:40.833502+0000","last_active":"2024-04-22T01:17:42.847485+0000","last_peered":"2024-04-22T01:17:42.847485+0000","last_clean":"2024-04-22T01:17:42.847485+0000","last_became_active":"2024-04-22T01:17:40.833162+0000","last_became_peered":"2024-04-22T01:17:40.833162+0000","last_unstale":"2024-04-22T01:17:42.847485+0000","last_undegraded":"2024-04-22T01:17:42.847485+0000","last_fullsized":"2024-04-22T01:17:42.847485+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:21.104552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886117+0000","last_change":"2024-04-22T01:17:41.874080+0000","last_active":"2024-04-22T01:17:42.886117+0000","last_peered":"2024-04-22T01:17:42.886117+0000","last_clean":"2024-04-22T01:17:42.886117+0000","last_became_active":"2024-04-22T01:17:41.873605+0000","last_became_peered":"2024-04-22T01:17:41.873605+0000","last_unstale":"2024-04-22T01:17:42.886117+0000","last_undegraded":"2024-04-22T01:17:42.886117+0000","last_fullsized":"2024-04-22T01:17:42.886117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:21:07.002244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.860297+0000","last_change":"2024-04-22T01:17:40.847444+0000","last_active":"2024-04-22T01:17:41.860297+0000","last_peered":"2024-04-22T01:17:41.860297+0000","last_clean":"2024-04-22T01:17:41.860297+0000","last_became_active":"2024-04-22T01:17:40.846762+0000","last_became_peered":"2024-04-22T01:17:40.846762+0000","last_unstale":"2024-04-22T01:17:41.860297+0000","last_undegraded":"2024-04-22T01:17:41.860297+0000","last_fullsized":"2024-04-22T01:17:41.860297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:11.544234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848947+0000","last_change":"2024-04-22T01:17:41.866658+0000","last_active":"2024-04-22T01:17:42.848947+0000","last_peered":"2024-04-22T01:17:42.848947+0000","last_clean":"2024-04-22T01:17:42.848947+0000","last_became_active":"2024-04-22T01:17:41.865952+0000","last_became_peered":"2024-04-22T01:17:41.865952+0000","last_unstale":"2024-04-22T01:17:42.848947+0000","last_undegraded":"2024-04-22T01:17:42.848947+0000","last_fullsized":"2024-04-22T01:17:42.848947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:05.493690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850066+0000","last_change":"2024-04-22T01:17:40.841354+0000","last_active":"2024-04-22T01:17:42.850066+0000","last_peered":"2024-04-22T01:17:42.850066+0000","last_clean":"2024-04-22T01:17:42.850066+0000","last_became_active":"2024-04-22T01:17:40.840513+0000","last_became_peered":"2024-04-22T01:17:40.840513+0000","last_unstale":"2024-04-22T01:17:42.850066+0000","last_undegraded":"2024-04-22T01:17:42.850066+0000","last_fullsized":"2024-04-22T01:17:42.850066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:45:40.156277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.872515+0000","last_change":"2024-04-22T01:17:41.869513+0000","last_active":"2024-04-22T01:17:41.872515+0000","last_peered":"2024-04-22T01:17:41.872515+0000","last_clean":"2024-04-22T01:17:41.872515+0000","last_became_active":"2024-04-22T01:17:41.868739+0000","last_became_peered":"2024-04-22T01:17:41.868739+0000","last_unstale":"2024-04-22T01:17:41.872515+0000","last_undegraded":"2024-04-22T01:17:41.872515+0000","last_fullsized":"2024-04-22T01:17:41.872515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:05.400978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848430+0000","last_change":"2024-04-22T01:17:40.833422+0000","last_active":"2024-04-22T01:17:42.848430+0000","last_peered":"2024-04-22T01:17:42.848430+0000","last_clean":"2024-04-22T01:17:42.848430+0000","last_became_active":"2024-04-22T01:17:40.833036+0000","last_became_peered":"2024-04-22T01:17:40.833036+0000","last_unstale":"2024-04-22T01:17:42.848430+0000","last_undegraded":"2024-04-22T01:17:42.848430+0000","last_fullsized":"2024-04-22T01:17:42.848430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:33.059281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.885727+0000","last_change":"2024-04-22T01:17:41.877650+0000","last_active":"2024-04-22T01:17:41.885727+0000","last_peered":"2024-04-22T01:17:41.885727+0000","last_clean":"2024-04-22T01:17:41.885727+0000","last_became_active":"2024-04-22T01:17:41.877369+0000","last_became_peered":"2024-04-22T01:17:41.877369+0000","last_unstale":"2024-04-22T01:17:41.885727+0000","last_undegraded":"2024-04-22T01:17:41.885727+0000","last_fullsized":"2024-04-22T01:17:41.885727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:53.908425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"77'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.849380+0000","last_change":"2024-04-22T01:17:40.836475+0000","last_active":"2024-04-22T01:17:42.849380+0000","last_peered":"2024-04-22T01:17:42.849380+0000","last_clean":"2024-04-22T01:17:42.849380+0000","last_became_active":"2024-04-22T01:17:40.835921+0000","last_became_peered":"2024-04-22T01:17:40.835921+0000","last_unstale":"2024-04-22T01:17:42.849380+0000","last_undegraded":"2024-04-22T01:17:42.849380+0000","last_fullsized":"2024-04-22T01:17:42.849380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:21:16.138190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850216+0000","last_change":"2024-04-22T01:17:41.868936+0000","last_active":"2024-04-22T01:17:42.850216+0000","last_peered":"2024-04-22T01:17:42.850216+0000","last_clean":"2024-04-22T01:17:42.850216+0000","last_became_active":"2024-04-22T01:17:41.868198+0000","last_became_peered":"2024-04-22T01:17:41.868198+0000","last_unstale":"2024-04-22T01:17:42.850216+0000","last_undegraded":"2024-04-22T01:17:42.850216+0000","last_fullsized":"2024-04-22T01:17:42.850216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:34:39.563352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848326+0000","last_change":"2024-04-22T01:17:40.842649+0000","last_active":"2024-04-22T01:17:42.848326+0000","last_peered":"2024-04-22T01:17:42.848326+0000","last_clean":"2024-04-22T01:17:42.848326+0000","last_became_active":"2024-04-22T01:17:40.842050+0000","last_became_peered":"2024-04-22T01:17:40.842050+0000","last_unstale":"2024-04-22T01:17:42.848326+0000","last_undegraded":"2024-04-22T01:17:42.848326+0000","last_fullsized":"2024-04-22T01:17:42.848326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:16.993968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850440+0000","last_change":"2024-04-22T01:17:41.881798+0000","last_active":"2024-04-22T01:17:42.850440+0000","last_peered":"2024-04-22T01:17:42.850440+0000","last_clean":"2024-04-22T01:17:42.850440+0000","last_became_active":"2024-04-22T01:17:41.881417+0000","last_became_peered":"2024-04-22T01:17:41.881417+0000","last_unstale":"2024-04-22T01:17:42.850440+0000","last_undegraded":"2024-04-22T01:17:42.850440+0000","last_fullsized":"2024-04-22T01:17:42.850440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:30.648935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.859254+0000","last_change":"2024-04-22T01:17:40.843316+0000","last_active":"2024-04-22T01:17:41.859254+0000","last_peered":"2024-04-22T01:17:41.859254+0000","last_clean":"2024-04-22T01:17:41.859254+0000","last_became_active":"2024-04-22T01:17:40.842543+0000","last_became_peered":"2024-04-22T01:17:40.842543+0000","last_unstale":"2024-04-22T01:17:41.859254+0000","last_undegraded":"2024-04-22T01:17:41.859254+0000","last_fullsized":"2024-04-22T01:17:41.859254+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:14.457408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850061+0000","last_change":"2024-04-22T01:17:41.894050+0000","last_active":"2024-04-22T01:17:42.850061+0000","last_peered":"2024-04-22T01:17:42.850061+0000","last_clean":"2024-04-22T01:17:42.850061+0000","last_became_active":"2024-04-22T01:17:41.893759+0000","last_became_peered":"2024-04-22T01:17:41.893759+0000","last_unstale":"2024-04-22T01:17:42.850061+0000","last_undegraded":"2024-04-22T01:17:42.850061+0000","last_fullsized":"2024-04-22T01:17:42.850061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:08.613173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847743+0000","last_change":"2024-04-22T01:17:40.828400+0000","last_active":"2024-04-22T01:17:42.847743+0000","last_peered":"2024-04-22T01:17:42.847743+0000","last_clean":"2024-04-22T01:17:42.847743+0000","last_became_active":"2024-04-22T01:17:40.827685+0000","last_became_peered":"2024-04-22T01:17:40.827685+0000","last_unstale":"2024-04-22T01:17:42.847743+0000","last_undegraded":"2024-04-22T01:17:42.847743+0000","last_fullsized":"2024-04-22T01:17:42.847743+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:33:18.254370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850475+0000","last_change":"2024-04-22T01:17:41.885754+0000","last_active":"2024-04-22T01:17:42.850475+0000","last_peered":"2024-04-22T01:17:42.850475+0000","last_clean":"2024-04-22T01:17:42.850475+0000","last_became_active":"2024-04-22T01:17:41.885378+0000","last_became_peered":"2024-04-22T01:17:41.885378+0000","last_unstale":"2024-04-22T01:17:42.850475+0000","last_undegraded":"2024-04-22T01:17:42.850475+0000","last_fullsized":"2024-04-22T01:17:42.850475+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:26.530451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886239+0000","last_change":"2024-04-22T01:17:40.847699+0000","last_active":"2024-04-22T01:17:42.886239+0000","last_peered":"2024-04-22T01:17:42.886239+0000","last_clean":"2024-04-22T01:17:42.886239+0000","last_became_active":"2024-04-22T01:17:40.847393+0000","last_became_peered":"2024-04-22T01:17:40.847393+0000","last_unstale":"2024-04-22T01:17:42.886239+0000","last_undegraded":"2024-04-22T01:17:42.886239+0000","last_fullsized":"2024-04-22T01:17:42.886239+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:25.526836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.882630+0000","last_change":"2024-04-22T01:17:41.880855+0000","last_active":"2024-04-22T01:17:41.882630+0000","last_peered":"2024-04-22T01:17:41.882630+0000","last_clean":"2024-04-22T01:17:41.882630+0000","last_became_active":"2024-04-22T01:17:41.880209+0000","last_became_peered":"2024-04-22T01:17:41.880209+0000","last_unstale":"2024-04-22T01:17:41.882630+0000","last_undegraded":"2024-04-22T01:17:41.882630+0000","last_fullsized":"2024-04-22T01:17:41.882630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:15.283639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.889866+0000","last_change":"2024-04-22T01:17:40.839274+0000","last_active":"2024-04-22T01:17:41.889866+0000","last_peered":"2024-04-22T01:17:41.889866+0000","last_clean":"2024-04-22T01:17:41.889866+0000","last_became_active":"2024-04-22T01:17:40.837914+0000","last_became_peered":"2024-04-22T01:17:40.837914+0000","last_unstale":"2024-04-22T01:17:41.889866+0000","last_undegraded":"2024-04-22T01:17:41.889866+0000","last_fullsized":"2024-04-22T01:17:41.889866+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:42.567223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847498+0000","last_change":"2024-04-22T01:17:41.888298+0000","last_active":"2024-04-22T01:17:42.847498+0000","last_peered":"2024-04-22T01:17:42.847498+0000","last_clean":"2024-04-22T01:17:42.847498+0000","last_became_active":"2024-04-22T01:17:41.887999+0000","last_became_peered":"2024-04-22T01:17:41.887999+0000","last_unstale":"2024-04-22T01:17:42.847498+0000","last_undegraded":"2024-04-22T01:17:42.847498+0000","last_fullsized":"2024-04-22T01:17:42.847498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:25.245611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.893947+0000","last_change":"2024-04-22T01:17:40.833472+0000","last_active":"2024-04-22T01:17:41.893947+0000","last_peered":"2024-04-22T01:17:41.893947+0000","last_clean":"2024-04-22T01:17:41.893947+0000","last_became_active":"2024-04-22T01:17:40.831399+0000","last_became_peered":"2024-04-22T01:17:40.831399+0000","last_unstale":"2024-04-22T01:17:41.893947+0000","last_undegraded":"2024-04-22T01:17:41.893947+0000","last_fullsized":"2024-04-22T01:17:41.893947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:50.396784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.847890+0000","last_change":"2024-04-22T01:17:41.877959+0000","last_active":"2024-04-22T01:17:42.847890+0000","last_peered":"2024-04-22T01:17:42.847890+0000","last_clean":"2024-04-22T01:17:42.847890+0000","last_became_active":"2024-04-22T01:17:41.877408+0000","last_became_peered":"2024-04-22T01:17:41.877408+0000","last_unstale":"2024-04-22T01:17:42.847890+0000","last_undegraded":"2024-04-22T01:17:42.847890+0000","last_fullsized":"2024-04-22T01:17:42.847890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:39:23.996559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848263+0000","last_change":"2024-04-22T01:17:40.836219+0000","last_active":"2024-04-22T01:17:42.848263+0000","last_peered":"2024-04-22T01:17:42.848263+0000","last_clean":"2024-04-22T01:17:42.848263+0000","last_became_active":"2024-04-22T01:17:40.835727+0000","last_became_peered":"2024-04-22T01:17:40.835727+0000","last_unstale":"2024-04-22T01:17:42.848263+0000","last_undegraded":"2024-04-22T01:17:42.848263+0000","last_fullsized":"2024-04-22T01:17:42.848263+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:35.561611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.872127+0000","last_change":"2024-04-22T01:17:41.867855+0000","last_active":"2024-04-22T01:17:41.872127+0000","last_peered":"2024-04-22T01:17:41.872127+0000","last_clean":"2024-04-22T01:17:41.872127+0000","last_became_active":"2024-04-22T01:17:41.867055+0000","last_became_peered":"2024-04-22T01:17:41.867055+0000","last_unstale":"2024-04-22T01:17:41.872127+0000","last_undegraded":"2024-04-22T01:17:41.872127+0000","last_fullsized":"2024-04-22T01:17:41.872127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:26:43.570838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.839959+0000","last_change":"2024-04-22T01:17:40.839500+0000","last_active":"2024-04-22T01:17:41.839959+0000","last_peered":"2024-04-22T01:17:41.839959+0000","last_clean":"2024-04-22T01:17:41.839959+0000","last_became_active":"2024-04-22T01:17:40.839108+0000","last_became_peered":"2024-04-22T01:17:40.839108+0000","last_unstale":"2024-04-22T01:17:41.839959+0000","last_undegraded":"2024-04-22T01:17:41.839959+0000","last_fullsized":"2024-04-22T01:17:41.839959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:18.243611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886743+0000","last_change":"2024-04-22T01:17:41.873055+0000","last_active":"2024-04-22T01:17:42.886743+0000","last_peered":"2024-04-22T01:17:42.886743+0000","last_clean":"2024-04-22T01:17:42.886743+0000","last_became_active":"2024-04-22T01:17:41.867148+0000","last_became_peered":"2024-04-22T01:17:41.867148+0000","last_unstale":"2024-04-22T01:17:42.886743+0000","last_undegraded":"2024-04-22T01:17:42.886743+0000","last_fullsized":"2024-04-22T01:17:42.886743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:23:54.290857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.898219+0000","last_change":"2024-04-22T01:17:40.829490+0000","last_active":"2024-04-22T01:17:41.898219+0000","last_peered":"2024-04-22T01:17:41.898219+0000","last_clean":"2024-04-22T01:17:41.898219+0000","last_became_active":"2024-04-22T01:17:40.829131+0000","last_became_peered":"2024-04-22T01:17:40.829131+0000","last_unstale":"2024-04-22T01:17:41.898219+0000","last_undegraded":"2024-04-22T01:17:41.898219+0000","last_fullsized":"2024-04-22T01:17:41.898219+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:14.457408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886828+0000","last_change":"2024-04-22T01:17:41.898727+0000","last_active":"2024-04-22T01:17:42.886828+0000","last_peered":"2024-04-22T01:17:42.886828+0000","last_clean":"2024-04-22T01:17:42.886828+0000","last_became_active":"2024-04-22T01:17:41.894191+0000","last_became_peered":"2024-04-22T01:17:41.894191+0000","last_unstale":"2024-04-22T01:17:42.886828+0000","last_undegraded":"2024-04-22T01:17:42.886828+0000","last_fullsized":"2024-04-22T01:17:42.886828+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:06.191633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.848137+0000","last_change":"2024-04-22T01:17:40.841281+0000","last_active":"2024-04-22T01:17:41.848137+0000","last_peered":"2024-04-22T01:17:41.848137+0000","last_clean":"2024-04-22T01:17:41.848137+0000","last_became_active":"2024-04-22T01:17:40.840844+0000","last_became_peered":"2024-04-22T01:17:40.840844+0000","last_unstale":"2024-04-22T01:17:41.848137+0000","last_undegraded":"2024-04-22T01:17:41.848137+0000","last_fullsized":"2024-04-22T01:17:41.848137+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:46:37.415042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850914+0000","last_change":"2024-04-22T01:17:41.885013+0000","last_active":"2024-04-22T01:17:42.850914+0000","last_peered":"2024-04-22T01:17:42.850914+0000","last_clean":"2024-04-22T01:17:42.850914+0000","last_became_active":"2024-04-22T01:17:41.884629+0000","last_became_peered":"2024-04-22T01:17:41.884629+0000","last_unstale":"2024-04-22T01:17:42.850914+0000","last_undegraded":"2024-04-22T01:17:42.850914+0000","last_fullsized":"2024-04-22T01:17:42.850914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:15.461024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.848066+0000","last_change":"2024-04-22T01:17:40.840312+0000","last_active":"2024-04-22T01:17:41.848066+0000","last_peered":"2024-04-22T01:17:41.848066+0000","last_clean":"2024-04-22T01:17:41.848066+0000","last_became_active":"2024-04-22T01:17:40.840017+0000","last_became_peered":"2024-04-22T01:17:40.840017+0000","last_unstale":"2024-04-22T01:17:41.848066+0000","last_undegraded":"2024-04-22T01:17:41.848066+0000","last_fullsized":"2024-04-22T01:17:41.848066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:40.246816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892570+0000","last_change":"2024-04-22T01:17:41.887240+0000","last_active":"2024-04-22T01:17:41.892570+0000","last_peered":"2024-04-22T01:17:41.892570+0000","last_clean":"2024-04-22T01:17:41.892570+0000","last_became_active":"2024-04-22T01:17:41.886905+0000","last_became_peered":"2024-04-22T01:17:41.886905+0000","last_unstale":"2024-04-22T01:17:41.892570+0000","last_undegraded":"2024-04-22T01:17:41.892570+0000","last_fullsized":"2024-04-22T01:17:41.892570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:12:27.511810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848863+0000","last_change":"2024-04-22T01:17:40.829125+0000","last_active":"2024-04-22T01:17:42.848863+0000","last_peered":"2024-04-22T01:17:42.848863+0000","last_clean":"2024-04-22T01:17:42.848863+0000","last_became_active":"2024-04-22T01:17:40.828255+0000","last_became_peered":"2024-04-22T01:17:40.828255+0000","last_unstale":"2024-04-22T01:17:42.848863+0000","last_undegraded":"2024-04-22T01:17:42.848863+0000","last_fullsized":"2024-04-22T01:17:42.848863+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:58:51.696063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850139+0000","last_change":"2024-04-22T01:17:41.882276+0000","last_active":"2024-04-22T01:17:42.850139+0000","last_peered":"2024-04-22T01:17:42.850139+0000","last_clean":"2024-04-22T01:17:42.850139+0000","last_became_active":"2024-04-22T01:17:41.881733+0000","last_became_peered":"2024-04-22T01:17:41.881733+0000","last_unstale":"2024-04-22T01:17:42.850139+0000","last_undegraded":"2024-04-22T01:17:42.850139+0000","last_fullsized":"2024-04-22T01:17:42.850139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:52.460357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"78'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.888484+0000","last_change":"2024-04-22T01:17:40.828678+0000","last_active":"2024-04-22T01:17:42.888484+0000","last_peered":"2024-04-22T01:17:42.888484+0000","last_clean":"2024-04-22T01:17:42.888484+0000","last_became_active":"2024-04-22T01:17:40.828297+0000","last_became_peered":"2024-04-22T01:17:40.828297+0000","last_unstale":"2024-04-22T01:17:42.888484+0000","last_undegraded":"2024-04-22T01:17:42.888484+0000","last_fullsized":"2024-04-22T01:17:42.888484+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:40.246816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.892154+0000","last_change":"2024-04-22T01:17:41.873726+0000","last_active":"2024-04-22T01:17:41.892154+0000","last_peered":"2024-04-22T01:17:41.892154+0000","last_clean":"2024-04-22T01:17:41.892154+0000","last_became_active":"2024-04-22T01:17:41.871866+0000","last_became_peered":"2024-04-22T01:17:41.871866+0000","last_unstale":"2024-04-22T01:17:41.892154+0000","last_undegraded":"2024-04-22T01:17:41.892154+0000","last_fullsized":"2024-04-22T01:17:41.892154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:50.588832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.860314+0000","last_change":"2024-04-22T01:17:40.848697+0000","last_active":"2024-04-22T01:17:41.860314+0000","last_peered":"2024-04-22T01:17:41.860314+0000","last_clean":"2024-04-22T01:17:41.860314+0000","last_became_active":"2024-04-22T01:17:40.848084+0000","last_became_peered":"2024-04-22T01:17:40.848084+0000","last_unstale":"2024-04-22T01:17:41.860314+0000","last_undegraded":"2024-04-22T01:17:41.860314+0000","last_fullsized":"2024-04-22T01:17:41.860314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:04.490075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.886697+0000","last_change":"2024-04-22T01:17:41.873843+0000","last_active":"2024-04-22T01:17:42.886697+0000","last_peered":"2024-04-22T01:17:42.886697+0000","last_clean":"2024-04-22T01:17:42.886697+0000","last_became_active":"2024-04-22T01:17:41.873524+0000","last_became_peered":"2024-04-22T01:17:41.873524+0000","last_unstale":"2024-04-22T01:17:42.886697+0000","last_undegraded":"2024-04-22T01:17:42.886697+0000","last_fullsized":"2024-04-22T01:17:42.886697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:37.293915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.858819+0000","last_change":"2024-04-22T01:17:40.847822+0000","last_active":"2024-04-22T01:17:41.858819+0000","last_peered":"2024-04-22T01:17:41.858819+0000","last_clean":"2024-04-22T01:17:41.858819+0000","last_became_active":"2024-04-22T01:17:40.847176+0000","last_became_peered":"2024-04-22T01:17:40.847176+0000","last_unstale":"2024-04-22T01:17:41.858819+0000","last_undegraded":"2024-04-22T01:17:41.858819+0000","last_fullsized":"2024-04-22T01:17:41.858819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:25.526836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850237+0000","last_change":"2024-04-22T01:17:41.882199+0000","last_active":"2024-04-22T01:17:42.850237+0000","last_peered":"2024-04-22T01:17:42.850237+0000","last_clean":"2024-04-22T01:17:42.850237+0000","last_became_active":"2024-04-22T01:17:41.881614+0000","last_became_peered":"2024-04-22T01:17:41.881614+0000","last_unstale":"2024-04-22T01:17:42.850237+0000","last_undegraded":"2024-04-22T01:17:42.850237+0000","last_fullsized":"2024-04-22T01:17:42.850237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:17.997584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848731+0000","last_change":"2024-04-22T01:17:40.844114+0000","last_active":"2024-04-22T01:17:42.848731+0000","last_peered":"2024-04-22T01:17:42.848731+0000","last_clean":"2024-04-22T01:17:42.848731+0000","last_became_active":"2024-04-22T01:17:40.839294+0000","last_became_peered":"2024-04-22T01:17:40.839294+0000","last_unstale":"2024-04-22T01:17:42.848731+0000","last_undegraded":"2024-04-22T01:17:42.848731+0000","last_fullsized":"2024-04-22T01:17:42.848731+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:12:26.508195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850139+0000","last_change":"2024-04-22T01:17:41.892576+0000","last_active":"2024-04-22T01:17:42.850139+0000","last_peered":"2024-04-22T01:17:42.850139+0000","last_clean":"2024-04-22T01:17:42.850139+0000","last_became_active":"2024-04-22T01:17:41.892262+0000","last_became_peered":"2024-04-22T01:17:41.892262+0000","last_unstale":"2024-04-22T01:17:42.850139+0000","last_undegraded":"2024-04-22T01:17:42.850139+0000","last_fullsized":"2024-04-22T01:17:42.850139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:47:49.221695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.848083+0000","last_change":"2024-04-22T01:17:40.830460+0000","last_active":"2024-04-22T01:17:42.848083+0000","last_peered":"2024-04-22T01:17:42.848083+0000","last_clean":"2024-04-22T01:17:42.848083+0000","last_became_active":"2024-04-22T01:17:40.829847+0000","last_became_peered":"2024-04-22T01:17:40.829847+0000","last_unstale":"2024-04-22T01:17:42.848083+0000","last_undegraded":"2024-04-22T01:17:42.848083+0000","last_fullsized":"2024-04-22T01:17:42.848083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:25.341958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.885995+0000","last_change":"2024-04-22T01:17:41.879127+0000","last_active":"2024-04-22T01:17:41.885995+0000","last_peered":"2024-04-22T01:17:41.885995+0000","last_clean":"2024-04-22T01:17:41.885995+0000","last_became_active":"2024-04-22T01:17:41.878595+0000","last_became_peered":"2024-04-22T01:17:41.878595+0000","last_unstale":"2024-04-22T01:17:41.885995+0000","last_undegraded":"2024-04-22T01:17:41.885995+0000","last_fullsized":"2024-04-22T01:17:41.885995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:19:44.717342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T01:17:42.850606+0000","last_change":"2024-04-22T01:17:40.838793+0000","last_active":"2024-04-22T01:17:42.850606+0000","last_peered":"2024-04-22T01:17:42.850606+0000","last_clean":"2024-04-22T01:17:42.850606+0000","last_became_active":"2024-04-22T01:17:40.837776+0000","last_became_peered":"2024-04-22T01:17:40.837776+0000","last_unstale":"2024-04-22T01:17:42.850606+0000","last_undegraded":"2024-04-22T01:17:42.850606+0000","last_fullsized":"2024-04-22T01:17:42.850606+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:13.336740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.873352+0000","last_change":"2024-04-22T01:17:41.869252+0000","last_active":"2024-04-22T01:17:41.873352+0000","last_peered":"2024-04-22T01:17:41.873352+0000","last_clean":"2024-04-22T01:17:41.873352+0000","last_became_active":"2024-04-22T01:17:41.868563+0000","last_became_peered":"2024-04-22T01:17:41.868563+0000","last_unstale":"2024-04-22T01:17:41.873352+0000","last_undegraded":"2024-04-22T01:17:41.873352+0000","last_fullsized":"2024-04-22T01:17:41.873352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:50.843950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.839841+0000","last_change":"2024-04-22T01:17:40.832593+0000","last_active":"2024-04-22T01:17:41.839841+0000","last_peered":"2024-04-22T01:17:41.839841+0000","last_clean":"2024-04-22T01:17:41.839841+0000","last_became_active":"2024-04-22T01:17:40.830928+0000","last_became_peered":"2024-04-22T01:17:40.830928+0000","last_unstale":"2024-04-22T01:17:41.839841+0000","last_undegraded":"2024-04-22T01:17:41.839841+0000","last_fullsized":"2024-04-22T01:17:41.839841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:39.704746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.877088+0000","last_change":"2024-04-22T01:17:41.872198+0000","last_active":"2024-04-22T01:17:41.877088+0000","last_peered":"2024-04-22T01:17:41.877088+0000","last_clean":"2024-04-22T01:17:41.877088+0000","last_became_active":"2024-04-22T01:17:41.871402+0000","last_became_peered":"2024-04-22T01:17:41.871402+0000","last_unstale":"2024-04-22T01:17:41.877088+0000","last_undegraded":"2024-04-22T01:17:41.877088+0000","last_fullsized":"2024-04-22T01:17:41.877088+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:40.798903+0000","last_clean_scrub_stamp":"2024-04-22T01:17:40.798903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:15.283639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.848860+0000","last_change":"2024-04-22T01:17:40.840871+0000","last_active":"2024-04-22T01:17:41.848860+0000","last_peered":"2024-04-22T01:17:41.848860+0000","last_clean":"2024-04-22T01:17:41.848860+0000","last_became_active":"2024-04-22T01:17:40.840540+0000","last_became_peered":"2024-04-22T01:17:40.840540+0000","last_unstale":"2024-04-22T01:17:41.848860+0000","last_undegraded":"2024-04-22T01:17:41.848860+0000","last_fullsized":"2024-04-22T01:17:41.848860+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:17:39.795287+0000","last_clean_scrub_stamp":"2024-04-22T01:17:39.795287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:11.544234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":270,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-22T01:17:41.858827+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:17:41.858827+0000","last_peered":"2024-04-22T01:17:41.858827+0000","last_clean":"2024-04-22T01:17:41.858827+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:17:41.858827+0000","last_undegraded":"2024-04-22T01:17:41.858827+0000","last_fullsized":"2024-04-22T01:17:41.858827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1540096,"data_stored":1019554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":437468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":435890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":962560,"data_stored":430880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36268,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335572,"statfs":{"total":96636764160,"available":96599625728,"internally_reserved":0,"allocated":946176,"data_stored":429746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36292,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335548,"statfs":{"total":96636764160,"available":96599601152,"internally_reserved":0,"allocated":966656,"data_stored":436598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335568,"statfs":{"total":96636764160,"available":96599621632,"internally_reserved":0,"allocated":950272,"data_stored":429276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1548288,"data_stored":1026224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:17:45.278 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:17:45.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-22T01:17:45.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 Processor -- start 2024-04-22T01:17:45.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 -- start start 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f25d815a320 0x7f25d806f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f25d80ce420 0x7f25d8072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25d806da70 con 0x7f25d8071b70 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f25d806dd50 con 0x7f25d80ce420 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25e0634640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f25d806e030 con 0x7f25d815a320 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f25d80ce420 0x7f25d8072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f25d80ce420 0x7f25d8072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44180/0 (socket says 172.21.15.70:44180) 2024-04-22T01:17:45.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 -- 172.21.15.70:0/3500425752 learned_addr learned my addr 172.21.15.70:0/3500425752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25ddbcf640 1 --1- 172.21.15.70:0/3500425752 >> v1:172.21.15.70:6789/0 conn(0x7f25d8071b70 0x7f25d80b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:37908/0 (socket says 172.21.15.70:37908) 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25de3d0640 1 --2- 172.21.15.70:0/3500425752 >> v2:172.21.15.98:3300/0 conn(0x7f25d815a320 0x7f25d806f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 -- 172.21.15.70:0/3500425752 >> v2:172.21.15.98:3300/0 conn(0x7f25d815a320 msgr2=0x7f25d806f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 --2- 172.21.15.70:0/3500425752 >> v2:172.21.15.98:3300/0 conn(0x7f25d815a320 0x7f25d806f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 -- 172.21.15.70:0/3500425752 >> v1:172.21.15.70:6789/0 conn(0x7f25d8071b70 legacy=0x7f25d80b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25debd1640 1 -- 172.21.15.70:0/3500425752 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d806e310 con 0x7f25d80ce420 2024-04-22T01:17:45.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.445+0000 7f25dd3ce640 1 -- 172.21.15.70:0/3500425752 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233789605 0 0) 0x7f25d806da70 con 0x7f25d8071b70 2024-04-22T01:17:45.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25debd1640 1 --2- 172.21.15.70:0/3500425752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d80ce420 0x7f25d8072000 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f25d8072d80 tx=0x7f25d001daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbb6fb1165993cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25dd3ce640 1 -- 172.21.15.70:0/3500425752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d000a2d0 con 0x7f25d80ce420 2024-04-22T01:17:45.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25dd3ce640 1 -- 172.21.15.70:0/3500425752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d00a6860 con 0x7f25d80ce420 2024-04-22T01:17:45.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25dd3ce640 1 -- 172.21.15.70:0/3500425752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d00a6c30 con 0x7f25d80ce420 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- 172.21.15.70:0/3500425752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d80ce420 msgr2=0x7f25d8072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 --2- 172.21.15.70:0/3500425752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d80ce420 0x7f25d8072000 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f25d8072d80 tx=0x7f25d001daf0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- 172.21.15.70:0/3500425752 shutdown_connections 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 --2- 172.21.15.70:0/3500425752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d80ce420 0x7f25d8072000 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 --2- 172.21.15.70:0/3500425752 >> v2:172.21.15.98:3300/0 conn(0x7f25d815a320 0x7f25d806f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- 172.21.15.70:0/3500425752 >> 172.21.15.70:0/3500425752 conn(0x7f25d80b1d40 msgr2=0x7f25d80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:45.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- 172.21.15.70:0/3500425752 shutdown_connections 2024-04-22T01:17:45.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- 172.21.15.70:0/3500425752 wait complete. 2024-04-22T01:17:45.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 Processor -- start 2024-04-22T01:17:45.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- start start 2024-04-22T01:17:45.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d8071b70 0x7f25d814e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d80ce420 0x7f25d813f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 0x7f25d813e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25d806f810 con 0x7f25d815a320 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25d806f690 con 0x7f25d8071b70 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.449+0000 7f25e0634640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f25d80b4470 con 0x7f25d80ce420 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 0x7f25d813e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 0x7f25d813e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44194/0 (socket says 172.21.15.70:44194) 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 -- 172.21.15.70:0/246764988 learned_addr learned my addr 172.21.15.70:0/246764988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25de3d0640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d8071b70 0x7f25d814e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ddbcf640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d80ce420 0x7f25d813f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 -- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d80ce420 msgr2=0x7f25d813f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d80ce420 0x7f25d813f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 -- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d8071b70 msgr2=0x7f25d814e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d8071b70 0x7f25d814e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 -- 172.21.15.70:0/246764988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d813fbe0 con 0x7f25d815a320 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ddbcf640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d80ce420 0x7f25d813f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25debd1640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 0x7f25d813e460 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f25d00b0040 tx=0x7f25d001a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0008700 con 0x7f25d815a320 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d00bf070 con 0x7f25d815a320 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d813fe60 con 0x7f25d815a320 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0008b00 con 0x7f25d815a320 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d8153b00 con 0x7f25d815a320 2024-04-22T01:17:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d8068210 con 0x7f25d815a320 2024-04-22T01:17:45.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f25d0008ca0 con 0x7f25d815a320 2024-04-22T01:17:45.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ceffd640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f25bc0678f0 0x7f25bc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f25d0122060 con 0x7f25d815a320 2024-04-22T01:17:45.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25de3d0640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f25bc0678f0 0x7f25bc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.453+0000 7f25de3d0640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f25bc0678f0 0x7f25bc069db0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f25d400b930 tx=0x7f25d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.457+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f25d813fbe0 con 0x7f25d815a320 2024-04-22T01:17:45.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.681+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f25d8155070 con 0x7f25d815a320 2024-04-22T01:17:45.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25ceffd640 1 -- 172.21.15.70:0/246764988 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f25d00176e0 con 0x7f25d815a320 2024-04-22T01:17:45.686 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:17:45.686 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:17:45.686 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-22T01:17:45.686 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-22T01:17:45.686 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-22T01:17:45.687 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-22T01:17:45.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f25bc0678f0 msgr2=0x7f25bc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f25bc0678f0 0x7f25bc069db0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f25d400b930 tx=0x7f25d4008040 comp rx=0 tx=0).stop 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 msgr2=0x7f25d813e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 0x7f25d813e460 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f25d00b0040 tx=0x7f25d001a890 comp rx=0 tx=0).stop 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 shutdown_connections 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f25bc0678f0 0x7f25bc069db0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25d815a320 0x7f25d813e460 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d80ce420 0x7f25d813f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 --2- 172.21.15.70:0/246764988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d8071b70 0x7f25d814e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 >> 172.21.15.70:0/246764988 conn(0x7f25d80b1d40 msgr2=0x7f25d8158a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 shutdown_connections 2024-04-22T01:17:45.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.685+0000 7f25e0634640 1 -- 172.21.15.70:0/246764988 wait complete. 2024-04-22T01:17:45.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:17:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.817+0000 7ffb56b51640 1 Processor -- start 2024-04-22T01:17:45.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.817+0000 7ffb56b51640 1 -- start start 2024-04-22T01:17:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb56b51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb50071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb56b51640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffb50072060 0x7ffb5006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb56b51640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb500cd7b0 con 0x7ffb501455b0 2024-04-22T01:17:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb56b51640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ffb500cc710 con 0x7ffb50072060 2024-04-22T01:17:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb56b51640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffb500cc9f0 con 0x7ffb500cc150 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb50071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb50071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:35926/0 (socket says 172.21.15.70:35926) 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 -- 172.21.15.70:0/1080158134 learned_addr learned my addr 172.21.15.70:0/1080158134 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb550ee640 1 --2- 172.21.15.70:0/1080158134 >> v2:172.21.15.98:3300/0 conn(0x7ffb50072060 0x7ffb5006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb477fe640 1 -- 172.21.15.70:0/1080158134 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130806437 0 0) 0x7ffb500cd7b0 con 0x7ffb501455b0 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb477fe640 1 -- 172.21.15.70:0/1080158134 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb38003530 con 0x7ffb501455b0 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 -- 172.21.15.70:0/1080158134 >> v2:172.21.15.98:3300/0 conn(0x7ffb50072060 msgr2=0x7ffb5006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 --2- 172.21.15.70:0/1080158134 >> v2:172.21.15.98:3300/0 conn(0x7ffb50072060 0x7ffb5006e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 -- 172.21.15.70:0/1080158134 >> v1:172.21.15.98:6789/0 conn(0x7ffb501455b0 legacy=0x7ffb500cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 -- 172.21.15.70:0/1080158134 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb500cccd0 con 0x7ffb500cc150 2024-04-22T01:17:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb550ee640 1 --2- 172.21.15.70:0/1080158134 >> v2:172.21.15.98:3300/0 conn(0x7ffb50072060 0x7ffb5006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:17:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb47fff640 1 --2- 172.21.15.70:0/1080158134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb50071b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009870 tx=0x7ffb4001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f676a4167446c0f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.821+0000 7ffb477fe640 1 -- 172.21.15.70:0/1080158134 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb400b0070 con 0x7ffb500cc150 2024-04-22T01:17:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb477fe640 1 -- 172.21.15.70:0/1080158134 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb400ab600 con 0x7ffb500cc150 2024-04-22T01:17:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb477fe640 1 -- 172.21.15.70:0/1080158134 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb400ab9b0 con 0x7ffb500cc150 2024-04-22T01:17:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- 172.21.15.70:0/1080158134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 msgr2=0x7ffb50071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 --2- 172.21.15.70:0/1080158134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb50071b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009870 tx=0x7ffb4001c440 comp rx=0 tx=0).stop 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- 172.21.15.70:0/1080158134 shutdown_connections 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 --2- 172.21.15.70:0/1080158134 >> v2:172.21.15.98:3300/0 conn(0x7ffb50072060 0x7ffb5006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 --2- 172.21.15.70:0/1080158134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb50071b20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- 172.21.15.70:0/1080158134 >> 172.21.15.70:0/1080158134 conn(0x7ffb5006b1f0 msgr2=0x7ffb5006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- 172.21.15.70:0/1080158134 shutdown_connections 2024-04-22T01:17:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- 172.21.15.70:0/1080158134 wait complete. 2024-04-22T01:17:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 Processor -- start 2024-04-22T01:17:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- start start 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb50072060 0x7ffb50159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb501389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 0x7ffb50165460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5006f2e0 con 0x7ffb501455b0 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5006f160 con 0x7ffb50072060 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb56b51640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffb50072470 con 0x7ffb500cc150 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 0x7ffb50165460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 0x7ffb50165460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44196/0 (socket says 172.21.15.70:44196) 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 -- 172.21.15.70:0/2757758481 learned_addr learned my addr 172.21.15.70:0/2757758481 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb47fff640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb501389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb548ed640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb50072060 0x7ffb50159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 -- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 msgr2=0x7ffb501389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb501389f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 -- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb50072060 msgr2=0x7ffb50159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb50072060 0x7ffb50159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 -- 172.21.15.70:0/2757758481 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb50143ae0 con 0x7ffb501455b0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.825+0000 7ffb550ee640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 0x7ffb50165460 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800efc0 tx=0x7ffb4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb48018030 con 0x7ffb501455b0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4800cdf0 con 0x7ffb501455b0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb50143dc0 con 0x7ffb501455b0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb48003bc0 con 0x7ffb501455b0 2024-04-22T01:17:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb50144660 con 0x7ffb501455b0 2024-04-22T01:17:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb50143f90 con 0x7ffb501455b0 2024-04-22T01:17:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb4801b070 con 0x7ffb501455b0 2024-04-22T01:17:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb457fa640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb3c0678c0 0x7ffb3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7ffb48086810 con 0x7ffb501455b0 2024-04-22T01:17:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb548ed640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb3c0678c0 0x7ffb3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.829+0000 7ffb548ed640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb3c0678c0 0x7ffb3c069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c007fd0 tx=0x7ffb4c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:45.833+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ffb48010040 con 0x7ffb501455b0 2024-04-22T01:17:46.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb50068210 con 0x7ffb501455b0 2024-04-22T01:17:46.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb457fa640 1 -- 172.21.15.70:0/2757758481 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5174 (secure 0 0 0) 0x7ffb48012340 con 0x7ffb501455b0 2024-04-22T01:17:46.018 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:46.018 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:41.174335+0000","modified":"2024-04-22T01:17:43.233600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6759":{"gid":6759,"name":"d","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6833/258015518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":258015518},{"type":"v1","addr":"172.21.15.98:6833","nonce":258015518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6759,"qdb_cluster":[6759]},"id":5}]} 2024-04-22T01:17:46.018 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 37 2024-04-22T01:17:46.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb3c0678c0 msgr2=0x7ffb3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb3c0678c0 0x7ffb3c069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c007fd0 tx=0x7ffb4c009040 comp rx=0 tx=0).stop 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 msgr2=0x7ffb50165460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 0x7ffb50165460 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800efc0 tx=0x7ffb4800c490 comp rx=0 tx=0).stop 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 shutdown_connections 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb3c0678c0 0x7ffb3c069d80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb501455b0 0x7ffb50165460 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb500cc150 0x7ffb501389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 --2- 172.21.15.70:0/2757758481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb50072060 0x7ffb50159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 >> 172.21.15.70:0/2757758481 conn(0x7ffb5006b1f0 msgr2=0x7ffb500cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 shutdown_connections 2024-04-22T01:17:46.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.017+0000 7ffb56b51640 1 -- 172.21.15.70:0/2757758481 wait complete. 2024-04-22T01:17:46.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:17:41.174335+0000', 'modified': '2024-04-22T01:17:43.233600+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6759}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6759': {'gid': 6759, 'name': 'd', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6833/258015518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6832', 'nonce': 258015518}, {'type': 'v1', 'addr': '172.21.15.98:6833', 'nonce': 258015518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6759, 'qdb_cluster': [6759]} 2024-04-22T01:17:46.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:17:46.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-22T01:17:46.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd83d6640 1 Processor -- start 2024-04-22T01:17:46.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd83d6640 1 -- start start 2024-04-22T01:17:46.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd83d6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdd0166ee0 0x7efdd01692d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd83d6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd00d0750 con 0x7efdd006d3f0 2024-04-22T01:17:46.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd83d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd00d0a30 con 0x7efdd0099c50 2024-04-22T01:17:46.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd83d6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efdd01619b0 con 0x7efdd0166ee0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efdd006d3f0 0x7efdd00cfee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38770/0 (socket says 172.21.15.70:38770) 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 -- 172.21.15.70:0/2007239042 learned_addr learned my addr 172.21.15.70:0/2007239042 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd6973640 1 --2- 172.21.15.70:0/2007239042 >> v2:172.21.15.98:3300/0 conn(0x7efdd0166ee0 0x7efdd01692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818834432 0 0) 0x7efdd00d0750 con 0x7efdd006d3f0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdbc003530 con 0x7efdd006d3f0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 252912471 0 0) 0x7efdbc003530 con 0x7efdd006d3f0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 >> v2:172.21.15.98:3300/0 conn(0x7efdd0166ee0 msgr2=0x7efdd01692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 --2- 172.21.15.70:0/2007239042 >> v2:172.21.15.98:3300/0 conn(0x7efdd0166ee0 0x7efdd01692d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 >> v1:172.21.15.98:6789/0 conn(0x7efdd0099c50 legacy=0x7efdd009a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd00d0d10 con 0x7efdd006d3f0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7efdc00016a0 con 0x7efdd006d3f0 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 >> v1:172.21.15.70:6789/0 conn(0x7efdd006d3f0 legacy=0x7efdd00cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdbc003a30 0x7efdbc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdbc004810 0x7efdbc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc0051d0 0x7efdbc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdd01619b0 con 0x7efdbc004810 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc00c0e0 con 0x7efdbc0051d0 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efdbc00c2e0 con 0x7efdbc003a30 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdbc004810 0x7efdbc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd6973640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc0051d0 0x7efdbc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd6172640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdbc003a30 0x7efdbc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 -- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdbc003a30 msgr2=0x7efdbc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdbc003a30 0x7efdbc003e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 -- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc0051d0 msgr2=0x7efdbc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc0051d0 0x7efdbc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd5971640 1 -- 172.21.15.70:0/2007239042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc003530 con 0x7efdbc004810 2024-04-22T01:17:46.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd6973640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc0051d0 0x7efdbc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:46.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.569+0000 7efdd6973640 1 -- 172.21.15.70:0/2007239042 reap_dead start 2024-04-22T01:17:46.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdbc004810 0x7efdbc004c90 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c290 tx=0x7efdc001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce05252e2fc1a84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0007d60 con 0x7efdbc004810 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc00a9070 con 0x7efdbc004810 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5170640 1 -- 172.21.15.70:0/2007239042 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00a5d50 con 0x7efdbc004810 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- 172.21.15.70:0/2007239042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdbc004810 msgr2=0x7efdbc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdbc004810 0x7efdbc004c90 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c290 tx=0x7efdc001ccc0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- 172.21.15.70:0/2007239042 shutdown_connections 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 --2- 172.21.15.70:0/2007239042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdbc004810 0x7efdbc004c90 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- 172.21.15.70:0/2007239042 >> 172.21.15.70:0/2007239042 conn(0x7efdd00b3660 msgr2=0x7efdd00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- 172.21.15.70:0/2007239042 shutdown_connections 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- 172.21.15.70:0/2007239042 wait complete. 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 Processor -- start 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- start start 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc004810 0x7efdd00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 0x7efdd01541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd0071460 0x7efdd0154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdd00d1360 con 0x7efdd0070fe0 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd00d17f0 con 0x7efdbc004810 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd83d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efdd006d860 con 0x7efdd0071460 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 0x7efdd01541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 0x7efdd01541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41966/0 (socket says 172.21.15.70:41966) 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 -- 172.21.15.70:0/3310666423 learned_addr learned my addr 172.21.15.70:0/3310666423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 -- 172.21.15.70:0/3310666423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd0071460 msgr2=0x7efdd0154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd0071460 0x7efdd0154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 -- 172.21.15.70:0/3310666423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc004810 msgr2=0x7efdd00c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc004810 0x7efdd00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.573+0000 7efdd5971640 1 -- 172.21.15.70:0/3310666423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd0159db0 con 0x7efdd0070fe0 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd5971640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 0x7efdd01541e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efdc001caf0 tx=0x7efdbc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdc6ffd640 1 -- 172.21.15.70:0/3310666423 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0000ee0 con 0x7efdd0070fe0 2024-04-22T01:17:46.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdc6ffd640 1 -- 172.21.15.70:0/3310666423 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc00a5b00 con 0x7efdd0070fe0 2024-04-22T01:17:46.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd015a090 con 0x7efdd0070fe0 2024-04-22T01:17:46.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdc6ffd640 1 -- 172.21.15.70:0/3310666423 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0007180 con 0x7efdd0070fe0 2024-04-22T01:17:46.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd00c6f20 con 0x7efdd0070fe0 2024-04-22T01:17:46.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdc6ffd640 1 -- 172.21.15.70:0/3310666423 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efdc0007320 con 0x7efdd0070fe0 2024-04-22T01:17:46.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdc6ffd640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb4067a00 0x7efdb4069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdc6ffd640 1 -- 172.21.15.70:0/3310666423 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7efdc011f1e0 con 0x7efdd0070fe0 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd6172640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb4067a00 0x7efdb4069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd6172640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb4067a00 0x7efdb4069ec0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efdcc007fd0 tx=0x7efdcc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 Processor -- start 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 -- start start 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff924a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9200680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff9200fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eff904e5e0 con 0x55eff924a810 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eff8fd5c50 con 0x55eff9200bc0 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55eff904a5a0 con 0x55eff9249ef0 2024-04-22T01:17:46.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd5971640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9200680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd5971640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9200680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41970/0 (socket says 172.21.15.70:41970) 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.577+0000 7efdd5971640 1 -- 172.21.15.70:0/4107597133 learned_addr learned my addr 172.21.15.70:0/4107597133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff924a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6973640 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff9200fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd5971640 1 -- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 msgr2=0x55eff924a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd5971640 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff924a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd5971640 1 -- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 msgr2=0x55eff9200fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd5971640 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff9200fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd5971640 1 -- 172.21.15.70:0/4107597133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff8fd3d10 con 0x55eff924a810 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd5971640 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9200680 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efdc00eb8f0 tx=0x7efdc00eb860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9c7336082fa950 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb3fff640 1 -- 172.21.15.70:0/4107597133 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00bd040 con 0x55eff924a810 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb3fff640 1 -- 172.21.15.70:0/4107597133 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc00a9070 con 0x55eff924a810 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb3fff640 1 -- 172.21.15.70:0/4107597133 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00f8070 con 0x55eff924a810 2024-04-22T01:17:46.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/4107597133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 msgr2=0x55eff9200680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9200680 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efdc00eb8f0 tx=0x7efdc00eb860 comp rx=0 tx=0).stop 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/4107597133 shutdown_connections 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff9200fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9200680 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- 172.21.15.70:0/4107597133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff924a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/4107597133 >> 172.21.15.70:0/4107597133 conn(0x55eff91fcb70 msgr2=0x55eff91fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/4107597133 shutdown_connections 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/4107597133 wait complete. 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- start start 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 Processor -- start 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 0x55eff922e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9249ef0 0x55eff922e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff924a810 0x55eff922ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eff904e5e0 con 0x55eff9200bc0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eff9008520 con 0x55eff9249ef0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55eff9052c10 con 0x55eff924a810 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 10 client.0 _ll_register_callbacks cb 0x55eff91eb5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 0x55eff922e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 0x55eff922e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41982/0 (socket says 172.21.15.70:41982) 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 -- 172.21.15.70:0/2325761675 learned_addr learned my addr 172.21.15.70:0/2325761675 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 -- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff924a810 msgr2=0x55eff922ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff924a810 0x55eff922ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 -- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9249ef0 msgr2=0x55eff922e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9249ef0 0x55eff922e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 -- 172.21.15.70:0/2325761675 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff8fe67b0 con 0x55eff9200bc0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdd6172640 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 0x55eff922e2c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efdcc001420 tx=0x7efdcc0012a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f10ddfd3c7f7d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:17:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb17fa640 1 -- 172.21.15.70:0/2325761675 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc01f410 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb17fa640 1 -- 172.21.15.70:0/2325761675 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7efdcc01f5b0 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc00a9070 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb17fa640 1 -- 172.21.15.70:0/2325761675 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efdcc02c050 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 10 client.6894 fetch_fsmap learned FSMap version 37 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55eff8fd5ea0 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb17fa640 1 -- 172.21.15.70:0/2325761675 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc00a9070 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efdb17fa640 1 -- 172.21.15.70:0/2325761675 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7efdcc02f020 con 0x55eff9200bc0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 10 client.6894 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 10 client.6894 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 20 client.6894 populate_metadata read hostname 'smithi070' 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff922f5e0 0x55eff925b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 4 client.6894 mds_command: new command op to 6759 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.581+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 --> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eff8fe9ab0 con 0x55eff922f5e0 2024-04-22T01:17:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efdd5971640 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff922f5e0 0x55eff925b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efdd5971640 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff922f5e0 0x55eff925b460 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc8e79d9098ca60a server_cookie=5f60057934902a0d in_seq=0 out_seq=0 2024-04-22T01:17:46.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efdb17fa640 10 client.6894 ms_handle_connect on v2:172.21.15.98:6832/258015518 2024-04-22T01:17:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efdb17fa640 1 -- 172.21.15.70:0/2325761675 <== mds.0 v2:172.21.15.98:6832/258015518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eff8fe9ab0 con 0x55eff922f5e0 2024-04-22T01:17:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efdb17fa640 10 client.6894 handle_command_reply: tid=0 2024-04-22T01:17:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 client.6894 shutdown 2024-04-22T01:17:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 msgr2=0x55eff922e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 0x55eff922e2c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efdcc001420 tx=0x7efdcc0012a0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 shutdown_connections 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff924a810 0x55eff922ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9249ef0 0x55eff922e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff922f5e0 0x55eff925b460 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 --2- 172.21.15.70:0/2325761675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff9200bc0 0x55eff922e2c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efdb17fa640 0 client.6894 ms_handle_reset on v2:172.21.15.98:6832/258015518 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 >> 172.21.15.70:0/2325761675 conn(0x55eff91fcb70 msgr2=0x55eff91ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 shutdown_connections 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 1 -- 172.21.15.70:0/2325761675 wait complete. 2024-04-22T01:17:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.585+0000 7efddb54f000 20 client.6894 trim_cache size 0 max 16384 2024-04-22T01:17:46.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.601+0000 7efddb54f000 1 Processor -- start 2024-04-22T01:17:46.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.601+0000 7efddb54f000 1 -- start start 2024-04-22T01:17:46.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff92b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff92b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff92b5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdd02011c0 con 0x55eff924a810 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd02001a0 con 0x55eff9200bc0 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efdd00c6440 con 0x55eff9249ef0 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff92b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff92b5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41984/0 (socket says 172.21.15.70:41984) 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/2732127332 learned_addr learned my addr 172.21.15.70:0/2732127332 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6172640 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff92b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd5971640 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff92b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 msgr2=0x55eff92b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff92b4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 msgr2=0x55eff92b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff92b4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/2732127332 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff8fd6e80 con 0x55eff924a810 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff92b5280 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efdd009af60 tx=0x7efdc8003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322b5dfe31a2dc2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdb37fe640 1 -- 172.21.15.70:0/2732127332 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc801b070 con 0x55eff924a810 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdb37fe640 1 -- 172.21.15.70:0/2732127332 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc8016400 con 0x55eff924a810 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdb37fe640 1 -- 172.21.15.70:0/2732127332 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc80167f0 con 0x55eff924a810 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- 172.21.15.70:0/2732127332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 msgr2=0x55eff92b5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff92b5280 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efdd009af60 tx=0x7efdc8003e30 comp rx=0 tx=0).stop 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- 172.21.15.70:0/2732127332 shutdown_connections 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff92b5280 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff92b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- 172.21.15.70:0/2732127332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff92b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- 172.21.15.70:0/2732127332 >> 172.21.15.70:0/2732127332 conn(0x55eff91fcb70 msgr2=0x55eff9204f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- 172.21.15.70:0/2732127332 shutdown_connections 2024-04-22T01:17:46.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- 172.21.15.70:0/2732127332 wait complete. 2024-04-22T01:17:46.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- start start 2024-04-22T01:17:46.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 Processor -- start 2024-04-22T01:17:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff925dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff9274ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9275530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eff904d8f0 con 0x55eff924a810 2024-04-22T01:17:46.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd01ffb30 con 0x55eff9200bc0 2024-04-22T01:17:46.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x55eff901b340 con 0x55eff9249ef0 2024-04-22T01:17:46.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efddb54f000 10 client.0 _ll_register_callbacks cb 0x7efdcc007b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:17:46.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9275530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9275530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42000/0 (socket says 172.21.15.70:42000) 2024-04-22T01:17:46.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/1429757211 learned_addr learned my addr 172.21.15.70:0/1429757211 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6172640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff925dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd5971640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff9274ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 msgr2=0x55eff9274ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff9274ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 msgr2=0x55eff925dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff925dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 -- 172.21.15.70:0/1429757211 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff901b760 con 0x55eff924a810 2024-04-22T01:17:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdd6973640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9275530 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x55eff8fd64d0 tx=0x7efdc8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e2c18e5e98e840 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.605+0000 7efdb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:17:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 1 -- 172.21.15.70:0/1429757211 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc801b050 con 0x55eff924a810 2024-04-22T01:17:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 1 -- 172.21.15.70:0/1429757211 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7efdc8020430 con 0x55eff924a810 2024-04-22T01:17:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc8016400 con 0x55eff924a810 2024-04-22T01:17:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 1 -- 172.21.15.70:0/1429757211 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efdc8009ce0 con 0x55eff924a810 2024-04-22T01:17:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 10 client.6900 fetch_fsmap learned FSMap version 37 2024-04-22T01:17:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55eff90a79e0 con 0x55eff924a810 2024-04-22T01:17:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 1 -- 172.21.15.70:0/1429757211 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc8016400 con 0x55eff924a810 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 1 -- 172.21.15.70:0/1429757211 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7efdc801f070 con 0x55eff924a810 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 10 client.6900 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 10 client.6900 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 20 client.6900 populate_metadata read hostname 'smithi070' 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff9275a70 0x55eff92b9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 4 client.6900 mds_command: new command op to 6759 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:17:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 --> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] -- command(tid 0: {"prefix": "status"}) -- 0x55eff905bb20 con 0x55eff9275a70 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd6172640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff9275a70 0x55eff92b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 10 client.6900 ms_handle_connect on v2:172.21.15.98:6832/258015518 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd6172640 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff9275a70 0x55eff92b9140 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d7311e7856e5bee server_cookie=9f6cffda5c7133bc in_seq=0 out_seq=0 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 1 -- 172.21.15.70:0/1429757211 <== mds.0 v2:172.21.15.98:6832/258015518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55eff905bb20 con 0x55eff9275a70 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 10 client.6900 handle_command_reply: tid=0 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 client.6900 shutdown 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 msgr2=0x55eff9275530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9275530 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x55eff8fd64d0 tx=0x7efdc8009750 comp rx=0 tx=0).stop 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 shutdown_connections 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:6832/258015518,v1:172.21.15.98:6833/258015518] conn(0x55eff9275a70 0x55eff92b9140 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eff924a810 0x55eff9275530 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdb17fa640 0 client.6900 ms_handle_reset on v2:172.21.15.98:6832/258015518 2024-04-22T01:17:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eff9249ef0 0x55eff9274ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 --2- 172.21.15.70:0/1429757211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eff9200bc0 0x55eff925dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 >> 172.21.15.70:0/1429757211 conn(0x55eff91fcb70 msgr2=0x55eff9205800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 shutdown_connections 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 1 -- 172.21.15.70:0/1429757211 wait complete. 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efddb54f000 20 client.6900 trim_cache size 0 max 16384 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:17:46.622 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 6759, 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 4.7345609309999999, 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 37, 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 78, 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 78, 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 9.8620151230000008 2024-04-22T01:17:46.623 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:17:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb4067a00 msgr2=0x7efdb4069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb4067a00 0x7efdb4069ec0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efdcc007fd0 tx=0x7efdcc009040 comp rx=0 tx=0).stop 2024-04-22T01:17:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 msgr2=0x7efdd01541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 0x7efdd01541e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efdc001caf0 tx=0x7efdbc005840 comp rx=0 tx=0).stop 2024-04-22T01:17:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 shutdown_connections 2024-04-22T01:17:46.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efdb4067a00 0x7efdb4069ec0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd0071460 0x7efdd0154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdd0070fe0 0x7efdd01541e0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 --2- 172.21.15.70:0/3310666423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdbc004810 0x7efdd00c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 >> 172.21.15.70:0/3310666423 conn(0x7efdd00b3660 msgr2=0x7efdd00b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 shutdown_connections 2024-04-22T01:17:46.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.609+0000 7efdd83d6640 1 -- 172.21.15.70:0/3310666423 wait complete. 2024-04-22T01:17:46.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:17:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.737+0000 7f1ec5930640 1 Processor -- start 2024-04-22T01:17:46.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.737+0000 7f1ec5930640 1 -- start start 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ec5930640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1ec006e420 0x7f1ec006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ec5930640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 0x7f1ec0066f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ec5930640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec006ed40 con 0x7f1ec00cfcc0 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ec5930640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec006bc00 con 0x7f1ec006e420 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ec5930640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec006bee0 con 0x7f1ec00d0db0 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1ec00cfcc0 0x7f1ec00d0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38784/0 (socket says 172.21.15.70:38784) 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 -- 172.21.15.70:0/4279753921 learned_addr learned my addr 172.21.15.70:0/4279753921 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679103888 0 0) 0x7f1ec006ed40 con 0x7f1ec00cfcc0 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eac003530 con 0x7f1ec00cfcc0 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1eb67fc640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 0x7f1ec0066f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebeffd640 1 --2- 172.21.15.70:0/4279753921 >> v2:172.21.15.98:3300/0 conn(0x7f1ec006e420 0x7f1ec006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2013784194 0 0) 0x7f1eac003530 con 0x7f1ec00cfcc0 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 msgr2=0x7f1ec0066f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 0x7f1ec0066f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 >> v2:172.21.15.98:3300/0 conn(0x7f1ec006e420 msgr2=0x7f1ec006e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 --2- 172.21.15.70:0/4279753921 >> v2:172.21.15.98:3300/0 conn(0x7f1ec006e420 0x7f1ec006e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec006c1c0 con 0x7f1ec00cfcc0 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1eb8001a00 con 0x7f1ec00cfcc0 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 >> v1:172.21.15.70:6789/0 conn(0x7f1ec00cfcc0 legacy=0x7f1ec00d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eac0039c0 0x7f1eac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 0x7f1eac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eac004d50 0x7f1eac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec006bc00 con 0x7f1eac004d50 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec006bee0 con 0x7f1eac0039c0 2024-04-22T01:17:46.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1eac00bb00 con 0x7f1ec00d0db0 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eac004d50 0x7f1eac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1eb67fc640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 0x7f1eac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebeffd640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eac0039c0 0x7f1eac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 -- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 msgr2=0x7f1eac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec00d0db0 0x7f1eac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 -- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eac0039c0 msgr2=0x7f1eac003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eac0039c0 0x7f1eac003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 -- 172.21.15.70:0/4279753921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac003530 con 0x7f1eac004d50 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.741+0000 7f1ebf7fe640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eac004d50 0x7f1eac0051d0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800a8a0 tx=0x7f1eb801d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d938d470df3d3c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800a4c0 con 0x7f1eac004d50 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb80a9020 con 0x7f1eac004d50 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebe7fc640 1 -- 172.21.15.70:0/4279753921 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80072d0 con 0x7f1eac004d50 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/4279753921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eac004d50 msgr2=0x7f1eac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 --2- 172.21.15.70:0/4279753921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eac004d50 0x7f1eac0051d0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800a8a0 tx=0x7f1eb801d3c0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebf7fe640 1 -- 172.21.15.70:0/4279753921 reap_dead start 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/4279753921 shutdown_connections 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/4279753921 >> 172.21.15.70:0/4279753921 conn(0x7f1ec00b3d90 msgr2=0x7f1ec00b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/4279753921 shutdown_connections 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/4279753921 wait complete. 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 Processor -- start 2024-04-22T01:17:46.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- start start 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 0x7f1ec015c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec01f47e0 0x7f1ec00ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec00aea90 0x7f1ec0152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec015e180 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec006c780 con 0x7f1ec00aea90 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec00ae180 con 0x7f1ec01f47e0 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 0x7f1ec015c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 0x7f1ec015c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42024/0 (socket says 172.21.15.70:42024) 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 -- 172.21.15.70:0/125803777 learned_addr learned my addr 172.21.15.70:0/125803777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1eb67fc640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec01f47e0 0x7f1ec00ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebf7fe640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec00aea90 0x7f1ec0152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 -- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec01f47e0 msgr2=0x7f1ec00ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec01f47e0 0x7f1ec00ae550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 -- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec00aea90 msgr2=0x7f1ec0152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec00aea90 0x7f1ec0152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 -- 172.21.15.70:0/125803777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec015b110 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebf7fe640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec00aea90 0x7f1ec0152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebeffd640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 0x7f1ec015c1b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000f5a0 tx=0x7f1eb000fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0019020 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb0014870 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec015b3f0 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0014c60 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.745+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec015b9b0 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.749+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec00aeea0 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.749+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1eb001c020 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.749+0000 7f1ebcff9640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea00678c0 0x7f1ea0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:46.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.749+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f1eb0086630 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.749+0000 7f1eb67fc640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea00678c0 0x7f1ea0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:46.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.749+0000 7f1eb67fc640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea00678c0 0x7f1ea0069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0152be0 tx=0x7f1ea8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:46.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.753+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1ec00aeea0 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.937+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1ec006b850 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.937+0000 7f1ebcff9640 1 -- 172.21.15.70:0/125803777 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7f1eb0023020 con 0x7f1ec00b1ee0 2024-04-22T01:17:46.940 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:17:46.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea00678c0 msgr2=0x7f1ea0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea00678c0 0x7f1ea0069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0152be0 tx=0x7f1ea8008040 comp rx=0 tx=0).stop 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 msgr2=0x7f1ec015c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 0x7f1ec015c1b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000f5a0 tx=0x7f1eb000fa60 comp rx=0 tx=0).stop 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 shutdown_connections 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea00678c0 0x7f1ea0069d80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec00aea90 0x7f1ec0152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec01f47e0 0x7f1ec00ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 --2- 172.21.15.70:0/125803777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ec00b1ee0 0x7f1ec015c1b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 >> 172.21.15.70:0/125803777 conn(0x7f1ec00b3d90 msgr2=0x7f1ec00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 shutdown_connections 2024-04-22T01:17:46.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:46.941+0000 7f1ec5930640 1 -- 172.21.15.70:0/125803777 wait complete. 2024-04-22T01:17:46.957 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:17:46.957 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:17:46.958 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:17:46.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:46.958 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:17:46.964 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:17:46.964 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:17:46.965 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42355sec preferred_lft 42355sec 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:17:46.966 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:46.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:46.968 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:17:47.021 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:17:47.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:47.023 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:47.074 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:47.075 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:17:47.075 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:17:47.075 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:17:47.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:17:47.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:17:47.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:17:47.075 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:17:47.077 INFO:tasks.ceph.mds.d.smithi098.stderr:2024-04-22T01:17:47.076+0000 7f47cdab5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:17:47.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:17:47.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:47.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:47.193 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:17:47.241 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:17:47.242 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:17:47.242 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:17:47.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:47.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:17:47.298 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:17:47.298 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:47.299 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:47.345 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:17:47.345 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:47.346 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:47.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:17:47.454 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:47.504 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:17:47.504 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:17:47.504 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:17:47.504 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:17:47.504 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:17:47.504 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:17:47.505 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:47.506 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:17:47.506 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:47.506 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:17:47.506 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:17:47.506 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:17:47.506 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:47.507 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:47.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:47.509 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:17:47.557 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:17:47.558 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:17:47.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:17:47.618 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:47.665 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:17:47.665 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:17:47.666 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:17:47.667 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:47.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:47.669 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:47.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:47.669 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:17:47.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:17:47.681+0000 7ff7c2a66580 -1 init, newargv = 0x5620192b1a90 newargc=15 2024-04-22T01:17:47.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[43324]: starting ceph client 2024-04-22T01:17:47.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[43324]: starting fuse 2024-04-22T01:17:48.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:17:48.803 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:17:48.852 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:17:48.853 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:17:48.854 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:48.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:48.855 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:17:48.901 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:17:48.902 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:17:48.902 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:17:48.903 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:17:48.904 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:17:48.904 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:17:48.904 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:17:49.024 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.43324.asok 2024-04-22T01:17:49.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:49.035 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43324.asok 2024-04-22T01:17:49.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43324.asok status 2024-04-22T01:17:49.183 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:17:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:17:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:17:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:17:49.183 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "43324", 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:17:49.184 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 6912, 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 6912 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:49.185 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3803548926 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:49.186 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3803548926 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.6912 192.168.0.1:0/3803548926", 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3803548926", 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 37, 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 78, 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:17:49.187 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:17:49.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.197 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:17:49.197 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:49.203 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:17:49.203 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:49.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:17:49.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:17:49.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 Processor -- start 2024-04-22T01:17:49.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 -- start start 2024-04-22T01:17:49.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415aa30 0x7f01e40b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:49.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01e415bb50 0x7f01e40ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:49.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e406dbd0 con 0x7f01e40aec70 2024-04-22T01:17:49.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f01e40af0a0 con 0x7f01e415bb50 2024-04-22T01:17:49.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e9c0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f01e406de70 con 0x7f01e415aa30 2024-04-22T01:17:49.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415aa30 0x7f01e40b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:49.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01daffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01e415bb50 0x7f01e40ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:49.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415aa30 0x7f01e40b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:59508/0 (socket says 172.21.15.70:59508) 2024-04-22T01:17:49.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01e37fe640 1 -- 172.21.15.70:0/2959343693 learned_addr learned my addr 172.21.15.70:0/2959343693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:49.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01daffd640 1 -- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415aa30 msgr2=0x7f01e40b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:49.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01daffd640 1 --2- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415aa30 0x7f01e40b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01daffd640 1 -- 172.21.15.70:0/2959343693 >> v1:172.21.15.98:6789/0 conn(0x7f01e40aec70 legacy=0x7f01e415e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:49.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.425+0000 7f01daffd640 1 -- 172.21.15.70:0/2959343693 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e406e150 con 0x7f01e415bb50 2024-04-22T01:17:49.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01daffd640 1 --2- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415bb50 0x7f01e40ae730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0097c0 tx=0x7f01cc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f99f27ca996d3ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:49.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e2ffd640 1 -- 172.21.15.70:0/2959343693 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0b0070 con 0x7f01e415bb50 2024-04-22T01:17:49.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e2ffd640 1 -- 172.21.15.70:0/2959343693 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc0ab600 con 0x7f01e415bb50 2024-04-22T01:17:49.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e2ffd640 1 -- 172.21.15.70:0/2959343693 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0ab9d0 con 0x7f01e415bb50 2024-04-22T01:17:49.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415bb50 msgr2=0x7f01e40ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:49.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415bb50 0x7f01e40ae730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0097c0 tx=0x7f01cc01c4a0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2959343693 shutdown_connections 2024-04-22T01:17:49.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415bb50 0x7f01e40ae730 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2959343693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415aa30 0x7f01e40b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2959343693 >> 172.21.15.70:0/2959343693 conn(0x7f01e406b1f0 msgr2=0x7f01e406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:49.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2959343693 shutdown_connections 2024-04-22T01:17:49.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.429+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2959343693 wait complete. 2024-04-22T01:17:49.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 Processor -- start 2024-04-22T01:17:49.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- start start 2024-04-22T01:17:49.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 0x7f01e4148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:49.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415aa30 0x7f01e4153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415bb50 0x7f01e4148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01e415af90 con 0x7f01e40aec70 2024-04-22T01:17:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01e415ae10 con 0x7f01e415aa30 2024-04-22T01:17:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f01e406e950 con 0x7f01e415bb50 2024-04-22T01:17:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 0x7f01e4148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 0x7f01e4148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42030/0 (socket says 172.21.15.70:42030) 2024-04-22T01:17:49.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 -- 172.21.15.70:0/2269095630 learned_addr learned my addr 172.21.15.70:0/2269095630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:49.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01daffd640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415aa30 0x7f01e4153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:49.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e3fff640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415bb50 0x7f01e4148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:49.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 -- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415bb50 msgr2=0x7f01e4148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:49.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415bb50 0x7f01e4148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 -- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415aa30 msgr2=0x7f01e4153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415aa30 0x7f01e4153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 -- 172.21.15.70:0/2269095630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e4153a70 con 0x7f01e40aec70 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01daffd640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415aa30 0x7f01e4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e37fe640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 0x7f01e4148830 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f01d400e7f0 tx=0x7f01d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4017070 con 0x7f01e40aec70 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d400c940 con 0x7f01e40aec70 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d400cd30 con 0x7f01e40aec70 2024-04-22T01:17:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e4153d50 con 0x7f01e40aec70 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e4153760 con 0x7f01e40aec70 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.433+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e4068210 con 0x7f01e40aec70 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.437+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01d401a070 con 0x7f01e40aec70 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.437+0000 7f01e0ff9640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01c40678c0 0x7f01c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.437+0000 7f01daffd640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01c40678c0 0x7f01c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.437+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f01d4086200 con 0x7f01e40aec70 2024-04-22T01:17:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.437+0000 7f01daffd640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01c40678c0 0x7f01c4069d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0ad000 tx=0x7f01cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:49.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.441+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f01d4056a00 con 0x7f01e40aec70 2024-04-22T01:17:49.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f01e406f190 con 0x7f01e40aec70 2024-04-22T01:17:49.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e0ff9640 1 -- 172.21.15.70:0/2269095630 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7f01d4056300 con 0x7f01e40aec70 2024-04-22T01:17:49.638 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:17:49.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01c40678c0 msgr2=0x7f01c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:49.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01c40678c0 0x7f01c4069d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0ad000 tx=0x7f01cc008040 comp rx=0 tx=0).stop 2024-04-22T01:17:49.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 msgr2=0x7f01e4148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:49.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 0x7f01e4148830 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f01d400e7f0 tx=0x7f01d400ecb0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 shutdown_connections 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f01c40678c0 0x7f01c4069d80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e415bb50 0x7f01e4148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e415aa30 0x7f01e4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 --2- 172.21.15.70:0/2269095630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01e40aec70 0x7f01e4148830 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 >> 172.21.15.70:0/2269095630 conn(0x7f01e406b1f0 msgr2=0x7f01e4157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 shutdown_connections 2024-04-22T01:17:49.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:49.637+0000 7f01e9c0e640 1 -- 172.21.15.70:0/2269095630 wait complete. 2024-04-22T01:17:49.657 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:17:49.657 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:17:49.657 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:17:49.658 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:17:49.658 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:49.658 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:17:49.658 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:17:49.658 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:17:49.658 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:17:49.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.659 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:17:49.666 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42356sec preferred_lft 42356sec 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:17:49.668 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:17:49.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.669 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:17:49.721 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:17:49.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.723 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:17:49.775 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:17:49.775 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:17:49.775 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:17:49.776 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:17:49.776 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:17:49.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:17:49.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:17:49.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:17:49.777 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:17:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:17:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:17:49.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:17:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:49.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.899 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:17:49.904 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:17:49.904 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:17:49.905 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:17:49.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:49.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:17:49.961 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:17:49.962 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:49.962 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:17:50.010 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:17:50.010 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:50.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:17:50.119 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:17:50.167 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:50.168 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:50.169 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:50.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:50.170 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:17:50.218 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:17:50.218 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:17:50.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:17:50.279 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:17:50.327 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:17:50.328 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:50.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:50.329 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:17:50.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:17:50.336+0000 7fdb62e8c580 -1 init, newargv = 0x563a9cfa86b0 newargc=15 2024-04-22T01:17:50.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34380]: starting ceph client 2024-04-22T01:17:50.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34380]: starting fuse 2024-04-22T01:17:51.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:17:51.460 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:51.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:51.512 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:17:51.558 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:17:51.559 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:17:51.559 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:17:51.560 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.34380.asok 2024-04-22T01:17:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:17:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:17:51.692 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34380.asok 2024-04-22T01:17:51.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:51.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34380.asok status 2024-04-22T01:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "34380", 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5258, 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5258 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1875170439 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1875170439 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5258 192.168.0.1:0/1875170439", 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/1875170439", 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 37, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 78, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:17:51.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:51.860 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:17:51.860 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:51.866 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:17:51.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:51.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:51.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:51.925 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:51.925 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:51.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:52.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 Processor -- start 2024-04-22T01:17:52.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 -- start start 2024-04-22T01:17:52.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf01455b0 0x7f6bf00cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6bf00cc150 0x7f6bf0071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf00d0250 con 0x7f6bf0072060 2024-04-22T01:17:52.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf00724d0 con 0x7f6bf00cc150 2024-04-22T01:17:52.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bf53f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf00cd7b0 con 0x7f6bf01455b0 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6bf00cc150 0x7f6bf0071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6bf00cc150 0x7f6bf0071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:39176/0 (socket says 172.21.15.70:39176) 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bef7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6bf0072060 0x7f6bf015f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:46946/0 (socket says 172.21.15.70:46946) 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 -- 172.21.15.70:0/481564336 learned_addr learned my addr 172.21.15.70:0/481564336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6beeffd640 1 --2- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf01455b0 0x7f6bf00cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 -- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf01455b0 msgr2=0x7f6bf00cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:52.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 --2- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf01455b0 0x7f6bf00cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 -- 172.21.15.70:0/481564336 >> v1:172.21.15.98:6789/0 conn(0x7f6bf0072060 legacy=0x7f6bf015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.105+0000 7f6bee7fc640 1 -- 172.21.15.70:0/481564336 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf00cc710 con 0x7f6bf00cc150 2024-04-22T01:17:52.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bee7fc640 1 --2- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf00cc150 0x7f6bf0071b20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009870 tx=0x7f6be401c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27204d6bd6f50f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bedffb640 1 -- 172.21.15.70:0/481564336 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be40b0070 con 0x7f6bf00cc150 2024-04-22T01:17:52.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bedffb640 1 -- 172.21.15.70:0/481564336 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be40ab600 con 0x7f6bf00cc150 2024-04-22T01:17:52.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bedffb640 1 -- 172.21.15.70:0/481564336 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be40ab9d0 con 0x7f6bf00cc150 2024-04-22T01:17:52.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 -- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf00cc150 msgr2=0x7f6bf0071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf00cc150 0x7f6bf0071b20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009870 tx=0x7f6be401c440 comp rx=0 tx=0).stop 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 -- 172.21.15.70:0/481564336 shutdown_connections 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf00cc150 0x7f6bf0071b20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/481564336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf01455b0 0x7f6bf00cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 -- 172.21.15.70:0/481564336 >> 172.21.15.70:0/481564336 conn(0x7f6bf006b1f0 msgr2=0x7f6bf006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 -- 172.21.15.70:0/481564336 shutdown_connections 2024-04-22T01:17:52.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 -- 172.21.15.70:0/481564336 wait complete. 2024-04-22T01:17:52.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.109+0000 7f6bf53f6640 1 Processor -- start 2024-04-22T01:17:52.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- start start 2024-04-22T01:17:52.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf0072060 0x7f6bf0143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf00cc150 0x7f6bf0143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 0x7f6bf015cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf0168b40 con 0x7f6bf01455b0 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf006f210 con 0x7f6bf0072060 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf0168cc0 con 0x7f6bf00cc150 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 0x7f6bf015cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 0x7f6bf015cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42038/0 (socket says 172.21.15.70:42038) 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 -- 172.21.15.70:0/2134816227 learned_addr learned my addr 172.21.15.70:0/2134816227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6beeffd640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf0072060 0x7f6bf0143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bee7fc640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf00cc150 0x7f6bf0143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 -- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf00cc150 msgr2=0x7f6bf0143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf00cc150 0x7f6bf0143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 -- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf0072060 msgr2=0x7f6bf0143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf0072060 0x7f6bf0143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 -- 172.21.15.70:0/2134816227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf015c820 con 0x7f6bf01455b0 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6beeffd640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf0072060 0x7f6bf0143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bef7fe640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 0x7f6bf015cfe0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c700 tx=0x7f6be800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8017070 con 0x7f6bf01455b0 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be800ce10 con 0x7f6bf01455b0 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be800a840 con 0x7f6bf01455b0 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf013b430 con 0x7f6bf01455b0 2024-04-22T01:17:52.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf013b9f0 con 0x7f6bf01455b0 2024-04-22T01:17:52.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf013bc30 con 0x7f6bf01455b0 2024-04-22T01:17:52.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.113+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6be801a070 con 0x7f6bf01455b0 2024-04-22T01:17:52.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.117+0000 7f6bd77fe640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6bdc0678c0 0x7f6bdc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.117+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f6be80852f0 con 0x7f6bf01455b0 2024-04-22T01:17:52.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.117+0000 7f6beeffd640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6bdc0678c0 0x7f6bdc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.117+0000 7f6beeffd640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6bdc0678c0 0x7f6bdc069d80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009bb0 tx=0x7f6be0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.121+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6bf013bc30 con 0x7f6bf01455b0 2024-04-22T01:17:52.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.281+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bf0068210 con 0x7f6bf01455b0 2024-04-22T01:17:52.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.281+0000 7f6bd77fe640 1 -- 172.21.15.70:0/2134816227 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7f6be8055650 con 0x7f6bf01455b0 2024-04-22T01:17:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:52.284 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:52.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6bdc0678c0 msgr2=0x7f6bdc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6bdc0678c0 0x7f6bdc069d80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009bb0 tx=0x7f6be0008040 comp rx=0 tx=0).stop 2024-04-22T01:17:52.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 msgr2=0x7f6bf015cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 0x7f6bf015cfe0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c700 tx=0x7f6be800cbc0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 shutdown_connections 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6bdc0678c0 0x7f6bdc069d80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bf01455b0 0x7f6bf015cfe0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bf00cc150 0x7f6bf0143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 --2- 172.21.15.70:0/2134816227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bf0072060 0x7f6bf0143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 >> 172.21.15.70:0/2134816227 conn(0x7f6bf006b1f0 msgr2=0x7f6bf00af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 shutdown_connections 2024-04-22T01:17:52.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.285+0000 7f6bf53f6640 1 -- 172.21.15.70:0/2134816227 wait complete. 2024-04-22T01:17:52.302 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:39.433363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:40.173738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:52.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:52.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2bd1c640 1 Processor -- start 2024-04-22T01:17:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2bd1c640 1 -- start start 2024-04-22T01:17:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2bd1c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1d240cdd70 0x7f1d2406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2bd1c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2406b860 con 0x7f1d2415df10 2024-04-22T01:17:52.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2bd1c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2406bb40 con 0x7f1d240b21a0 2024-04-22T01:17:52.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2bd1c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1d24161a60 con 0x7f1d240cdd70 2024-04-22T01:17:52.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2a2b9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1d240cdd70 0x7f1d2406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d29ab8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1d2415df10 0x7f1d240b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38792/0 (socket says 172.21.15.70:38792) 2024-04-22T01:17:52.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d29ab8640 1 -- 172.21.15.70:0/406387720 learned_addr learned my addr 172.21.15.70:0/406387720 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:52.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d28ab6640 1 -- 172.21.15.70:0/406387720 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19486367 0 0) 0x7f1d2406b860 con 0x7f1d2415df10 2024-04-22T01:17:52.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/406387720 >> v1:172.21.15.98:6789/0 conn(0x7f1d240b21a0 legacy=0x7f1d240cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:52.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/406387720 >> v1:172.21.15.70:6789/0 conn(0x7f1d2415df10 legacy=0x7f1d240b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.429+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/406387720 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d24161d40 con 0x7f1d240cdd70 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2a2b9640 1 --2- 172.21.15.70:0/406387720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d240cdd70 0x7f1d2406b1f0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009d90 tx=0x7f1d1c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f6909ded47666d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d28ab6640 1 -- 172.21.15.70:0/406387720 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0b2070 con 0x7f1d240cdd70 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d28ab6640 1 -- 172.21.15.70:0/406387720 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1c0a5880 con 0x7f1d240cdd70 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d28ab6640 1 -- 172.21.15.70:0/406387720 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0a5c50 con 0x7f1d240cdd70 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/406387720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d240cdd70 msgr2=0x7f1d2406b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/406387720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d240cdd70 0x7f1d2406b1f0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009d90 tx=0x7f1d1c01c8b0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/406387720 shutdown_connections 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/406387720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d240cdd70 0x7f1d2406b1f0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/406387720 >> 172.21.15.70:0/406387720 conn(0x7f1d24099a60 msgr2=0x7f1d24099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/406387720 shutdown_connections 2024-04-22T01:17:52.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/406387720 wait complete. 2024-04-22T01:17:52.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 Processor -- start 2024-04-22T01:17:52.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 -- start start 2024-04-22T01:17:52.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1d240b21a0 0x7f1d24152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.433+0000 7f1d2bd1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d240cdd70 0x7f1d2415b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 0x7f1d240d0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2416bf30 con 0x7f1d2415df10 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2416bdb0 con 0x7f1d240cdd70 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2416c0b0 con 0x7f1d240b21a0 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 0x7f1d240d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 0x7f1d240d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42062/0 (socket says 172.21.15.70:42062) 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/3281849886 learned_addr learned my addr 172.21.15.70:0/3281849886 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d29ab8640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1d240b21a0 0x7f1d24152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1d240b21a0 msgr2=0x7f1d24152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1d240b21a0 0x7f1d24152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d240cdd70 msgr2=0x7f1d2415b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d292b7640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d240cdd70 0x7f1d2415b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d240cdd70 0x7f1d2415b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 -- 172.21.15.70:0/3281849886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2415bc40 con 0x7f1d2415df10 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d29ab8640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1d240b21a0 0x7f1d24152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d292b7640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d240cdd70 0x7f1d2415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2a2b9640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 0x7f1d240d0430 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0b0040 tx=0x7f1d1c019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0bd020 con 0x7f1d2415df10 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1c0ad6c0 con 0x7f1d2415df10 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2415bf20 con 0x7f1d2415df10 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0b2040 con 0x7f1d2415df10 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2415c4e0 con 0x7f1d2415df10 2024-04-22T01:17:52.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d24068920 con 0x7f1d2415df10 2024-04-22T01:17:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d1c0c0020 con 0x7f1d2415df10 2024-04-22T01:17:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d1a7fc640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d080678f0 0x7f1d08069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d29ab8640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d080678f0 0x7f1d08069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f1d1c120950 con 0x7f1d2415df10 2024-04-22T01:17:52.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.437+0000 7f1d29ab8640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d080678f0 0x7f1d08069db0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1d20000f70 tx=0x7f1d20009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.441+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1d1c0c8020 con 0x7f1d2415df10 2024-04-22T01:17:52.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.601+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d2406c0e0 con 0x7f1d2415df10 2024-04-22T01:17:52.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d1a7fc640 1 -- 172.21.15.70:0/3281849886 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f1d1c016560 con 0x7f1d2415df10 2024-04-22T01:17:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d080678f0 msgr2=0x7f1d08069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d080678f0 0x7f1d08069db0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1d20000f70 tx=0x7f1d20009040 comp rx=0 tx=0).stop 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 msgr2=0x7f1d240d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 0x7f1d240d0430 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0b0040 tx=0x7f1d1c019650 comp rx=0 tx=0).stop 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 shutdown_connections 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1d080678f0 0x7f1d08069db0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d2415df10 0x7f1d240d0430 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d240cdd70 0x7f1d2415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 --2- 172.21.15.70:0/3281849886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1d240b21a0 0x7f1d24152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 >> 172.21.15.70:0/3281849886 conn(0x7f1d24099a60 msgr2=0x7f1d2406fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:52.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 shutdown_connections 2024-04-22T01:17:52.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.605+0000 7f1d2bd1c640 1 -- 172.21.15.70:0/3281849886 wait complete. 2024-04-22T01:17:52.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:17:52.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 Processor -- start 2024-04-22T01:17:52.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 -- start start 2024-04-22T01:17:52.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c40ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13c40cd050 0x7f13c40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c415c290 con 0x7f13c40b4450 2024-04-22T01:17:52.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f13c415c570 con 0x7f13c40cd050 2024-04-22T01:17:52.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13cc194640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f13c415c850 con 0x7f13c4072280 2024-04-22T01:17:52.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13c9f30640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f13c40b4450 0x7f13c4071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:46958/0 (socket says 172.21.15.70:46958) 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13c9f30640 1 -- 172.21.15.70:0/755238260 learned_addr learned my addr 172.21.15.70:0/755238260 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13ca731640 1 --2- 172.21.15.70:0/755238260 >> v2:172.21.15.98:3300/0 conn(0x7f13c40cd050 0x7f13c40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13c972f640 1 --2- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c40ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13ca731640 1 -- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 msgr2=0x7f13c40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13ca731640 1 --2- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c40ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13ca731640 1 -- 172.21.15.70:0/755238260 >> v1:172.21.15.98:6789/0 conn(0x7f13c40b4450 legacy=0x7f13c4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.745+0000 7f13ca731640 1 -- 172.21.15.70:0/755238260 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c415cb30 con 0x7f13c40cd050 2024-04-22T01:17:52.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13ca731640 1 --2- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c40b2a40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009d90 tx=0x7f13bc01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b6565d88e6f00e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13c8f2e640 1 -- 172.21.15.70:0/755238260 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0b2070 con 0x7f13c40cd050 2024-04-22T01:17:52.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13c8f2e640 1 -- 172.21.15.70:0/755238260 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13bc0a59c0 con 0x7f13c40cd050 2024-04-22T01:17:52.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13c8f2e640 1 -- 172.21.15.70:0/755238260 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0a5d90 con 0x7f13c40cd050 2024-04-22T01:17:52.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 -- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 msgr2=0x7f13c40b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 --2- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c40b2a40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009d90 tx=0x7f13bc01c8e0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 -- 172.21.15.70:0/755238260 shutdown_connections 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 --2- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c40b2a40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 --2- 172.21.15.70:0/755238260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c40ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 -- 172.21.15.70:0/755238260 >> 172.21.15.70:0/755238260 conn(0x7f13c40b1d40 msgr2=0x7f13c40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 -- 172.21.15.70:0/755238260 shutdown_connections 2024-04-22T01:17:52.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 -- 172.21.15.70:0/755238260 wait complete. 2024-04-22T01:17:52.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 Processor -- start 2024-04-22T01:17:52.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.749+0000 7f13cc194640 1 -- start start 2024-04-22T01:17:52.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c413f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 0x7f13c4150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13c40cd980 con 0x7f13c40b4450 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13c40cd800 con 0x7f13c40cd050 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f13c40b3790 con 0x7f13c4072280 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 0x7f13c4150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 0x7f13c4150290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42076/0 (socket says 172.21.15.70:42076) 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 -- 172.21.15.70:0/949616198 learned_addr learned my addr 172.21.15.70:0/949616198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c9f30640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c413f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ca731640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 -- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 msgr2=0x7f13c413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c413f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 -- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 msgr2=0x7f13c406f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c406f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 -- 172.21.15.70:0/949616198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c406e1c0 con 0x7f13c40b4450 2024-04-22T01:17:52.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c972f640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 0x7f13c4150290 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f13b400e960 tx=0x7f13b400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4017070 con 0x7f13c40b4450 2024-04-22T01:17:52.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b400ca00 con 0x7f13c40b4450 2024-04-22T01:17:52.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c406e4a0 con 0x7f13c40b4450 2024-04-22T01:17:52.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b400cdb0 con 0x7f13c40b4450 2024-04-22T01:17:52.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c406ea60 con 0x7f13c40b4450 2024-04-22T01:17:52.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c4068210 con 0x7f13c40b4450 2024-04-22T01:17:52.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13b401a070 con 0x7f13c40b4450 2024-04-22T01:17:52.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ba7fc640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a80678c0 0x7f13a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:52.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f13b4086510 con 0x7f13c40b4450 2024-04-22T01:17:52.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c9f30640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a80678c0 0x7f13a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:52.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.753+0000 7f13c9f30640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a80678c0 0x7f13a8069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f13c414f530 tx=0x7f13c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:52.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.757+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f13b4057030 con 0x7f13c40b4450 2024-04-22T01:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.921+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f13c40d01e0 con 0x7f13c40b4450 2024-04-22T01:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.921+0000 7f13ba7fc640 1 -- 172.21.15.70:0/949616198 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f13b4056930 con 0x7f13c40b4450 2024-04-22T01:17:52.923 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.921+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a80678c0 msgr2=0x7f13a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.921+0000 7f13cc194640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a80678c0 0x7f13a8069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f13c414f530 tx=0x7f13c0009040 comp rx=0 tx=0).stop 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.921+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 msgr2=0x7f13c4150290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.921+0000 7f13cc194640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 0x7f13c4150290 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f13b400e960 tx=0x7f13b400ee20 comp rx=0 tx=0).stop 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 shutdown_connections 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f13a80678c0 0x7f13a8069d80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13c40cd050 0x7f13c406f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13c40b4450 0x7f13c4150290 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 --2- 172.21.15.70:0/949616198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13c4072280 0x7f13c413f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 >> 172.21.15.70:0/949616198 conn(0x7f13c40b1d40 msgr2=0x7f13c40cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 shutdown_connections 2024-04-22T01:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:52.925+0000 7f13cc194640 1 -- 172.21.15.70:0/949616198 wait complete. 2024-04-22T01:17:52.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:17:53.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa84f5640 1 Processor -- start 2024-04-22T01:17:53.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa84f5640 1 -- start start 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa84f5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7fa00b98a0 0x7f7fa00d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa84f5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa00b9ce0 con 0x7f7fa00cea10 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa84f5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa00b75e0 con 0x7f7fa015de20 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa84f5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7fa00b78c0 con 0x7f7fa00b98a0 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7fa00cea10 0x7f7fa00cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38808/0 (socket says 172.21.15.70:38808) 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 -- 172.21.15.70:0/212291067 learned_addr learned my addr 172.21.15.70:0/212291067 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa5a90640 1 --2- 172.21.15.70:0/212291067 >> v2:172.21.15.98:3300/0 conn(0x7f7fa00b98a0 0x7f7fa00d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1376255933 0 0) 0x7f7fa00b9ce0 con 0x7f7fa00cea10 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f8c003530 con 0x7f7fa00cea10 2024-04-22T01:17:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 765441498 0 0) 0x7f7f8c003530 con 0x7f7fa00cea10 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 >> v2:172.21.15.98:3300/0 conn(0x7f7fa00b98a0 msgr2=0x7f7fa00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 --2- 172.21.15.70:0/212291067 >> v2:172.21.15.98:3300/0 conn(0x7f7fa00b98a0 0x7f7fa00d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 >> v1:172.21.15.98:6789/0 conn(0x7f7fa015de20 legacy=0x7f7fa00b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa00b7ba0 con 0x7f7fa00cea10 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7f98001a00 con 0x7f7fa00cea10 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 >> v1:172.21.15.70:6789/0 conn(0x7f7fa00cea10 legacy=0x7f7fa00cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c0039c0 0x7f7f8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c004810 0x7f7f8c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f8c0051d0 0x7f7f8c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa00b78c0 con 0x7f7f8c0051d0 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c00c080 con 0x7f7f8c0039c0 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8c00c280 con 0x7f7f8c004810 2024-04-22T01:17:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f8c0051d0 0x7f7f8c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6291640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c0039c0 0x7f7f8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa5a90640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c004810 0x7f7f8c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 -- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c004810 msgr2=0x7f7f8c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c004810 0x7f7f8c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 -- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c0039c0 msgr2=0x7f7f8c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c0039c0 0x7f7f8c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa6a92640 1 -- 172.21.15.70:0/212291067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c003530 con 0x7f7f8c0051d0 2024-04-22T01:17:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.065+0000 7f7fa5a90640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c004810 0x7f7f8c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa6a92640 1 -- 172.21.15.70:0/212291067 reap_dead start 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f8c0051d0 0x7f7f8c00b9d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800a840 tx=0x7f7f9801d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8e5ab04390a39e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800a460 con 0x7f7f8c0051d0 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f980ac640 con 0x7f7f8c0051d0 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa528f640 1 -- 172.21.15.70:0/212291067 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f980aca30 con 0x7f7f8c0051d0 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- 172.21.15.70:0/212291067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f8c0051d0 msgr2=0x7f7f8c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f8c0051d0 0x7f7f8c00b9d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800a840 tx=0x7f7f9801d360 comp rx=0 tx=0).stop 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- 172.21.15.70:0/212291067 shutdown_connections 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/212291067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f8c0051d0 0x7f7f8c00b9d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- 172.21.15.70:0/212291067 >> 172.21.15.70:0/212291067 conn(0x7f7fa0072d20 msgr2=0x7f7fa0073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- 172.21.15.70:0/212291067 shutdown_connections 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- 172.21.15.70:0/212291067 wait complete. 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 Processor -- start 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- start start 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c0051d0 0x7f7fa01f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa00c9bd0 0x7f7fa00ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 0x7f7fa00c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa015e200 con 0x7f7fa00c7a70 2024-04-22T01:17:53.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa00b8160 con 0x7f7fa00c9bd0 2024-04-22T01:17:53.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.069+0000 7f7fa84f5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa00cee60 con 0x7f7f8c0051d0 2024-04-22T01:17:53.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 0x7f7fa00c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 0x7f7fa00c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42090/0 (socket says 172.21.15.70:42090) 2024-04-22T01:17:53.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 -- 172.21.15.70:0/3498327179 learned_addr learned my addr 172.21.15.70:0/3498327179 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:53.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6291640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c0051d0 0x7f7fa01f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa5a90640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa00c9bd0 0x7f7fa00ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 -- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c0051d0 msgr2=0x7f7fa01f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c0051d0 0x7f7fa01f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 -- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa00c9bd0 msgr2=0x7f7fa00ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa00c9bd0 0x7f7fa00ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 -- 172.21.15.70:0/3498327179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa00c8820 con 0x7f7fa00c7a70 2024-04-22T01:17:53.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6a92640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 0x7f7fa00c7f20 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7f9801d190 tx=0x7f7f8c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f98001fc0 con 0x7f7fa00c7a70 2024-04-22T01:17:53.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f98001260 con 0x7f7fa00c7a70 2024-04-22T01:17:53.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa00c8b00 con 0x7f7fa00c7a70 2024-04-22T01:17:53.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f98004b70 con 0x7f7fa00c7a70 2024-04-22T01:17:53.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa015f550 con 0x7f7fa00c7a70 2024-04-22T01:17:53.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa0068210 con 0x7f7fa00c7a70 2024-04-22T01:17:53.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f980ac640 con 0x7f7fa00c7a70 2024-04-22T01:17:53.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7f96ffd640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7f840678f0 0x7f7f84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f7f9811f810 con 0x7f7fa00c7a70 2024-04-22T01:17:53.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6291640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7f840678f0 0x7f7f84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.073+0000 7f7fa6291640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7f840678f0 0x7f7f84069db0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c000c10 tx=0x7f7f9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.077+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7f980bc070 con 0x7f7fa00c7a70 2024-04-22T01:17:53.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7fa00cf7d0 con 0x7f7fa00c7a70 2024-04-22T01:17:53.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7f96ffd640 1 -- 172.21.15.70:0/3498327179 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f7f98017070 con 0x7f7fa00c7a70 2024-04-22T01:17:53.237 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:17:53.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7f840678f0 msgr2=0x7f7f84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7f840678f0 0x7f7f84069db0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c000c10 tx=0x7f7f9c009040 comp rx=0 tx=0).stop 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 msgr2=0x7f7fa00c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 0x7f7fa00c7f20 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7f9801d190 tx=0x7f7f8c005840 comp rx=0 tx=0).stop 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 shutdown_connections 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7f840678f0 0x7f7f84069db0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7fa00c7a70 0x7f7fa00c7f20 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa00c9bd0 0x7f7fa00ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 --2- 172.21.15.70:0/3498327179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f8c0051d0 0x7f7fa01f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 >> 172.21.15.70:0/3498327179 conn(0x7f7fa0072d20 msgr2=0x7f7fa00cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 shutdown_connections 2024-04-22T01:17:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.237+0000 7f7fa84f5640 1 -- 172.21.15.70:0/3498327179 wait complete. 2024-04-22T01:17:53.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:17:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 Processor -- start 2024-04-22T01:17:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 -- start start 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 0x7f44700cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f447006f480 0x7f4470071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44700cd7a0 con 0x7f447015bf30 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f447015f8b0 con 0x7f447006f480 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4477817640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f447015fb00 con 0x7f447015bb50 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f447015bf30 0x7f447006ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38816/0 (socket says 172.21.15.70:38816) 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 -- 172.21.15.70:0/1745771617 learned_addr learned my addr 172.21.15.70:0/1745771617 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824579478 0 0) 0x7f44700cd7a0 con 0x7f447015bf30 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4475db4640 1 --2- 172.21.15.70:0/1745771617 >> v2:172.21.15.98:3300/0 conn(0x7f447006f480 0x7f4470071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f445c003530 con 0x7f447015bf30 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f44755b3640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 0x7f44700cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538393991 0 0) 0x7f445c003530 con 0x7f447015bf30 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 msgr2=0x7f44700cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 0x7f44700cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 >> v2:172.21.15.98:3300/0 conn(0x7f447006f480 msgr2=0x7f4470071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 --2- 172.21.15.70:0/1745771617 >> v2:172.21.15.98:3300/0 conn(0x7f447006f480 0x7f4470071af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447015fde0 con 0x7f447015bf30 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f4460001680 con 0x7f447015bf30 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 >> v1:172.21.15.70:6789/0 conn(0x7f447015bf30 legacy=0x7f447006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c003a30 0x7f445c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f445c004810 0x7f445c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 0x7f445c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f447015f8b0 con 0x7f445c004810 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f447015fb00 con 0x7f445c003a30 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f445c00b9d0 con 0x7f447015bb50 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f445c004810 0x7f445c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4475db4640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 0x7f445c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 -- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 msgr2=0x7f445c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f44755b3640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c003a30 0x7f445c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447015bb50 0x7f445c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 -- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c003a30 msgr2=0x7f445c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c003a30 0x7f445c003e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.365+0000 7f4474db2640 1 -- 172.21.15.70:0/1745771617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c003530 con 0x7f445c004810 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4474db2640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f445c004810 0x7f445c004c90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f446000c240 tx=0x7f446001f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717cacb51a6c47f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460007d60 con 0x7f445c004810 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4460005ae0 con 0x7f445c004810 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4467fff640 1 -- 172.21.15.70:0/1745771617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460007070 con 0x7f445c004810 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- 172.21.15.70:0/1745771617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f445c004810 msgr2=0x7f445c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 --2- 172.21.15.70:0/1745771617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f445c004810 0x7f445c004c90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f446000c240 tx=0x7f446001f3f0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4475db4640 1 -- 172.21.15.70:0/1745771617 reap_dead start 2024-04-22T01:17:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- 172.21.15.70:0/1745771617 shutdown_connections 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- 172.21.15.70:0/1745771617 >> 172.21.15.70:0/1745771617 conn(0x7f44700cf9e0 msgr2=0x7f44700cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- 172.21.15.70:0/1745771617 shutdown_connections 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- 172.21.15.70:0/1745771617 wait complete. 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 Processor -- start 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- start start 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c004810 0x7f4470157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 0x7f4470152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447014cfd0 0x7f447014d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44700cf460 con 0x7f4470152000 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44700cf5e0 con 0x7f445c004810 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4477817640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f44701f7810 con 0x7f447014cfd0 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4474db2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 0x7f4470152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4474db2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 0x7f4470152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42116/0 (socket says 172.21.15.70:42116) 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4474db2640 1 -- 172.21.15.70:0/4027419139 learned_addr learned my addr 172.21.15.70:0/4027419139 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f44755b3640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c004810 0x7f4470157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.369+0000 7f4475db4640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447014cfd0 0x7f447014d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4474db2640 1 -- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447014cfd0 msgr2=0x7f447014d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4474db2640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447014cfd0 0x7f447014d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4474db2640 1 -- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c004810 msgr2=0x7f4470157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4474db2640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c004810 0x7f4470157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4474db2640 1 -- 172.21.15.70:0/4027419139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4470152b70 con 0x7f4470152000 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4474db2640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 0x7f4470152480 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f446000c240 tx=0x7f445c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460005c90 con 0x7f4470152000 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4460005e30 con 0x7f4470152000 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4470152e50 con 0x7f4470152000 2024-04-22T01:17:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460004a10 con 0x7f4470152000 2024-04-22T01:17:53.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4470153410 con 0x7f4470152000 2024-04-22T01:17:53.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f447004f510 con 0x7f4470152000 2024-04-22T01:17:53.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f44600ba030 con 0x7f4470152000 2024-04-22T01:17:53.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4465ffb640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44540678c0 0x7f4454069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f4460122350 con 0x7f4470152000 2024-04-22T01:17:53.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f44755b3640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44540678c0 0x7f4454069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.373+0000 7f44755b3640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44540678c0 0x7f4454069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f446c009a00 tx=0x7f446c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.377+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4460019100 con 0x7f4470152000 2024-04-22T01:17:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.537+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f44700d0740 con 0x7f4470152000 2024-04-22T01:17:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.537+0000 7f4465ffb640 1 -- 172.21.15.70:0/4027419139 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) ==== 80+0+0 (secure 0 0 0) 0x7f44600f2740 con 0x7f4470152000 2024-04-22T01:17:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:17:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44540678c0 msgr2=0x7f4454069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44540678c0 0x7f4454069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f446c009a00 tx=0x7f446c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 msgr2=0x7f4470152480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 0x7f4470152480 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f446000c240 tx=0x7f445c005840 comp rx=0 tx=0).stop 2024-04-22T01:17:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 shutdown_connections 2024-04-22T01:17:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f44540678c0 0x7f4454069d80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447014cfd0 0x7f447014d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4470152000 0x7f4470152480 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 --2- 172.21.15.70:0/4027419139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f445c004810 0x7f4470157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 >> 172.21.15.70:0/4027419139 conn(0x7f44700cf9e0 msgr2=0x7f447006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 shutdown_connections 2024-04-22T01:17:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.541+0000 7f4477817640 1 -- 172.21.15.70:0/4027419139 wait complete. 2024-04-22T01:17:53.558 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:17:53.559 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:17:53.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:17:53.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.681+0000 7f821f260640 1 Processor -- start 2024-04-22T01:17:53.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.681+0000 7f821f260640 1 -- start start 2024-04-22T01:17:53.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821f260640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f8218072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821f260640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82180b3cd0 0x7f8218161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821f260640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821815bf20 con 0x7f821806ed40 2024-04-22T01:17:53.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821f260640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f821806e8b0 con 0x7f82180b3cd0 2024-04-22T01:17:53.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821f260640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f82180aec30 con 0x7f821815bb40 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f821806ed40 0x7f82180b35c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38822/0 (socket says 172.21.15.70:38822) 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 -- 172.21.15.70:0/1253065657 learned_addr learned my addr 172.21.15.70:0/1253065657 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821cffc640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f8218072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821d7fd640 1 --2- 172.21.15.70:0/1253065657 >> v2:172.21.15.98:3300/0 conn(0x7f82180b3cd0 0x7f8218161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123397249 0 0) 0x7f821815bf20 con 0x7f821806ed40 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8200003530 con 0x7f821806ed40 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1279203633 0 0) 0x7f8200003530 con 0x7f821806ed40 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 msgr2=0x7f8218072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f8218072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 >> v2:172.21.15.98:3300/0 conn(0x7f82180b3cd0 msgr2=0x7f8218161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 --2- 172.21.15.70:0/1253065657 >> v2:172.21.15.98:3300/0 conn(0x7f82180b3cd0 0x7f8218161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82180aef10 con 0x7f821806ed40 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f820c001680 con 0x7f821806ed40 2024-04-22T01:17:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 >> v1:172.21.15.70:6789/0 conn(0x7f821806ed40 legacy=0x7f82180b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8200004810 0x7f8200004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f82000051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f821806e8b0 con 0x7f8200004810 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82180aec30 con 0x7f82000039c0 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f820000b9d0 con 0x7f821815bb40 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8200004810 0x7f8200004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821cffc640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821d7fd640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f82000051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 -- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 msgr2=0x7f82000051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f82000051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 -- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000039c0 msgr2=0x7f8200003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:53.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f8217fff640 1 -- 172.21.15.70:0/1253065657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8200003530 con 0x7f8200004810 2024-04-22T01:17:53.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821cffc640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:53.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.685+0000 7f821d7fd640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f821815bb40 0x7f82000051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:53.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8200004810 0x7f8200004c90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f820c001060 tx=0x7f820c01f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f36a1cb1acf05c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c007d80 con 0x7f8200004810 2024-04-22T01:17:53.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f820c005b00 con 0x7f8200004810 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f82177fe640 1 -- 172.21.15.70:0/1253065657 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c007070 con 0x7f8200004810 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1253065657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8200004810 msgr2=0x7f8200004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 --2- 172.21.15.70:0/1253065657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8200004810 0x7f8200004c90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f820c001060 tx=0x7f820c01f410 comp rx=0 tx=0).stop 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821d7fd640 1 -- 172.21.15.70:0/1253065657 reap_dead start 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1253065657 shutdown_connections 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1253065657 >> 172.21.15.70:0/1253065657 conn(0x7f821806b1f0 msgr2=0x7f821806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1253065657 shutdown_connections 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1253065657 wait complete. 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 Processor -- start 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- start start 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8200004810 0x7f821815c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 0x7f8218159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8218152690 0x7f821815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82180ae850 con 0x7f821815fc60 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f821806e540 con 0x7f8200004810 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f82180ae9d0 con 0x7f8218152690 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 0x7f8218159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 0x7f8218159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42124/0 (socket says 172.21.15.70:42124) 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 -- 172.21.15.70:0/1574987436 learned_addr learned my addr 172.21.15.70:0/1574987436 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821d7fd640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8218152690 0x7f821815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821cffc640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8200004810 0x7f821815c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 -- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8218152690 msgr2=0x7f821815e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8218152690 0x7f821815e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 -- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8200004810 msgr2=0x7f821815c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8200004810 0x7f821815c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 -- 172.21.15.70:0/1574987436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821815d570 con 0x7f821815fc60 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821d7fd640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8218152690 0x7f821815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f8217fff640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 0x7f8218159cc0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f820c001060 tx=0x7f8200005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c005cb0 con 0x7f821815fc60 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f820c005e50 con 0x7f821815fc60 2024-04-22T01:17:53.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821815d850 con 0x7f821815fc60 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c004a10 con 0x7f821815fc60 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.689+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821815de10 con 0x7f821815fc60 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.693+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8218068210 con 0x7f821815fc60 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.693+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f820c0ba030 con 0x7f821815fc60 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.693+0000 7f82157fa640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82040678c0 0x7f8204069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.693+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f820c122930 con 0x7f821815fc60 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.693+0000 7f821cffc640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82040678c0 0x7f8204069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.693+0000 7f821cffc640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82040678c0 0x7f8204069d80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8208009910 tx=0x7f8208008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:53.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.697+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f820c019120 con 0x7f821815fc60 2024-04-22T01:17:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.857+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82180b0cb0 con 0x7f821815fc60 2024-04-22T01:17:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f82157fa640 1 -- 172.21.15.70:0/1574987436 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7f820c0f32e0 con 0x7f821815fc60 2024-04-22T01:17:53.862 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:17:53.862 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:17:53.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82040678c0 msgr2=0x7f8204069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82040678c0 0x7f8204069d80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8208009910 tx=0x7f8208008040 comp rx=0 tx=0).stop 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 msgr2=0x7f8218159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 0x7f8218159cc0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f820c001060 tx=0x7f8200005840 comp rx=0 tx=0).stop 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 shutdown_connections 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82040678c0 0x7f8204069d80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8218152690 0x7f821815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f821815fc60 0x7f8218159cc0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 --2- 172.21.15.70:0/1574987436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8200004810 0x7f821815c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 >> 172.21.15.70:0/1574987436 conn(0x7f821806b1f0 msgr2=0x7f82180b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 shutdown_connections 2024-04-22T01:17:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:53.861+0000 7f821f260640 1 -- 172.21.15.70:0/1574987436 wait complete. 2024-04-22T01:17:53.882 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:39.433363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:40.173738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:17:53.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:17:54.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.005+0000 7f9ebe4c8640 1 Processor -- start 2024-04-22T01:17:54.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.005+0000 7f9ebe4c8640 1 -- start start 2024-04-22T01:17:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9ebe4c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9eb80b98a0 0x7f9eb80d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9ebe4c8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9eb80cea10 0x7f9eb80cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9ebe4c8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eb80b9ce0 con 0x7f9eb815de20 2024-04-22T01:17:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9ebe4c8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9eb80b75e0 con 0x7f9eb80b98a0 2024-04-22T01:17:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9ebe4c8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9eb80b78c0 con 0x7f9eb80cea10 2024-04-22T01:17:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9eb80b98a0 0x7f9eb80d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9eb80b98a0 0x7f9eb80d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42128/0 (socket says 172.21.15.70:42128) 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 -- 172.21.15.70:0/1278721710 learned_addr learned my addr 172.21.15.70:0/1278721710 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9ebca65640 1 --2- 172.21.15.70:0/1278721710 >> v2:172.21.15.98:3300/0 conn(0x7f9eb80cea10 0x7f9eb80cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 -- 172.21.15.70:0/1278721710 >> v2:172.21.15.98:3300/0 conn(0x7f9eb80cea10 msgr2=0x7f9eb80cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 --2- 172.21.15.70:0/1278721710 >> v2:172.21.15.98:3300/0 conn(0x7f9eb80cea10 0x7f9eb80cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 -- 172.21.15.70:0/1278721710 >> v1:172.21.15.98:6789/0 conn(0x7f9eb815de20 legacy=0x7f9eb80b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 -- 172.21.15.70:0/1278721710 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb80b7ba0 con 0x7f9eb80b98a0 2024-04-22T01:17:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eaf7fe640 1 --2- 172.21.15.70:0/1278721710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80d1d80 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80b6d00 tx=0x7f9ea001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7650a1c376e8d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eb77fe640 1 -- 172.21.15.70:0/1278721710 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00aa610 con 0x7f9eb80b98a0 2024-04-22T01:17:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eb77fe640 1 -- 172.21.15.70:0/1278721710 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea00af070 con 0x7f9eb80b98a0 2024-04-22T01:17:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.009+0000 7f9eb77fe640 1 -- 172.21.15.70:0/1278721710 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00aa9d0 con 0x7f9eb80b98a0 2024-04-22T01:17:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/1278721710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb80b98a0 msgr2=0x7f9eb80d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/1278721710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80d1d80 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80b6d00 tx=0x7f9ea001c4b0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/1278721710 shutdown_connections 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/1278721710 >> v2:172.21.15.98:3300/0 conn(0x7f9eb80cea10 0x7f9eb80cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/1278721710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80d1d80 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/1278721710 >> 172.21.15.70:0/1278721710 conn(0x7f9eb8072d20 msgr2=0x7f9eb8073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/1278721710 shutdown_connections 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/1278721710 wait complete. 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 Processor -- start 2024-04-22T01:17:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- start start 2024-04-22T01:17:54.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb80cea10 0x7f9eb8152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 0x7f9eb80c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb80d22c0 con 0x7f9eb815de20 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb815e200 con 0x7f9eb80b98a0 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb80b8e90 con 0x7f9eb80cea10 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 0x7f9eb80c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 0x7f9eb80c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42144/0 (socket says 172.21.15.70:42144) 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 -- 172.21.15.70:0/844168501 learned_addr learned my addr 172.21.15.70:0/844168501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9eaf7fe640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb80cea10 0x7f9eb8152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9eb7fff640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 -- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb80cea10 msgr2=0x7f9eb8152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb80cea10 0x7f9eb8152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 -- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb80b98a0 msgr2=0x7f9eb80cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 -- 172.21.15.70:0/844168501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb815c2a0 con 0x7f9eb815de20 2024-04-22T01:17:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9eb7fff640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:17:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebca65640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 0x7f9eb80c9a90 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000efc0 tx=0x7f9eb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0018020 con 0x7f9eb815de20 2024-04-22T01:17:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eb0014480 con 0x7f9eb815de20 2024-04-22T01:17:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb815cff0 con 0x7f9eb815de20 2024-04-22T01:17:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0014870 con 0x7f9eb815de20 2024-04-22T01:17:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.013+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb815c510 con 0x7f9eb815de20 2024-04-22T01:17:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.017+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb8068210 con 0x7f9eb815de20 2024-04-22T01:17:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.017+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9eb001b020 con 0x7f9eb815de20 2024-04-22T01:17:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.017+0000 7f9eb57fa640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9e980678c0 0x7f9e98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.017+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f9eb00868d0 con 0x7f9eb815de20 2024-04-22T01:17:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.017+0000 7f9eb7fff640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9e980678c0 0x7f9e98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.017+0000 7f9eb7fff640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9e980678c0 0x7f9e98069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8068e70 tx=0x7f9ea8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.021+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9eb0022070 con 0x7f9eb815de20 2024-04-22T01:17:54.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.181+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9eb80b69c0 con 0x7f9eb815de20 2024-04-22T01:17:54.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.181+0000 7f9eb57fa640 1 -- 172.21.15.70:0/844168501 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f9eb815cff0 con 0x7f9eb815de20 2024-04-22T01:17:54.185 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9e980678c0 msgr2=0x7f9e98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9e980678c0 0x7f9e98069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8068e70 tx=0x7f9ea8009040 comp rx=0 tx=0).stop 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 msgr2=0x7f9eb80c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 0x7f9eb80c9a90 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000efc0 tx=0x7f9eb000c490 comp rx=0 tx=0).stop 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 shutdown_connections 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9e980678c0 0x7f9e98069d80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb815de20 0x7f9eb80c9a90 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb80cea10 0x7f9eb8152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 --2- 172.21.15.70:0/844168501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb80b98a0 0x7f9eb80cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 >> 172.21.15.70:0/844168501 conn(0x7f9eb8072d20 msgr2=0x7f9eb80b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 shutdown_connections 2024-04-22T01:17:54.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.185+0000 7f9ebe4c8640 1 -- 172.21.15.70:0/844168501 wait complete. 2024-04-22T01:17:54.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:17:54.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.313+0000 7fc6edd4a640 1 Processor -- start 2024-04-22T01:17:54.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.313+0000 7fc6edd4a640 1 -- start start 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80b4450 0x7fc6e8071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc6e80720b0 0x7fc6e80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e815c290 con 0x7fc6e80cd050 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc6e8072510 con 0x7fc6e80720b0 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e80b3a50 con 0x7fc6e80b4450 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc6e80720b0 0x7fc6e80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc6e80720b0 0x7fc6e80ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42146/0 (socket says 172.21.15.70:42146) 2024-04-22T01:17:54.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 -- 172.21.15.70:0/3970044245 learned_addr learned my addr 172.21.15.70:0/3970044245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6e7fff640 1 --1- 172.21.15.70:0/3970044245 >> v1:172.21.15.70:6789/0 conn(0x7fc6e80cd050 0x7fc6e815f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38834/0 (socket says 172.21.15.70:38834) 2024-04-22T01:17:54.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/3970044245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80b4450 0x7fc6e8071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 -- 172.21.15.70:0/3970044245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80b4450 msgr2=0x7fc6e8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 --2- 172.21.15.70:0/3970044245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80b4450 0x7fc6e8071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 -- 172.21.15.70:0/3970044245 >> v1:172.21.15.70:6789/0 conn(0x7fc6e80cd050 legacy=0x7fc6e815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 -- 172.21.15.70:0/3970044245 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e815c570 con 0x7fc6e80720b0 2024-04-22T01:17:54.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6deffd640 1 --2- 172.21.15.70:0/3970044245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80720b0 0x7fc6e80ccb10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009870 tx=0x7fc6d001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f291a04ca33df20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6e6ffd640 1 -- 172.21.15.70:0/3970044245 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00ab650 con 0x7fc6e80720b0 2024-04-22T01:17:54.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6e6ffd640 1 -- 172.21.15.70:0/3970044245 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d00b0070 con 0x7fc6e80720b0 2024-04-22T01:17:54.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6e6ffd640 1 -- 172.21.15.70:0/3970044245 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00aba10 con 0x7fc6e80720b0 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- 172.21.15.70:0/3970044245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80720b0 msgr2=0x7fc6e80ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/3970044245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80720b0 0x7fc6e80ccb10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009870 tx=0x7fc6d001c530 comp rx=0 tx=0).stop 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- 172.21.15.70:0/3970044245 shutdown_connections 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/3970044245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80720b0 0x7fc6e80ccb10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/3970044245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80b4450 0x7fc6e8071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- 172.21.15.70:0/3970044245 >> 172.21.15.70:0/3970044245 conn(0x7fc6e80b1d40 msgr2=0x7fc6e80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- 172.21.15.70:0/3970044245 shutdown_connections 2024-04-22T01:17:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.317+0000 7fc6edd4a640 1 -- 172.21.15.70:0/3970044245 wait complete. 2024-04-22T01:17:54.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 Processor -- start 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- start start 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6e80720b0 0x7fc6e814d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 0x7fc6e8166150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80cd050 0x7fc6e8151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80cd5e0 con 0x7fc6e80b4450 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80cd460 con 0x7fc6e80720b0 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e80b4830 con 0x7fc6e80cd050 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 0x7fc6e8166150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 0x7fc6e8166150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42158/0 (socket says 172.21.15.70:42158) 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 -- 172.21.15.70:0/774279079 learned_addr learned my addr 172.21.15.70:0/774279079 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6e80720b0 0x7fc6e814d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6e7fff640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80cd050 0x7fc6e8151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 -- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80cd050 msgr2=0x7fc6e8151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80cd050 0x7fc6e8151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 -- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6e80720b0 msgr2=0x7fc6e814d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6e80720b0 0x7fc6e814d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 -- 172.21.15.70:0/774279079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e8156a90 con 0x7fc6e80b4450 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6deffd640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 0x7fc6e8166150 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00ad040 tx=0x7fc6d001ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00be5f0 con 0x7fc6e80b4450 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d00be7b0 con 0x7fc6e80b4450 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e8156d70 con 0x7fc6e80b4450 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00beba0 con 0x7fc6e80b4450 2024-04-22T01:17:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e814d6d0 con 0x7fc6e80b4450 2024-04-22T01:17:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e8068210 con 0x7fc6e80b4450 2024-04-22T01:17:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.321+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6d00b43f0 con 0x7fc6e80b4450 2024-04-22T01:17:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.325+0000 7fc6e4ff9640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6c80678c0 0x7fc6c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.325+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6c80678c0 0x7fc6c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.325+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fc6d0120830 con 0x7fc6e80b4450 2024-04-22T01:17:54.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.325+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6c80678c0 0x7fc6c8069d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc6e815b7b0 tx=0x7fc6d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.329+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc6d0016150 con 0x7fc6e80b4450 2024-04-22T01:17:54.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6e81524d0 con 0x7fc6e80b4450 2024-04-22T01:17:54.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/774279079 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7fc6d00f0b10 con 0x7fc6e80b4450 2024-04-22T01:17:54.490 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6c80678c0 msgr2=0x7fc6c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6c80678c0 0x7fc6c8069d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc6e815b7b0 tx=0x7fc6d8008040 comp rx=0 tx=0).stop 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 msgr2=0x7fc6e8166150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 0x7fc6e8166150 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00ad040 tx=0x7fc6d001ad70 comp rx=0 tx=0).stop 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 shutdown_connections 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6c80678c0 0x7fc6c8069d80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6e80cd050 0x7fc6e8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e80b4450 0x7fc6e8166150 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 --2- 172.21.15.70:0/774279079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6e80720b0 0x7fc6e814d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 >> 172.21.15.70:0/774279079 conn(0x7fc6e80b1d40 msgr2=0x7fc6e81558a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 shutdown_connections 2024-04-22T01:17:54.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.489+0000 7fc6edd4a640 1 -- 172.21.15.70:0/774279079 wait complete. 2024-04-22T01:17:54.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:17:54.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 Processor -- start 2024-04-22T01:17:54.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 -- start start 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f031015bb60 0x7f03100d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f031006bd30 0x7f03100cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03100cffb0 0x7f03100ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f03100d0420 con 0x7f031006bd30 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f03100cd280 con 0x7f031015bb60 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0318739640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f031015bf40 con 0x7f03100cffb0 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0315cd4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f031006bd30 0x7f03100cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0315cd4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f031006bd30 0x7f03100cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42162/0 (socket says 172.21.15.70:42162) 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.625+0000 7f0315cd4640 1 -- 172.21.15.70:0/4288542973 learned_addr learned my addr 172.21.15.70:0/4288542973 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0315cd4640 1 -- 172.21.15.70:0/4288542973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03100cffb0 msgr2=0x7f03100ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0315cd4640 1 --2- 172.21.15.70:0/4288542973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03100cffb0 0x7f03100ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f03164d5640 1 --2- 172.21.15.70:0/4288542973 >> v2:172.21.15.98:3300/0 conn(0x7f031015bb60 0x7f03100d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0315cd4640 1 -- 172.21.15.70:0/4288542973 >> v2:172.21.15.98:3300/0 conn(0x7f031015bb60 msgr2=0x7f03100d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0315cd4640 1 --2- 172.21.15.70:0/4288542973 >> v2:172.21.15.98:3300/0 conn(0x7f031015bb60 0x7f03100d14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0315cd4640 1 -- 172.21.15.70:0/4288542973 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03100cc3d0 con 0x7f031006bd30 2024-04-22T01:17:54.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0315cd4640 1 --2- 172.21.15.70:0/4288542973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031006bd30 0x7f03100cfa70 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f031006e770 tx=0x7f030001c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4d013fc1e6db9a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f03154d3640 1 -- 172.21.15.70:0/4288542973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03000b0070 con 0x7f031006bd30 2024-04-22T01:17:54.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f03154d3640 1 -- 172.21.15.70:0/4288542973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03000ab600 con 0x7f031006bd30 2024-04-22T01:17:54.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f03154d3640 1 -- 172.21.15.70:0/4288542973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03000ab9d0 con 0x7f031006bd30 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 -- 172.21.15.70:0/4288542973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031006bd30 msgr2=0x7f03100cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 --2- 172.21.15.70:0/4288542973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031006bd30 0x7f03100cfa70 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f031006e770 tx=0x7f030001c230 comp rx=0 tx=0).stop 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 -- 172.21.15.70:0/4288542973 shutdown_connections 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 --2- 172.21.15.70:0/4288542973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03100cffb0 0x7f03100ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 --2- 172.21.15.70:0/4288542973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031006bd30 0x7f03100cfa70 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 --2- 172.21.15.70:0/4288542973 >> v2:172.21.15.98:3300/0 conn(0x7f031015bb60 0x7f03100d14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 -- 172.21.15.70:0/4288542973 >> 172.21.15.70:0/4288542973 conn(0x7f03100733e0 msgr2=0x7f0310099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 -- 172.21.15.70:0/4288542973 shutdown_connections 2024-04-22T01:17:54.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.629+0000 7f0318739640 1 -- 172.21.15.70:0/4288542973 wait complete. 2024-04-22T01:17:54.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 Processor -- start 2024-04-22T01:17:54.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- start start 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031006bd30 0x7f03100c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 0x7f03100c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031015bb60 0x7f03100b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f03100d19e0 con 0x7f03100cffb0 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03100ccf10 con 0x7f031006bd30 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f03100ada50 con 0x7f031015bb60 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 0x7f03100c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 0x7f03100c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42168/0 (socket says 172.21.15.70:42168) 2024-04-22T01:17:54.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 -- 172.21.15.70:0/2981010968 learned_addr learned my addr 172.21.15.70:0/2981010968 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 -- 172.21.15.70:0/2981010968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031015bb60 msgr2=0x7f03100b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031015bb60 0x7f03100b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 -- 172.21.15.70:0/2981010968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031006bd30 msgr2=0x7f03100c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031006bd30 0x7f03100c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 -- 172.21.15.70:0/2981010968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03100c8290 con 0x7f03100cffb0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0315cd4640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 0x7f03100c7d50 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f03000ae040 tx=0x7f0300018fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03000b0040 con 0x7f03100cffb0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03000b9070 con 0x7f03100cffb0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03000b48a0 con 0x7f03100cffb0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03100b3320 con 0x7f03100cffb0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03100b0ad0 con 0x7f03100cffb0 2024-04-22T01:17:54.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0310068920 con 0x7f03100cffb0 2024-04-22T01:17:54.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f03000ab4c0 con 0x7f03100cffb0 2024-04-22T01:17:54.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0306ffd640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f02f40678f0 0x7f02f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f03164d5640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f02f40678f0 0x7f02f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.633+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f030011fc20 con 0x7f03100cffb0 2024-04-22T01:17:54.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.637+0000 7f03164d5640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f02f40678f0 0x7f02f4069db0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f030c00a8b0 tx=0x7f030c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.637+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0300016110 con 0x7f03100cffb0 2024-04-22T01:17:54.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f03100af520 con 0x7f03100cffb0 2024-04-22T01:17:54.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0306ffd640 1 -- 172.21.15.70:0/2981010968 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f03000f05d0 con 0x7f03100cffb0 2024-04-22T01:17:54.786 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:17:54.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f02f40678f0 msgr2=0x7f02f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f02f40678f0 0x7f02f4069db0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f030c00a8b0 tx=0x7f030c008040 comp rx=0 tx=0).stop 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 msgr2=0x7f03100c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 0x7f03100c7d50 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f03000ae040 tx=0x7f0300018fd0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 shutdown_connections 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f02f40678f0 0x7f02f4069db0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031015bb60 0x7f03100b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f03100cffb0 0x7f03100c7d50 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 --2- 172.21.15.70:0/2981010968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f031006bd30 0x7f03100c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 >> 172.21.15.70:0/2981010968 conn(0x7f03100733e0 msgr2=0x7f031015ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 shutdown_connections 2024-04-22T01:17:54.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.785+0000 7f0318739640 1 -- 172.21.15.70:0/2981010968 wait complete. 2024-04-22T01:17:54.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:17:54.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 Processor -- start 2024-04-22T01:17:54.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 -- start start 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 0x7f3fb40ae990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3fb40b29d0 0x7f3fb406dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fb40adb70 con 0x7f3fb415e000 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3fb406db50 con 0x7f3fb40b29d0 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fbb96f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3fb40addf0 con 0x7f3fb40b3db0 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3fb415e000 0x7f3fb40b36a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38846/0 (socket says 172.21.15.70:38846) 2024-04-22T01:17:54.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 -- 172.21.15.70:0/908235229 learned_addr learned my addr 172.21.15.70:0/908235229 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:17:54.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb9f0c640 1 --2- 172.21.15.70:0/908235229 >> v2:172.21.15.98:3300/0 conn(0x7f3fb40b29d0 0x7f3fb406dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb8f0a640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 0x7f3fb40ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3222173750 0 0) 0x7f3fac001120 con 0x7f3fb415e000 2024-04-22T01:17:54.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fa0003530 con 0x7f3fb415e000 2024-04-22T01:17:54.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 65543426 0 0) 0x7f3fa0003530 con 0x7f3fb415e000 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 msgr2=0x7f3fb40ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 0x7f3fb40ae990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 >> v2:172.21.15.98:3300/0 conn(0x7f3fb40b29d0 msgr2=0x7f3fb406dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 --2- 172.21.15.70:0/908235229 >> v2:172.21.15.98:3300/0 conn(0x7f3fb40b29d0 0x7f3fb406dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb40ae0d0 con 0x7f3fb415e000 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3fac001850 con 0x7f3fb415e000 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 >> v1:172.21.15.70:6789/0 conn(0x7f3fb415e000 legacy=0x7f3fb40b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0004810 0x7f3fa0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 0x7f3fa0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb406db50 con 0x7f3fa0003a30 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb40addf0 con 0x7f3fa0004810 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa000bb80 con 0x7f3fb40b3db0 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb9f0c640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 0x7f3fa0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb8f0a640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0004810 0x7f3fa0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 -- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 msgr2=0x7f3fa0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb40b3db0 0x7f3fa0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 -- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0004810 msgr2=0x7f3fa0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0004810 0x7f3fa0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.909+0000 7f3fb970b640 1 -- 172.21.15.70:0/908235229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa0003530 con 0x7f3fa0003a30 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fb970b640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00c4c0 tx=0x7f3fac01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979f68e8e6c5f56c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00c0e0 con 0x7f3fa0003a30 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fac0a9020 con 0x7f3fa0003a30 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fabfff640 1 -- 172.21.15.70:0/908235229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0a5bc0 con 0x7f3fa0003a30 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 -- 172.21.15.70:0/908235229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa0003a30 msgr2=0x7f3fa0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/908235229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00c4c0 tx=0x7f3fac01ccc0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fb9f0c640 1 -- 172.21.15.70:0/908235229 reap_dead start 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 -- 172.21.15.70:0/908235229 shutdown_connections 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 -- 172.21.15.70:0/908235229 >> 172.21.15.70:0/908235229 conn(0x7f3fb40cf9e0 msgr2=0x7f3fb40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 -- 172.21.15.70:0/908235229 shutdown_connections 2024-04-22T01:17:54.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 -- 172.21.15.70:0/908235229 wait complete. 2024-04-22T01:17:54.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 Processor -- start 2024-04-22T01:17:54.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.913+0000 7f3fbb96f640 1 -- start start 2024-04-22T01:17:54.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0003a30 0x7f3fb415d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 0x7f3fb415b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb415b550 0x7f3fb415f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb406f0c0 con 0x7f3fb414dc00 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb406ef40 con 0x7f3fa0003a30 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3fb406f240 con 0x7f3fb415b550 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 0x7f3fb415b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 0x7f3fb415b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42190/0 (socket says 172.21.15.70:42190) 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 -- 172.21.15.70:0/2751724486 learned_addr learned my addr 172.21.15.70:0/2751724486 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb970b640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0003a30 0x7f3fb415d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 -- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb415b550 msgr2=0x7f3fb415f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb415b550 0x7f3fb415f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 -- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0003a30 msgr2=0x7f3fb415d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0003a30 0x7f3fb415d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 -- 172.21.15.70:0/2751724486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb415d7f0 con 0x7f3fb414dc00 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb8f0a640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 0x7f3fb415b010 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400f530 tx=0x7f3fa400f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4013810 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fa401a070 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb41640c0 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4013bf0 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb4164680 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb404f510 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3fa4012070 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fa9ffb640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f980678c0 0x7f3f98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f3fa4085cf0 con 0x7f3fb414dc00 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb970b640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f980678c0 0x7f3f98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.917+0000 7f3fb970b640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f980678c0 0x7f3f98069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00c490 tx=0x7f3fac001c20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:17:54.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:54.921+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3fa408a070 con 0x7f3fb414dc00 2024-04-22T01:17:55.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.081+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3fb40b4cb0 con 0x7f3fb414dc00 2024-04-22T01:17:55.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.081+0000 7f3fa9ffb640 1 -- 172.21.15.70:0/2751724486 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) ==== 80+0+0 (secure 0 0 0) 0x7f3fa401c310 con 0x7f3fb414dc00 2024-04-22T01:17:55.083 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:17:55.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f980678c0 msgr2=0x7f3f98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:55.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f980678c0 0x7f3f98069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00c490 tx=0x7f3fac001c20 comp rx=0 tx=0).stop 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 msgr2=0x7f3fb415b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 0x7f3fb415b010 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400f530 tx=0x7f3fa400f9f0 comp rx=0 tx=0).stop 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 shutdown_connections 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f980678c0 0x7f3f98069d80 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fb415b550 0x7f3fb415f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb414dc00 0x7f3fb415b010 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 --2- 172.21.15.70:0/2751724486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa0003a30 0x7f3fb415d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 >> 172.21.15.70:0/2751724486 conn(0x7f3fb40cf9e0 msgr2=0x7f3fb415a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 shutdown_connections 2024-04-22T01:17:55.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:17:55.085+0000 7f3fbb96f640 1 -- 172.21.15.70:0/2751724486 wait complete. 2024-04-22T01:17:55.100 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:17:55.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:17:55.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:17:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34380]: fuse finished with error 0 and tester_r 0 2024-04-22T01:17:55.115 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:01.217 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:18:01.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.217 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:18:01.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.225 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-22T01:18:01.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.243 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 134217728 ./subdir) 2024-04-22T01:18:01.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:18:01.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd9fc5640 1 Processor -- start 2024-04-22T01:18:01.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd9fc5640 1 -- start start 2024-04-22T01:18:01.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd9fc5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fd40b4720 0x7f2fd415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd9fc5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd406dbd0 con 0x7f2fd415aa30 2024-04-22T01:18:01.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd9fc5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd406de70 con 0x7f2fd40aee40 2024-04-22T01:18:01.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd9fc5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd415c090 con 0x7f2fd40b4720 2024-04-22T01:18:01.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd37fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2fd415aa30 0x7f2fd40ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41934/0 (socket says 172.21.15.70:41934) 2024-04-22T01:18:01.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd37fe640 1 -- 172.21.15.70:0/520611524 learned_addr learned my addr 172.21.15.70:0/520611524 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd3fff640 1 --2- 172.21.15.70:0/520611524 >> v2:172.21.15.98:3300/0 conn(0x7f2fd40b4720 0x7f2fd415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391050947 0 0) 0x7f2fd406dbd0 con 0x7f2fd415aa30 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fbc003530 con 0x7f2fd415aa30 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3280466104 0 0) 0x7f2fbc003530 con 0x7f2fd415aa30 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 >> v2:172.21.15.98:3300/0 conn(0x7f2fd40b4720 msgr2=0x7f2fd415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 --2- 172.21.15.70:0/520611524 >> v2:172.21.15.98:3300/0 conn(0x7f2fd40b4720 0x7f2fd415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 >> v1:172.21.15.98:6789/0 conn(0x7f2fd40aee40 legacy=0x7f2fd40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd406e150 con 0x7f2fd415aa30 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f2fc4001680 con 0x7f2fd415aa30 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 >> v1:172.21.15.70:6789/0 conn(0x7f2fd415aa30 legacy=0x7f2fd40ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc003a10 0x7f2fbc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fbc004810 0x7f2fbc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fbc0051d0 0x7f2fbc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c0a0 con 0x7f2fbc004810 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c250 con 0x7f2fbc0051d0 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c450 con 0x7f2fbc003a10 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd2ffd640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fbc004810 0x7f2fbc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd3fff640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fbc0051d0 0x7f2fbc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd37fe640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc003a10 0x7f2fbc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd2ffd640 1 -- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc003a10 msgr2=0x7f2fbc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd2ffd640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc003a10 0x7f2fbc003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd2ffd640 1 -- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fbc0051d0 msgr2=0x7f2fbc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:01.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd2ffd640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fbc0051d0 0x7f2fbc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd2ffd640 1 -- 172.21.15.70:0/520611524 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc003530 con 0x7f2fbc004810 2024-04-22T01:18:01.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd3fff640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fbc0051d0 0x7f2fbc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:01.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.469+0000 7f2fd3fff640 1 -- 172.21.15.70:0/520611524 reap_dead start 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd2ffd640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fbc004810 0x7f2fbc004c90 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8001e40 tx=0x7f2fc801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d140a9249357c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8001a90 con 0x7f2fbc004810 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc8003b80 con 0x7f2fbc004810 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd27fc640 1 -- 172.21.15.70:0/520611524 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80050d0 con 0x7f2fbc004810 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/520611524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fbc004810 msgr2=0x7f2fbc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fbc004810 0x7f2fbc004c90 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8001e40 tx=0x7f2fc801c700 comp rx=0 tx=0).stop 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/520611524 shutdown_connections 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/520611524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fbc004810 0x7f2fbc004c90 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/520611524 >> 172.21.15.70:0/520611524 conn(0x7f2fd406b1f0 msgr2=0x7f2fd406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/520611524 shutdown_connections 2024-04-22T01:18:01.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/520611524 wait complete. 2024-04-22T01:18:01.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 Processor -- start 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- start start 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc004810 0x7f2fd41f11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd4153e80 0x7f2fd4152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 0x7f2fd40b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd415ae10 con 0x7f2fd40b1870 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd40b4d00 con 0x7f2fd4153e80 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd9fc5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd415af90 con 0x7f2fbc004810 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 0x7f2fd40b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 0x7f2fd40b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60942/0 (socket says 172.21.15.70:60942) 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.473+0000 7f2fd3fff640 1 -- 172.21.15.70:0/2316591247 learned_addr learned my addr 172.21.15.70:0/2316591247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd2ffd640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd4153e80 0x7f2fd4152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd37fe640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc004810 0x7f2fd41f11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd3fff640 1 -- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc004810 msgr2=0x7f2fd41f11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd3fff640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc004810 0x7f2fd41f11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd3fff640 1 -- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd4153e80 msgr2=0x7f2fd4152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd3fff640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd4153e80 0x7f2fd4152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd3fff640 1 -- 172.21.15.70:0/2316591247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd4153120 con 0x7f2fd40b1870 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd37fe640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc004810 0x7f2fd41f11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd3fff640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 0x7f2fd40b1cf0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc010790 tx=0x7f2fcc010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00f040 con 0x7f2fd40b1870 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fcc01c070 con 0x7f2fd40b1870 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd4153400 con 0x7f2fd40b1870 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00ea80 con 0x7f2fd40b1870 2024-04-22T01:18:01.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd41f16e0 con 0x7f2fd40b1870 2024-04-22T01:18:01.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd4153920 con 0x7f2fd40b1870 2024-04-22T01:18:01.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2fcc00ec20 con 0x7f2fd40b1870 2024-04-22T01:18:01.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fbbfff640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fc00678c0 0x7f2fc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:01.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f2fcc085de0 con 0x7f2fd40b1870 2024-04-22T01:18:01.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd37fe640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fc00678c0 0x7f2fc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:01.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.477+0000 7f2fd37fe640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fc00678c0 0x7f2fc0069d80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4004880 tx=0x7f2fc40051b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:01.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.481+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2fd4153920 con 0x7f2fd40b1870 2024-04-22T01:18:01.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2fd4068210 con 0x7f2fd40b1870 2024-04-22T01:18:01.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fbbfff640 1 -- 172.21.15.70:0/2316591247 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7f2fcc056790 con 0x7f2fd40b1870 2024-04-22T01:18:01.682 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:18:01.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fc00678c0 msgr2=0x7f2fc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fc00678c0 0x7f2fc0069d80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4004880 tx=0x7f2fc40051b0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 msgr2=0x7f2fd40b1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 0x7f2fd40b1cf0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc010790 tx=0x7f2fcc010c50 comp rx=0 tx=0).stop 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 shutdown_connections 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2fc00678c0 0x7f2fc0069d80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fd40b1870 0x7f2fd40b1cf0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd4153e80 0x7f2fd4152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 --2- 172.21.15.70:0/2316591247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fbc004810 0x7f2fd41f11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 >> 172.21.15.70:0/2316591247 conn(0x7f2fd406b1f0 msgr2=0x7f2fd415cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 shutdown_connections 2024-04-22T01:18:01.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:01.680+0000 7f2fd9fc5640 1 -- 172.21.15.70:0/2316591247 wait complete. 2024-04-22T01:18:01.700 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /subdir 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:18:01.701 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:18:01.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.701 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:18:01.708 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:18:01.708 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:18:01.708 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:18:01.708 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:01.709 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42344sec preferred_lft 42344sec 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.710 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:01.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.711 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:18:01.761 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:18:01.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.763 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:18:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:18:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:18:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:18:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:18:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:18:01.819 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:18:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:18:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:18:01.820 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:18:01.820 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:18:01.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:18:01.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:18:01.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:18:01.821 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:18:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:18:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:18:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:18:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:01.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.947 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:18:01.952 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:18:01.952 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:18:01.953 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:18:01.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:01.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:18:02.004 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:18:02.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:02.005 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:02.050 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:18:02.050 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:02.051 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:02.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:18:02.155 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:02.203 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:18:02.203 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:18:02.204 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:02.206 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:02.206 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:02.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:02.206 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:18:02.250 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:18:02.250 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/subdir) 2024-04-22T01:18:02.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:18:02.309 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:02.358 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:18:02.358 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:18:02.358 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:18:02.358 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:18:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:02.361 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:02.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:02.362 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:18:02.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:18:02.371+0000 7f1a06bda580 -1 init, newargv = 0x5615d7df29a0 newargc=15 2024-04-22T01:18:02.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34471]: starting ceph client 2024-04-22T01:18:02.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34471]: starting fuse 2024-04-22T01:18:03.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:18:03.498 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:18:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:18:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:18:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:18:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:18:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:03.548 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:03.549 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:03.550 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:03.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:03.551 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:18:03.598 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:18:03.598 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:18:03.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:18:03.598 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:03.599 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:18:03.600 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:18:03.720 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.34471.asok 2024-04-22T01:18:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:18:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:03.732 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34471.asok 2024-04-22T01:18:03.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:03.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34471.asok status 2024-04-22T01:18:03.884 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:18:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:18:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:18:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:18:03.884 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "34471", 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/subdir" 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5294, 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5294 2024-04-22T01:18:03.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4083384607 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:03.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4083384607 2024-04-22T01:18:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5294 192.168.0.1:0/4083384607", 2024-04-22T01:18:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/4083384607", 2024-04-22T01:18:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 2, 2024-04-22T01:18:03.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 37, 2024-04-22T01:18:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 78, 2024-04-22T01:18:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:18:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:18:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:18:03.888 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:18:03.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:03.899 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:18:03.899 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:03.907 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:18:03.907 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:03.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:03.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:03.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:03.964 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec df -B1 .) 2024-04-22T01:18:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 1B-blocks Used Available Use% Mounted on 2024-04-22T01:18:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-fuse 134217728 0 134217728 0% /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:04.037 WARNING:tasks.cephfs.mount:['Filesystem 1B-blocks Used Available Use% Mounted on', 'ceph-fuse 134217728 0 134217728 0% /home/ubuntu/cephtest/mnt.1'] 2024-04-22T01:18:04.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:04.038 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 268435456 ./subdir) 2024-04-22T01:18:04.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:04.055 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec df -B1 .) 2024-04-22T01:18:04.089 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 1B-blocks Used Available Use% Mounted on 2024-04-22T01:18:04.089 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-fuse 268435456 0 268435456 0% /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:04.090 WARNING:tasks.cephfs.mount:['Filesystem 1B-blocks Used Available Use% Mounted on', 'ceph-fuse 268435456 0 268435456 0% /home/ubuntu/cephtest/mnt.1'] 2024-04-22T01:18:04.091 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:18:04.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:04.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:04.195 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:18:04.195 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:04.195 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:04.195 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:04.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:04.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 Processor -- start 2024-04-22T01:18:04.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 -- start start 2024-04-22T01:18:04.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05e815dfb0 0x7f05e80b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05e80cea80 0x7f05e80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05e815e390 con 0x7f05e80b5730 2024-04-22T01:18:04.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f05e80cf420 con 0x7f05e815dfb0 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05ed5d6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f05e8161e00 con 0x7f05e80cea80 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05e815dfb0 0x7f05e80b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f05e80b5730 0x7f05e80b5bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41946/0 (socket says 172.21.15.70:41946) 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e77fe640 1 -- 172.21.15.70:0/231614446 learned_addr learned my addr 172.21.15.70:0/231614446 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e67fc640 1 --2- 172.21.15.70:0/231614446 >> v2:172.21.15.98:3300/0 conn(0x7f05e80cea80 0x7f05e80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e6ffd640 1 -- 172.21.15.70:0/231614446 >> v2:172.21.15.98:3300/0 conn(0x7f05e80cea80 msgr2=0x7f05e80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e6ffd640 1 --2- 172.21.15.70:0/231614446 >> v2:172.21.15.98:3300/0 conn(0x7f05e80cea80 0x7f05e80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e6ffd640 1 -- 172.21.15.70:0/231614446 >> v1:172.21.15.70:6789/0 conn(0x7f05e80b5730 legacy=0x7f05e80b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e6ffd640 1 -- 172.21.15.70:0/231614446 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e81620e0 con 0x7f05e815dfb0 2024-04-22T01:18:04.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.372+0000 7f05e67fc640 1 --2- 172.21.15.70:0/231614446 >> v2:172.21.15.98:3300/0 conn(0x7f05e80cea80 0x7f05e80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05e6ffd640 1 --2- 172.21.15.70:0/231614446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e80b95d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f05e80d15a0 tx=0x7f05d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c431dd4a3aaa2f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05e5ffb640 1 -- 172.21.15.70:0/231614446 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d80b1070 con 0x7f05e815dfb0 2024-04-22T01:18:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05e5ffb640 1 -- 172.21.15.70:0/231614446 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d80ac5f0 con 0x7f05e815dfb0 2024-04-22T01:18:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05e5ffb640 1 -- 172.21.15.70:0/231614446 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d80ac9c0 con 0x7f05e815dfb0 2024-04-22T01:18:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 -- 172.21.15.70:0/231614446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 msgr2=0x7f05e80b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/231614446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e80b95d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f05e80d15a0 tx=0x7f05d801c480 comp rx=0 tx=0).stop 2024-04-22T01:18:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 -- 172.21.15.70:0/231614446 shutdown_connections 2024-04-22T01:18:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/231614446 >> v2:172.21.15.98:3300/0 conn(0x7f05e80cea80 0x7f05e80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/231614446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e80b95d0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 -- 172.21.15.70:0/231614446 >> 172.21.15.70:0/231614446 conn(0x7f05e806c9b0 msgr2=0x7f05e806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 -- 172.21.15.70:0/231614446 shutdown_connections 2024-04-22T01:18:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 -- 172.21.15.70:0/231614446 wait complete. 2024-04-22T01:18:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 Processor -- start 2024-04-22T01:18:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.376+0000 7f05ed5d6640 1 -- start start 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e80b5730 0x7f05e80cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e80cea80 0x7f05e80cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e8157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05e816b5e0 con 0x7f05e815dfb0 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05e8161a90 con 0x7f05e80cea80 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05e816b760 con 0x7f05e80b5730 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e8157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e8157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60950/0 (socket says 172.21.15.70:60950) 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 -- 172.21.15.70:0/1253364843 learned_addr learned my addr 172.21.15.70:0/1253364843 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 -- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e80b5730 msgr2=0x7f05e80cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e67fc640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e80cea80 0x7f05e80cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e80b5730 0x7f05e80cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 -- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e80cea80 msgr2=0x7f05e80cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e80cea80 0x7f05e80cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 -- 172.21.15.70:0/1253364843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e8158530 con 0x7f05e815dfb0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e77fe640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e8157950 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f05e000ca80 tx=0x7f05e000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0017070 con 0x7f05e815dfb0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e000ac30 con 0x7f05e815dfb0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e8158870 con 0x7f05e815dfb0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0012600 con 0x7f05e815dfb0 2024-04-22T01:18:04.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e81590b0 con 0x7f05e815dfb0 2024-04-22T01:18:04.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e81592f0 con 0x7f05e815dfb0 2024-04-22T01:18:04.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05e001a070 con 0x7f05e815dfb0 2024-04-22T01:18:04.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05cf7fe640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05d40678c0 0x7f05d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f05e0085990 con 0x7f05e815dfb0 2024-04-22T01:18:04.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e6ffd640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05d40678c0 0x7f05d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.380+0000 7f05e6ffd640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05d40678c0 0x7f05d4069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f05d80ae000 tx=0x7f05d8004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.384+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f05e81592f0 con 0x7f05e815dfb0 2024-04-22T01:18:04.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.548+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05e8068210 con 0x7f05e815dfb0 2024-04-22T01:18:04.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05cf7fe640 1 -- 172.21.15.70:0/1253364843 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7f05e8158870 con 0x7f05e815dfb0 2024-04-22T01:18:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:04.553 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:04.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05d40678c0 msgr2=0x7f05d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05d40678c0 0x7f05d4069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f05d80ae000 tx=0x7f05d8004740 comp rx=0 tx=0).stop 2024-04-22T01:18:04.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 msgr2=0x7f05e8157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e8157950 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f05e000ca80 tx=0x7f05e000cf40 comp rx=0 tx=0).stop 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 shutdown_connections 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05d40678c0 0x7f05d4069d80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e815dfb0 0x7f05e8157950 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e80cea80 0x7f05e80cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 --2- 172.21.15.70:0/1253364843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e80b5730 0x7f05e80cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 >> 172.21.15.70:0/1253364843 conn(0x7f05e806c9b0 msgr2=0x7f05e80b6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 shutdown_connections 2024-04-22T01:18:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.552+0000 7f05ed5d6640 1 -- 172.21.15.70:0/1253364843 wait complete. 2024-04-22T01:18:04.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:39.433363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:40.173738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:04.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:04.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.688+0000 7f9c41503640 1 Processor -- start 2024-04-22T01:18:04.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.688+0000 7f9c41503640 1 -- start start 2024-04-22T01:18:04.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c41503640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9c3c072280 0x7f9c3c0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c41503640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c15c290 con 0x7f9c3c0cd050 2024-04-22T01:18:04.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c41503640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c15c570 con 0x7f9c3c0b4450 2024-04-22T01:18:04.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c41503640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3c15c850 con 0x7f9c3c072280 2024-04-22T01:18:04.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c3a7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9c3c072280 0x7f9c3c0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c3b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9c3c0cd050 0x7f9c3c0b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41950/0 (socket says 172.21.15.70:41950) 2024-04-22T01:18:04.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c3b7fe640 1 -- 172.21.15.70:0/3109431231 learned_addr learned my addr 172.21.15.70:0/3109431231 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:04.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c39ffb640 1 -- 172.21.15.70:0/3109431231 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383414103 0 0) 0x7f9c3c15c290 con 0x7f9c3c0cd050 2024-04-22T01:18:04.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c39ffb640 1 -- 172.21.15.70:0/3109431231 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c24003530 con 0x7f9c3c0cd050 2024-04-22T01:18:04.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c3a7fc640 1 -- 172.21.15.70:0/3109431231 >> v1:172.21.15.98:6789/0 conn(0x7f9c3c0b4450 legacy=0x7f9c3c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c3a7fc640 1 -- 172.21.15.70:0/3109431231 >> v1:172.21.15.70:6789/0 conn(0x7f9c3c0cd050 legacy=0x7f9c3c0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.692+0000 7f9c3a7fc640 1 -- 172.21.15.70:0/3109431231 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c15cb30 con 0x7f9c3c072280 2024-04-22T01:18:04.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c3a7fc640 1 --2- 172.21.15.70:0/3109431231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c0ccb10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9c300097c0 tx=0x7f9c3001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc1a07d031a6812 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:04.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c39ffb640 1 -- 172.21.15.70:0/3109431231 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c300b0070 con 0x7f9c3c072280 2024-04-22T01:18:04.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c39ffb640 1 -- 172.21.15.70:0/3109431231 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c300ab600 con 0x7f9c3c072280 2024-04-22T01:18:04.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c39ffb640 1 -- 172.21.15.70:0/3109431231 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c300ab9d0 con 0x7f9c3c072280 2024-04-22T01:18:04.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- 172.21.15.70:0/3109431231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 msgr2=0x7f9c3c0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 --2- 172.21.15.70:0/3109431231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c0ccb10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9c300097c0 tx=0x7f9c3001c410 comp rx=0 tx=0).stop 2024-04-22T01:18:04.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- 172.21.15.70:0/3109431231 shutdown_connections 2024-04-22T01:18:04.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 --2- 172.21.15.70:0/3109431231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c0ccb10 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- 172.21.15.70:0/3109431231 >> 172.21.15.70:0/3109431231 conn(0x7f9c3c0b1d40 msgr2=0x7f9c3c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:04.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- 172.21.15.70:0/3109431231 shutdown_connections 2024-04-22T01:18:04.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- 172.21.15.70:0/3109431231 wait complete. 2024-04-22T01:18:04.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 Processor -- start 2024-04-22T01:18:04.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- start start 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c1f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c3c0b4450 0x7f9c3c14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c3c0cd050 0x7f9c3c1f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c0cd980 con 0x7f9c3c072280 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c0cd800 con 0x7f9c3c0cd050 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c41503640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c169b20 con 0x7f9c3c0b4450 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.696+0000 7f9c3affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c1f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c1f4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60972/0 (socket says 172.21.15.70:60972) 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 -- 172.21.15.70:0/3456207129 learned_addr learned my addr 172.21.15.70:0/3456207129 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3b7fe640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c3c0cd050 0x7f9c3c1f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3a7fc640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c3c0b4450 0x7f9c3c14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 -- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c3c0b4450 msgr2=0x7f9c3c14e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c3c0b4450 0x7f9c3c14e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 -- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c3c0cd050 msgr2=0x7f9c3c1f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c3c0cd050 0x7f9c3c1f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 -- 172.21.15.70:0/3456207129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c1f5e50 con 0x7f9c3c072280 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3b7fe640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c3c0cd050 0x7f9c3c1f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3affd640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c1f4f00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00c500 tx=0x7f9c2c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c016020 con 0x7f9c3c072280 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2c005220 con 0x7f9c3c072280 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c14aa00 con 0x7f9c3c072280 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c007030 con 0x7f9c3c072280 2024-04-22T01:18:04.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c14e990 con 0x7f9c3c072280 2024-04-22T01:18:04.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c068210 con 0x7f9c3c072280 2024-04-22T01:18:04.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9c2c0071f0 con 0x7f9c3c072280 2024-04-22T01:18:04.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c237fe640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c280678f0 0x7f9c28069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:04.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f9c2c085780 con 0x7f9c3c072280 2024-04-22T01:18:04.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3a7fc640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c280678f0 0x7f9c28069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.700+0000 7f9c3a7fc640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c280678f0 0x7f9c28069db0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0b27c0 tx=0x7f9c30008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.704+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9c2c020020 con 0x7f9c3c072280 2024-04-22T01:18:04.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.856+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9c3c0d01e0 con 0x7f9c3c072280 2024-04-22T01:18:04.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.856+0000 7f9c237fe640 1 -- 172.21.15.70:0/3456207129 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f9c3c1f5e50 con 0x7f9c3c072280 2024-04-22T01:18:04.860 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c280678f0 msgr2=0x7f9c28069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c280678f0 0x7f9c28069db0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0b27c0 tx=0x7f9c30008040 comp rx=0 tx=0).stop 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 msgr2=0x7f9c3c1f4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c1f4f00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00c500 tx=0x7f9c2c00c9c0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 shutdown_connections 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9c280678f0 0x7f9c28069db0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c3c0cd050 0x7f9c3c1f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c3c0b4450 0x7f9c3c14e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 --2- 172.21.15.70:0/3456207129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c3c072280 0x7f9c3c1f4f00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 >> 172.21.15.70:0/3456207129 conn(0x7f9c3c0b1d40 msgr2=0x7f9c3c0cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 shutdown_connections 2024-04-22T01:18:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.860+0000 7f9c41503640 1 -- 172.21.15.70:0/3456207129 wait complete. 2024-04-22T01:18:04.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:18:05.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.996+0000 7f1bfa8c9640 1 Processor -- start 2024-04-22T01:18:05.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:04.996+0000 7f1bfa8c9640 1 -- start start 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bfa8c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 0x7f1bf415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bfa8c9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf415f000 con 0x7f1bf4155370 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bfa8c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf40b3800 con 0x7f1bf40ce920 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bfa8c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf40b3ae0 con 0x7f1bf415e6e0 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf8e66640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1bf4155370 0x7f1bf406d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41962/0 (socket says 172.21.15.70:41962) 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf8e66640 1 -- 172.21.15.70:0/1906577527 learned_addr learned my addr 172.21.15.70:0/1906577527 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 0x7f1bf415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361511892 0 0) 0x7f1bf415f000 con 0x7f1bf4155370 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be0003530 con 0x7f1bf4155370 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2578431080 0 0) 0x7f1be0003530 con 0x7f1bf4155370 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 msgr2=0x7f1bf415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 0x7f1bf415eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 >> v1:172.21.15.98:6789/0 conn(0x7f1bf40ce920 legacy=0x7f1bf4154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40cde40 con 0x7f1bf4155370 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1bec001a00 con 0x7f1bf4155370 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 >> v1:172.21.15.70:6789/0 conn(0x7f1bf4155370 legacy=0x7f1bf406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be00039f0 0x7f1be0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 0x7f1be0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be0004d50 0x7f1be00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf40b3ae0 con 0x7f1be00039f0 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1be000ba80 con 0x7f1be0004d50 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1be000bc80 con 0x7f1bf415e6e0 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be00039f0 0x7f1be0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf8e66640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be0004d50 0x7f1be00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 0x7f1be0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 -- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 msgr2=0x7f1be0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf415e6e0 0x7f1be0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 -- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be0004d50 msgr2=0x7f1be00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be0004d50 0x7f1be00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf3fff640 1 -- 172.21.15.70:0/1906577527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be0003530 con 0x7f1be00039f0 2024-04-22T01:18:05.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.000+0000 7f1bf8e66640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be0004d50 0x7f1be00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:05.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be00039f0 0x7f1be0003e10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009ab0 tx=0x7f1bdc01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bfcea7ba352155 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc0b2070 con 0x7f1be00039f0 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bdc0a55f0 con 0x7f1be00039f0 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf2ffd640 1 -- 172.21.15.70:0/1906577527 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc0096b0 con 0x7f1be00039f0 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/1906577527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be00039f0 msgr2=0x7f1be0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/1906577527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1be00039f0 0x7f1be0003e10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009ab0 tx=0x7f1bdc01ca20 comp rx=0 tx=0).stop 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 -- 172.21.15.70:0/1906577527 reap_dead start 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/1906577527 shutdown_connections 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/1906577527 >> 172.21.15.70:0/1906577527 conn(0x7f1bf40b1d40 msgr2=0x7f1bf40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/1906577527 shutdown_connections 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/1906577527 wait complete. 2024-04-22T01:18:05.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 Processor -- start 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- start start 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be00039f0 0x7f1bf41f5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf4156510 0x7f1bf41571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 0x7f1bf414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf40b3440 con 0x7f1bf4071dd0 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf40b32c0 con 0x7f1be00039f0 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bfa8c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf40b35c0 con 0x7f1bf4156510 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 0x7f1bf414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 0x7f1bf414fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60994/0 (socket says 172.21.15.70:60994) 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 -- 172.21.15.70:0/2244147753 learned_addr learned my addr 172.21.15.70:0/2244147753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:05.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be00039f0 0x7f1bf41f5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf4156510 0x7f1bf41571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 -- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf4156510 msgr2=0x7f1bf41571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf4156510 0x7f1bf41571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 -- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be00039f0 msgr2=0x7f1bf41f5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be00039f0 0x7f1bf41f5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf8e66640 1 -- 172.21.15.70:0/2244147753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf414ee20 con 0x7f1bf4071dd0 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.004+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf4156510 0x7f1bf41571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf8e66640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 0x7f1bf414fbc0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406fdc0 tx=0x7f1bec00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00ec10 con 0x7f1bf4071dd0 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bec018070 con 0x7f1bf4071dd0 2024-04-22T01:18:05.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf414f100 con 0x7f1bf4071dd0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec013420 con 0x7f1bf4071dd0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf414f6c0 con 0x7f1bf4071dd0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf4068210 con 0x7f1bf4071dd0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1bec0135c0 con 0x7f1bf4071dd0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf0ff9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1bd40678c0 0x7f1bd4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f1bec086150 con 0x7f1bf4071dd0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1bd40678c0 0x7f1bd4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.008+0000 7f1bf3fff640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1bd40678c0 0x7f1bd4069d80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0af000 tx=0x7f1bdc0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.012+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1bec056b90 con 0x7f1bf4071dd0 2024-04-22T01:18:05.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.172+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1bf415ab10 con 0x7f1bf4071dd0 2024-04-22T01:18:05.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bf0ff9640 1 -- 172.21.15.70:0/2244147753 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f1bec056490 con 0x7f1bf4071dd0 2024-04-22T01:18:05.177 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1bd40678c0 msgr2=0x7f1bd4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1bd40678c0 0x7f1bd4069d80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0af000 tx=0x7f1bdc0046d0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 msgr2=0x7f1bf414fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 0x7f1bf414fbc0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406fdc0 tx=0x7f1bec00e530 comp rx=0 tx=0).stop 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 shutdown_connections 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1bd40678c0 0x7f1bd4069d80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf4071dd0 0x7f1bf414fbc0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bf4156510 0x7f1bf41571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 --2- 172.21.15.70:0/2244147753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be00039f0 0x7f1bf41f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 >> 172.21.15.70:0/2244147753 conn(0x7f1bf40b1d40 msgr2=0x7f1bf40cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 shutdown_connections 2024-04-22T01:18:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.176+0000 7f1bfa8c9640 1 -- 172.21.15.70:0/2244147753 wait complete. 2024-04-22T01:18:05.196 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:18:05.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.312+0000 7f96fed09640 1 Processor -- start 2024-04-22T01:18:05.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.312+0000 7f96fed09640 1 -- start start 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fed09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 0x7f96f8071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fed09640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f815c290 con 0x7f96f80720b0 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fed09640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f815c570 con 0x7f96f80cd220 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fed09640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f96f8072510 con 0x7f96f80b4450 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f7fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f96f80720b0 0x7f96f80ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41974/0 (socket says 172.21.15.70:41974) 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f7fff640 1 -- 172.21.15.70:0/796474994 learned_addr learned my addr 172.21.15.70:0/796474994 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1855106742 0 0) 0x7f96f815c290 con 0x7f96f80720b0 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96e0003530 con 0x7f96f80720b0 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fcaa5640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 0x7f96f8071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 23358321 0 0) 0x7f96e0003530 con 0x7f96f80720b0 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 msgr2=0x7f96f8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 0x7f96f8071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 >> v1:172.21.15.98:6789/0 conn(0x7f96f80cd220 legacy=0x7f96f80b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f815c850 con 0x7f96f80720b0 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f96ec001680 con 0x7f96f80720b0 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 >> v1:172.21.15.70:6789/0 conn(0x7f96f80720b0 legacy=0x7f96f80ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 0x7f96e00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e0004020 0x7f96e0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e0004d50 0x7f96e00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96e000bb60 con 0x7f96e0004d50 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96e000bd10 con 0x7f96e0004020 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f96e000bf10 con 0x7f96f80b4450 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e0004d50 0x7f96e00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fcaa5640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 0x7f96e00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f7fff640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e0004020 0x7f96e0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 -- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 msgr2=0x7f96e00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f80b4450 0x7f96e00039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 -- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e0004020 msgr2=0x7f96e0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e0004020 0x7f96e0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 -- 172.21.15.70:0/796474994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e0003530 con 0x7f96e0004d50 2024-04-22T01:18:05.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f7fff640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e0004020 0x7f96e0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:05.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e0004d50 0x7f96e00051d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f96f0001e70 tx=0x7f96f001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dab4636f8713374 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0001a90 con 0x7f96e0004d50 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f0003b80 con 0x7f96e0004d50 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.316+0000 7f96f77fe640 1 -- 172.21.15.70:0/796474994 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0005080 con 0x7f96e0004d50 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/796474994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e0004d50 msgr2=0x7f96e00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 --2- 172.21.15.70:0/796474994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e0004d50 0x7f96e00051d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f96f0001e70 tx=0x7f96f001c780 comp rx=0 tx=0).stop 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 -- 172.21.15.70:0/796474994 reap_dead start 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/796474994 shutdown_connections 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/796474994 >> 172.21.15.70:0/796474994 conn(0x7f96f80b1d40 msgr2=0x7f96f80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/796474994 shutdown_connections 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/796474994 wait complete. 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 Processor -- start 2024-04-22T01:18:05.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- start start 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f814bd00 0x7f96f814dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8149af0 0x7f96f8149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 0x7f96f81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96f80b4830 con 0x7f96f8150440 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96f815d050 con 0x7f96f8149af0 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f96f80b49b0 con 0x7f96f814bd00 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 0x7f96f81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 0x7f96f81508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32782/0 (socket says 172.21.15.70:32782) 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 -- 172.21.15.70:0/1516234774 learned_addr learned my addr 172.21.15.70:0/1516234774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fcaa5640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f814bd00 0x7f96f814dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96f7fff640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8149af0 0x7f96f8149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 -- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f814bd00 msgr2=0x7f96f814dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f814bd00 0x7f96f814dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 -- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8149af0 msgr2=0x7f96f8149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8149af0 0x7f96f8149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 -- 172.21.15.70:0/1516234774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f814f0c0 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fcaa5640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f814bd00 0x7f96f814dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fd2a6640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 0x7f96f81508c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f96f0098190 tx=0x7f96e0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f00b7070 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f0005820 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f814f3a0 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0001a90 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.320+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f814f960 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.324+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f8068210 con 0x7f96f8150440 2024-04-22T01:18:05.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.324+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f96f00ba070 con 0x7f96f8150440 2024-04-22T01:18:05.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.324+0000 7f96f57fa640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96e40678c0 0x7f96e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.324+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f96f011eeb0 con 0x7f96f8150440 2024-04-22T01:18:05.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.324+0000 7f96fcaa5640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96e40678c0 0x7f96e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.324+0000 7f96fcaa5640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96e40678c0 0x7f96e4069d80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098c0 tx=0x7f96e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.328+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f96f00ac020 con 0x7f96f8150440 2024-04-22T01:18:05.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.472+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f96f80d01e0 con 0x7f96f8150440 2024-04-22T01:18:05.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.472+0000 7f96f57fa640 1 -- 172.21.15.70:0/1516234774 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f96f0016490 con 0x7f96f8150440 2024-04-22T01:18:05.476 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:18:05.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96e40678c0 msgr2=0x7f96e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96e40678c0 0x7f96e4069d80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098c0 tx=0x7f96e8008040 comp rx=0 tx=0).stop 2024-04-22T01:18:05.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 msgr2=0x7f96f81508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 0x7f96f81508c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f96f0098190 tx=0x7f96e0005840 comp rx=0 tx=0).stop 2024-04-22T01:18:05.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 shutdown_connections 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f96e40678c0 0x7f96e4069d80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96f8150440 0x7f96f81508c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f8149af0 0x7f96f8149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 --2- 172.21.15.70:0/1516234774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96f814bd00 0x7f96f814dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 >> 172.21.15.70:0/1516234774 conn(0x7f96f80b1d40 msgr2=0x7f96f80b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 shutdown_connections 2024-04-22T01:18:05.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.476+0000 7f96fed09640 1 -- 172.21.15.70:0/1516234774 wait complete. 2024-04-22T01:18:05.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:18:05.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 Processor -- start 2024-04-22T01:18:05.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 -- start start 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdafc0ce9a0 0x7fdafc0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdafc0d1dc0 0x7fdafc0d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdafc0b8330 con 0x7fdafc15e040 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdafc0cf340 con 0x7fdafc0ce9a0 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb0406a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdafc0b8610 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdafc0ce9a0 0x7fdafc0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdafc0ce9a0 0x7fdafc0cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32786/0 (socket says 172.21.15.70:32786) 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 -- 172.21.15.70:0/3490176905 learned_addr learned my addr 172.21.15.70:0/3490176905 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 -- 172.21.15.70:0/3490176905 >> v2:172.21.15.98:3300/0 conn(0x7fdafc0d1dc0 msgr2=0x7fdafc0d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:05.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 --2- 172.21.15.70:0/3490176905 >> v2:172.21.15.98:3300/0 conn(0x7fdafc0d1dc0 0x7fdafc0d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 -- 172.21.15.70:0/3490176905 >> v1:172.21.15.70:6789/0 conn(0x7fdafc15e040 legacy=0x7fdafc0b7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb01605640 1 -- 172.21.15.70:0/3490176905 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc0b88f0 con 0x7fdafc0ce9a0 2024-04-22T01:18:05.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.600+0000 7fdb00e04640 1 -- 172.21.15.70:0/3490176905 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210217883 0 0) 0x7fdafc0b8330 con 0x7fdafc15e040 2024-04-22T01:18:05.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 --2- 172.21.15.70:0/3490176905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0ce9a0 0x7fdafc0cee00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0cacd0 tx=0x7fdaec01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c610832a33e7c3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb00e04640 1 -- 172.21.15.70:0/3490176905 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0b1070 con 0x7fdafc0ce9a0 2024-04-22T01:18:05.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb00e04640 1 -- 172.21.15.70:0/3490176905 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaec0ac630 con 0x7fdafc0ce9a0 2024-04-22T01:18:05.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb00e04640 1 -- 172.21.15.70:0/3490176905 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0a8050 con 0x7fdafc0ce9a0 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- 172.21.15.70:0/3490176905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0ce9a0 msgr2=0x7fdafc0cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3490176905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0ce9a0 0x7fdafc0cee00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0cacd0 tx=0x7fdaec01c3d0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- 172.21.15.70:0/3490176905 shutdown_connections 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3490176905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0ce9a0 0x7fdafc0cee00 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3490176905 >> v2:172.21.15.98:3300/0 conn(0x7fdafc0d1dc0 0x7fdafc0d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- 172.21.15.70:0/3490176905 >> 172.21.15.70:0/3490176905 conn(0x7fdafc072d20 msgr2=0x7fdafc073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- 172.21.15.70:0/3490176905 shutdown_connections 2024-04-22T01:18:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- 172.21.15.70:0/3490176905 wait complete. 2024-04-22T01:18:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 Processor -- start 2024-04-22T01:18:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- start start 2024-04-22T01:18:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdafc0ce9a0 0x7fdafc15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 0x7fdafc152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdafc15efb0 0x7fdafc153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc0b7f80 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc16a9f0 con 0x7fdafc15efb0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb0406a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdafc15eb20 con 0x7fdafc0ce9a0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 0x7fdafc152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 0x7fdafc152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32788/0 (socket says 172.21.15.70:32788) 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 -- 172.21.15.70:0/3812527153 learned_addr learned my addr 172.21.15.70:0/3812527153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 -- 172.21.15.70:0/3812527153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdafc0ce9a0 msgr2=0x7fdafc15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdafc0ce9a0 0x7fdafc15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 -- 172.21.15.70:0/3812527153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdafc15efb0 msgr2=0x7fdafc153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdafc15efb0 0x7fdafc153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.604+0000 7fdb01605640 1 -- 172.21.15.70:0/3812527153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc160930 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdb01605640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 0x7fdafc152f20 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0af040 tx=0x7fdaec01ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0b1040 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaec004020 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdafc160c10 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0043d0 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdafc15e4d0 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdafc04f510 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdaec0b4070 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdaf27fc640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdae00678f0 0x7fdae0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fdaec120220 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdb01e06640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdae00678f0 0x7fdae0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.608+0000 7fdb01e06640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdae00678f0 0x7fdae0069db0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8007fd0 tx=0x7fdaf8007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:05.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.612+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdaec0bb030 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.772+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdafc15aad0 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.772+0000 7fdaf27fc640 1 -- 172.21.15.70:0/3812527153 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) ==== 80+0+0 (secure 0 0 0) 0x7fdaec016110 con 0x7fdafc0d1dc0 2024-04-22T01:18:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdae00678f0 msgr2=0x7fdae0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdae00678f0 0x7fdae0069db0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8007fd0 tx=0x7fdaf8007920 comp rx=0 tx=0).stop 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 msgr2=0x7fdafc152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 0x7fdafc152f20 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0af040 tx=0x7fdaec01ac10 comp rx=0 tx=0).stop 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 shutdown_connections 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdae00678f0 0x7fdae0069db0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdafc15efb0 0x7fdafc153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdafc0d1dc0 0x7fdafc152f20 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 --2- 172.21.15.70:0/3812527153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdafc0ce9a0 0x7fdafc15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 >> 172.21.15.70:0/3812527153 conn(0x7fdafc072d20 msgr2=0x7fdafc15d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 shutdown_connections 2024-04-22T01:18:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:05.776+0000 7fdb0406a640 1 -- 172.21.15.70:0/3812527153 wait complete. 2024-04-22T01:18:05.794 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:18:05.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:05.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:05.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[43324]: fuse finished with error 0 and tester_r 0 2024-04-22T01:18:05.804 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:18:05.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:05.804 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:18:05.892 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:05.892 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:05.892 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:05.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:06.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.052+0000 7f12ad2ea640 1 Processor -- start 2024-04-22T01:18:06.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.052+0000 7f12ad2ea640 1 -- start start 2024-04-22T01:18:06.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12ad2ea640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f12a815aa30 0x7f12a80b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12ad2ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 0x7f12a80b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12ad2ea640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a806dbd0 con 0x7f12a815bb50 2024-04-22T01:18:06.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12ad2ea640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f12a806de70 con 0x7f12a815aa30 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12ad2ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f12a806e150 con 0x7f12a80aee40 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a67fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f12a815bb50 0x7f12a80ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41982/0 (socket says 172.21.15.70:41982) 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a67fc640 1 -- 172.21.15.70:0/2169232158 learned_addr learned my addr 172.21.15.70:0/2169232158 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1948488754 0 0) 0x7f12a806dbd0 con 0x7f12a815bb50 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a77fe640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 0x7f12a80b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2169232158 >> v2:172.21.15.98:3300/0 conn(0x7f12a815aa30 0x7f12a80b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1294003530 con 0x7f12a815bb50 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1587889829 0 0) 0x7f1294003530 con 0x7f12a815bb50 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 msgr2=0x7f12a80b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 0x7f12a80b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 >> v2:172.21.15.98:3300/0 conn(0x7f12a815aa30 msgr2=0x7f12a80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 --2- 172.21.15.70:0/2169232158 >> v2:172.21.15.98:3300/0 conn(0x7f12a815aa30 0x7f12a80b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a806e430 con 0x7f12a815bb50 2024-04-22T01:18:06.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1290001680 con 0x7f12a815bb50 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 >> v1:172.21.15.70:6789/0 conn(0x7f12a815bb50 legacy=0x7f12a80ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12940039c0 0x7f1294003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 0x7f1294004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294004d50 0x7f12940051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12a806de70 con 0x7f12940039c0 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12a806e150 con 0x7f1294004d50 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f129400b9d0 con 0x7f12a80aee40 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12940039c0 0x7f1294003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 -- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 msgr2=0x7f1294004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12a80aee40 0x7f1294004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 -- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294004d50 msgr2=0x7f12940051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a77fe640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294004d50 0x7f12940051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294004d50 0x7f12940051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.056+0000 7f12a6ffd640 1 -- 172.21.15.70:0/2169232158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1294003530 con 0x7f12940039c0 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12940039c0 0x7f1294003de0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1298009fc0 tx=0x7f129801cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3768ed06359f9384 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12980b2070 con 0x7f12940039c0 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12980a5ca0 con 0x7f12940039c0 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a5ffb640 1 -- 172.21.15.70:0/2169232158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298009c10 con 0x7f12940039c0 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2169232158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12940039c0 msgr2=0x7f1294003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2169232158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12940039c0 0x7f1294003de0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1298009fc0 tx=0x7f129801cc90 comp rx=0 tx=0).stop 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a77fe640 1 -- 172.21.15.70:0/2169232158 reap_dead start 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2169232158 shutdown_connections 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2169232158 >> 172.21.15.70:0/2169232158 conn(0x7f12a806b1f0 msgr2=0x7f12a806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2169232158 shutdown_connections 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2169232158 wait complete. 2024-04-22T01:18:06.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 Processor -- start 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- start start 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12940039c0 0x7f12a80b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 0x7f12a81524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a8159990 0x7f12a8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12a806ead0 con 0x7f12a81f9e90 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12a806e950 con 0x7f12a8159990 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f12a80b3100 con 0x7f12940039c0 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 0x7f12a81524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 0x7f12a81524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32808/0 (socket says 172.21.15.70:32808) 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 -- 172.21.15.70:0/2421900480 learned_addr learned my addr 172.21.15.70:0/2421900480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12940039c0 0x7f12a80b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a77fe640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a8159990 0x7f12a8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 -- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12940039c0 msgr2=0x7f12a80b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12940039c0 0x7f12a80b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 -- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a8159990 msgr2=0x7f12a8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a8159990 0x7f12a8071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 -- 172.21.15.70:0/2421900480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a80b0af0 con 0x7f12a81f9e90 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a77fe640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a8159990 0x7f12a8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12a67fc640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 0x7f12a81524b0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f129000c840 tx=0x7f129000cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290018070 con 0x7f12a81f9e90 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12900074e0 con 0x7f12a81f9e90 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a81fa320 con 0x7f12a81f9e90 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290004a30 con 0x7f12a81f9e90 2024-04-22T01:18:06.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.060+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a81fa8e0 con 0x7f12a81f9e90 2024-04-22T01:18:06.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.064+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a8068210 con 0x7f12a81f9e90 2024-04-22T01:18:06.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.064+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f129001b070 con 0x7f12a81f9e90 2024-04-22T01:18:06.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.064+0000 7f129f7fe640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12880678c0 0x7f1288069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.064+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12880678c0 0x7f1288069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.064+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f1290085370 con 0x7f12a81f9e90 2024-04-22T01:18:06.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.064+0000 7f12a6ffd640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12880678c0 0x7f1288069d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f12980af000 tx=0x7f12980a7040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.068+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1290055d20 con 0x7f12a81f9e90 2024-04-22T01:18:06.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.216+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12a806f190 con 0x7f12a81f9e90 2024-04-22T01:18:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.216+0000 7f129f7fe640 1 -- 172.21.15.70:0/2421900480 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7f1290055620 con 0x7f12a81f9e90 2024-04-22T01:18:06.221 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:06.222 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12880678c0 msgr2=0x7f1288069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12880678c0 0x7f1288069d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f12980af000 tx=0x7f12980a7040 comp rx=0 tx=0).stop 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 msgr2=0x7f12a81524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 0x7f12a81524b0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f129000c840 tx=0x7f129000cd00 comp rx=0 tx=0).stop 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 shutdown_connections 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f12880678c0 0x7f1288069d80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a8159990 0x7f12a8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12a81f9e90 0x7f12a81524b0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 --2- 172.21.15.70:0/2421900480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12940039c0 0x7f12a80b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 >> 172.21.15.70:0/2421900480 conn(0x7f12a806b1f0 msgr2=0x7f12a806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 shutdown_connections 2024-04-22T01:18:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.220+0000 7f12ad2ea640 1 -- 172.21.15.70:0/2421900480 wait complete. 2024-04-22T01:18:06.238 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:39.433363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:40.173738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:06.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:06.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.356+0000 7f775ecc2640 1 Processor -- start 2024-04-22T01:18:06.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.356+0000 7f775ecc2640 1 -- start start 2024-04-22T01:18:06.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f775815e6e0 0x7f775815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f775806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775815f000 con 0x7f77580ce920 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f77580b3800 con 0x7f775815e6e0 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f77580b3ae0 con 0x7f7758155370 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f775815e6e0 0x7f775815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f775815e6e0 0x7f775815eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59492/0 (socket says 172.21.15.70:59492) 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 -- 172.21.15.70:0/1431466625 learned_addr learned my addr 172.21.15.70:0/1431466625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775d25f640 1 --2- 172.21.15.70:0/1431466625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f775806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 -- 172.21.15.70:0/1431466625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 msgr2=0x7f775806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 --2- 172.21.15.70:0/1431466625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f775806d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 -- 172.21.15.70:0/1431466625 >> v1:172.21.15.98:6789/0 conn(0x7f77580ce920 legacy=0x7f7758154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 -- 172.21.15.70:0/1431466625 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77580cde40 con 0x7f775815e6e0 2024-04-22T01:18:06.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ca5e640 1 --2- 172.21.15.70:0/1431466625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815eac0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7750009e50 tx=0x7f775001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e58ce419772f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f77577fe640 1 -- 172.21.15.70:0/1431466625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500a5840 con 0x7f775815e6e0 2024-04-22T01:18:06.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f77577fe640 1 -- 172.21.15.70:0/1431466625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77500a59e0 con 0x7f775815e6e0 2024-04-22T01:18:06.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f77577fe640 1 -- 172.21.15.70:0/1431466625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500a5db0 con 0x7f775815e6e0 2024-04-22T01:18:06.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- 172.21.15.70:0/1431466625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 msgr2=0x7f775815eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 --2- 172.21.15.70:0/1431466625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815eac0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7750009e50 tx=0x7f775001c9c0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- 172.21.15.70:0/1431466625 shutdown_connections 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 --2- 172.21.15.70:0/1431466625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f775806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 --2- 172.21.15.70:0/1431466625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815eac0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- 172.21.15.70:0/1431466625 >> 172.21.15.70:0/1431466625 conn(0x7f77580b1d40 msgr2=0x7f77580cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- 172.21.15.70:0/1431466625 shutdown_connections 2024-04-22T01:18:06.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.360+0000 7f775ecc2640 1 -- 172.21.15.70:0/1431466625 wait complete. 2024-04-22T01:18:06.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 Processor -- start 2024-04-22T01:18:06.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- start start 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77580ce920 0x7f77581443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f77580b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f775806f260 con 0x7f775815e6e0 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f775806f0e0 con 0x7f77580ce920 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f77580b32c0 con 0x7f7758155370 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59494/0 (socket says 172.21.15.70:59494) 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 -- 172.21.15.70:0/3900017777 learned_addr learned my addr 172.21.15.70:0/3900017777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:06.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ca5e640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77580ce920 0x7f77581443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f7757fff640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f77580b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 -- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 msgr2=0x7f77580b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f77580b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 -- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77580ce920 msgr2=0x7f77581443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77580ce920 0x7f77581443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 -- 172.21.15.70:0/3900017777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77580b1080 con 0x7f775815e6e0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775d25f640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815c3d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f774800d600 tx=0x7f774800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77480125f0 con 0x7f775815e6e0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7748019070 con 0x7f775815e6e0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775815a000 con 0x7f775815e6e0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77480129f0 con 0x7f775815e6e0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77580b0340 con 0x7f775815e6e0 2024-04-22T01:18:06.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.364+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7758068210 con 0x7f775815e6e0 2024-04-22T01:18:06.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.368+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7748012b90 con 0x7f775815e6e0 2024-04-22T01:18:06.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.368+0000 7f77557fa640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77440678c0 0x7f7744069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.368+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f7748086260 con 0x7f775815e6e0 2024-04-22T01:18:06.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.368+0000 7f775ca5e640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77440678c0 0x7f7744069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.368+0000 7f775ca5e640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77440678c0 0x7f7744069d80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f77500ae000 tx=0x7f77500a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.372+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f77580b1080 con 0x7f775815e6e0 2024-04-22T01:18:06.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77580d08e0 con 0x7f775815e6e0 2024-04-22T01:18:06.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f77557fa640 1 -- 172.21.15.70:0/3900017777 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f7748056c10 con 0x7f775815e6e0 2024-04-22T01:18:06.538 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77440678c0 msgr2=0x7f7744069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77440678c0 0x7f7744069d80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f77500ae000 tx=0x7f77500a7070 comp rx=0 tx=0).stop 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 msgr2=0x7f775815c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815c3d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f774800d600 tx=0x7f774800dac0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 shutdown_connections 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f77440678c0 0x7f7744069d80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775815e6e0 0x7f775815c3d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7758155370 0x7f77580b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 --2- 172.21.15.70:0/3900017777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77580ce920 0x7f77581443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 >> 172.21.15.70:0/3900017777 conn(0x7f77580b1d40 msgr2=0x7f77580b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 shutdown_connections 2024-04-22T01:18:06.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.536+0000 7f775ecc2640 1 -- 172.21.15.70:0/3900017777 wait complete. 2024-04-22T01:18:06.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:18:06.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 Processor -- start 2024-04-22T01:18:06.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 -- start start 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f400b98a0 0x7f1f400d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1f400cea10 0x7f1f400ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f400b9ce0 con 0x7f1f4015de20 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1f400cf3e0 con 0x7f1f400cea10 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f47d4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1f400b75e0 con 0x7f1f400b98a0 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1f400cea10 0x7f1f400ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1f400cea10 0x7f1f400ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59506/0 (socket says 172.21.15.70:59506) 2024-04-22T01:18:06.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 -- 172.21.15.70:0/1459034864 learned_addr learned my addr 172.21.15.70:0/1459034864 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:06.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f452e5640 1 --2- 172.21.15.70:0/1459034864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f400b98a0 0x7f1f400d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 -- 172.21.15.70:0/1459034864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f400b98a0 msgr2=0x7f1f400d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 --2- 172.21.15.70:0/1459034864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f400b98a0 0x7f1f400d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 -- 172.21.15.70:0/1459034864 >> v1:172.21.15.70:6789/0 conn(0x7f1f4015de20 legacy=0x7f1f400b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.676+0000 7f1f462e7640 1 -- 172.21.15.70:0/1459034864 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f400b78c0 con 0x7f1f400cea10 2024-04-22T01:18:06.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f462e7640 1 --2- 172.21.15.70:0/1459034864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400ceea0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f1f400cf5c0 tx=0x7f1f3801f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1850181c052402 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f44ae4640 1 -- 172.21.15.70:0/1459034864 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f380b4070 con 0x7f1f400cea10 2024-04-22T01:18:06.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f44ae4640 1 -- 172.21.15.70:0/1459034864 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f380af400 con 0x7f1f400cea10 2024-04-22T01:18:06.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f44ae4640 1 -- 172.21.15.70:0/1459034864 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f380af7d0 con 0x7f1f400cea10 2024-04-22T01:18:06.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- 172.21.15.70:0/1459034864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 msgr2=0x7f1f400ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/1459034864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400ceea0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f1f400cf5c0 tx=0x7f1f3801f200 comp rx=0 tx=0).stop 2024-04-22T01:18:06.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- 172.21.15.70:0/1459034864 shutdown_connections 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/1459034864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400ceea0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/1459034864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f400b98a0 0x7f1f400d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- 172.21.15.70:0/1459034864 >> 172.21.15.70:0/1459034864 conn(0x7f1f40072d20 msgr2=0x7f1f40073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- 172.21.15.70:0/1459034864 shutdown_connections 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- 172.21.15.70:0/1459034864 wait complete. 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 Processor -- start 2024-04-22T01:18:06.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- start start 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f400b98a0 0x7f1f40147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400d2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f4015de20 0x7f1f400cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1f400b8160 con 0x7f1f400cea10 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f400b7e80 con 0x7f1f400b98a0 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f47d4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1f4015e200 con 0x7f1f4015de20 2024-04-22T01:18:06.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400d2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400d2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59514/0 (socket says 172.21.15.70:59514) 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 -- 172.21.15.70:0/533763074 learned_addr learned my addr 172.21.15.70:0/533763074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 -- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f4015de20 msgr2=0x7f1f400cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f4015de20 0x7f1f400cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 -- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f400b98a0 msgr2=0x7f1f40147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f45ae6640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f400b98a0 0x7f1f40147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f400b98a0 0x7f1f40147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.680+0000 7f1f452e5640 1 -- 172.21.15.70:0/533763074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f400d3250 con 0x7f1f400cea10 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f452e5640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400d2a20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400d800 tx=0x7f1f3400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400b6c0 con 0x7f1f400cea10 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f34019070 con 0x7f1f400cea10 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400ba60 con 0x7f1f400cea10 2024-04-22T01:18:06.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f400c8ea0 con 0x7f1f400cea10 2024-04-22T01:18:06.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f400d2f60 con 0x7f1f400cea10 2024-04-22T01:18:06.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f40068210 con 0x7f1f400cea10 2024-04-22T01:18:06.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1f3400bc00 con 0x7f1f400cea10 2024-04-22T01:18:06.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f327fc640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1f240678c0 0x7f1f24069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f1f340862a0 con 0x7f1f400cea10 2024-04-22T01:18:06.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f45ae6640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1f240678c0 0x7f1f24069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.684+0000 7f1f45ae6640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1f240678c0 0x7f1f24069d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c007fd0 tx=0x7f1f3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:06.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.688+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1f34056c40 con 0x7f1f400cea10 2024-04-22T01:18:06.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.848+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1f400cd620 con 0x7f1f400cea10 2024-04-22T01:18:06.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.848+0000 7f1f327fc640 1 -- 172.21.15.70:0/533763074 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f1f34056540 con 0x7f1f400cea10 2024-04-22T01:18:06.851 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.848+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1f240678c0 msgr2=0x7f1f24069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.848+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1f240678c0 0x7f1f24069d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c007fd0 tx=0x7f1f3c009040 comp rx=0 tx=0).stop 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 msgr2=0x7f1f400d2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400d2a20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400d800 tx=0x7f1f3400dcc0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 shutdown_connections 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1f240678c0 0x7f1f24069d80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f4015de20 0x7f1f400cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f400cea10 0x7f1f400d2a20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 --2- 172.21.15.70:0/533763074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f400b98a0 0x7f1f40147b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 >> 172.21.15.70:0/533763074 conn(0x7f1f40072d20 msgr2=0x7f1f40073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:06.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 shutdown_connections 2024-04-22T01:18:06.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.852+0000 7f1f47d4a640 1 -- 172.21.15.70:0/533763074 wait complete. 2024-04-22T01:18:06.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:18:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.988+0000 7fb11231d640 1 Processor -- start 2024-04-22T01:18:06.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb11231d640 1 -- start start 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb11231d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb10c15e6e0 0x7fb10c15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb11231d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb10c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb11231d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10c15f000 con 0x7fb10c0ce920 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb11231d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb10c0b3800 con 0x7fb10c15e6e0 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb11231d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c0b3ae0 con 0x7fb10c155370 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb10c0ce920 0x7fb10c154c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56758/0 (socket says 172.21.15.70:56758) 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10b7fe640 1 -- 172.21.15.70:0/3103994758 learned_addr learned my addr 172.21.15.70:0/3103994758 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1725110208 0 0) 0x7fb10c15f000 con 0x7fb10c0ce920 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0f4003530 con 0x7fb10c0ce920 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 --2- 172.21.15.70:0/3103994758 >> v2:172.21.15.98:3300/0 conn(0x7fb10c15e6e0 0x7fb10c15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb1108ba640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb10c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 457480719 0 0) 0x7fb0f4003530 con 0x7fb10c0ce920 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 msgr2=0x7fb10c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb10c06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 >> v2:172.21.15.98:3300/0 conn(0x7fb10c15e6e0 msgr2=0x7fb10c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 --2- 172.21.15.70:0/3103994758 >> v2:172.21.15.98:3300/0 conn(0x7fb10c15e6e0 0x7fb10c15eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c0cde40 con 0x7fb10c0ce920 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fb100001680 con 0x7fb10c0ce920 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 >> v1:172.21.15.70:6789/0 conn(0x7fb10c0ce920 legacy=0x7fb10c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0f40039c0 0x7fb0f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb0f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f4004d50 0x7fb0f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0b3800 con 0x7fb0f40039c0 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0b3ae0 con 0x7fb0f4004d50 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f400ba00 con 0x7fb10c155370 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0f40039c0 0x7fb0f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10b7fe640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb0f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb1108ba640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f4004d50 0x7fb0f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 -- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 msgr2=0x7fb0f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb0f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 -- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f4004d50 msgr2=0x7fb0f40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f4004d50 0x7fb0f40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10bfff640 1 -- 172.21.15.70:0/3103994758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f4003530 con 0x7fb0f40039c0 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb10b7fe640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c155370 0x7fb0f4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:06.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.992+0000 7fb1108ba640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f4004d50 0x7fb0f40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:07.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb10bfff640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0f40039c0 0x7fb0f4003de0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009ae0 tx=0x7fb0fc01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8c7243b31234a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0b2070 con 0x7fb0f40039c0 2024-04-22T01:18:07.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc0a5650 con 0x7fb0f40039c0 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb10affd640 1 -- 172.21.15.70:0/3103994758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0096b0 con 0x7fb0f40039c0 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- 172.21.15.70:0/3103994758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0f40039c0 msgr2=0x7fb0f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 --2- 172.21.15.70:0/3103994758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0f40039c0 0x7fb0f4003de0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009ae0 tx=0x7fb0fc01ca80 comp rx=0 tx=0).stop 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 -- 172.21.15.70:0/3103994758 reap_dead start 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- 172.21.15.70:0/3103994758 shutdown_connections 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- 172.21.15.70:0/3103994758 >> 172.21.15.70:0/3103994758 conn(0x7fb10c0b1d40 msgr2=0x7fb10c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- 172.21.15.70:0/3103994758 shutdown_connections 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- 172.21.15.70:0/3103994758 wait complete. 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 Processor -- start 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- start start 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb10c139250 0x7fb10c14f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c139630 0x7fb10c153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 0x7fb10c1512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0b2c30 con 0x7fb10c152410 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0b2ab0 con 0x7fb10c139250 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb11231d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c0b2db0 con 0x7fb10c139630 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 0x7fb10c1512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 0x7fb10c1512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59532/0 (socket says 172.21.15.70:59532) 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 -- 172.21.15.70:0/1881634532 learned_addr learned my addr 172.21.15.70:0/1881634532 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb10bfff640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb10c139250 0x7fb10c14f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb10b7fe640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c139630 0x7fb10c153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 -- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c139630 msgr2=0x7fb10c153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c139630 0x7fb10c153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 -- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb10c139250 msgr2=0x7fb10c14f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb10c139250 0x7fb10c14f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:06.996+0000 7fb1108ba640 1 -- 172.21.15.70:0/1881634532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c152960 con 0x7fb10c152410 2024-04-22T01:18:07.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb1108ba640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 0x7fb10c1512e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0b26b0 tx=0x7fb104009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104019070 con 0x7fb10c152410 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb104007af0 con 0x7fb10c152410 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c152c40 con 0x7fb10c152410 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104014400 con 0x7fb10c152410 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c153200 con 0x7fb10c152410 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c068210 con 0x7fb10c152410 2024-04-22T01:18:07.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb10401c070 con 0x7fb10c152410 2024-04-22T01:18:07.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb108ff9640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0f80678f0 0x7fb0f8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fb104085f00 con 0x7fb10c152410 2024-04-22T01:18:07.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb10bfff640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0f80678f0 0x7fb0f8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.000+0000 7fb10bfff640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0f80678f0 0x7fb0f8069db0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009ae0 tx=0x7fb0fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.004+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb1040562d0 con 0x7fb10c152410 2024-04-22T01:18:07.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.152+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb10c0d08e0 con 0x7fb10c152410 2024-04-22T01:18:07.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.152+0000 7fb108ff9640 1 -- 172.21.15.70:0/1881634532 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7fb10405a5d0 con 0x7fb10c152410 2024-04-22T01:18:07.155 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.152+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0f80678f0 msgr2=0x7fb0f8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.152+0000 7fb11231d640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0f80678f0 0x7fb0f8069db0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009ae0 tx=0x7fb0fc008040 comp rx=0 tx=0).stop 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.152+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 msgr2=0x7fb10c1512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.152+0000 7fb11231d640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 0x7fb10c1512e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0b26b0 tx=0x7fb104009c10 comp rx=0 tx=0).stop 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 shutdown_connections 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0f80678f0 0x7fb0f8069db0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb10c152410 0x7fb10c1512e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10c139630 0x7fb10c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 --2- 172.21.15.70:0/1881634532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb10c139250 0x7fb10c14f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 >> 172.21.15.70:0/1881634532 conn(0x7fb10c0b1d40 msgr2=0x7fb10c1579c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 shutdown_connections 2024-04-22T01:18:07.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.156+0000 7fb11231d640 1 -- 172.21.15.70:0/1881634532 wait complete. 2024-04-22T01:18:07.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:18:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 Processor -- start 2024-04-22T01:18:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 -- start start 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcff806b1f0 0x7fcff806b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff815c470 con 0x7fcff80cdfe0 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcff806bb80 con 0x7fcff815bd00 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffea80640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcff80b1f80 con 0x7fcff806b1f0 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59538/0 (socket says 172.21.15.70:59538) 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 -- 172.21.15.70:0/1933256801 learned_addr learned my addr 172.21.15.70:0/1933256801 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffd01d640 1 --2- 172.21.15.70:0/1933256801 >> v2:172.21.15.98:3300/0 conn(0x7fcff806b1f0 0x7fcff806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 -- 172.21.15.70:0/1933256801 >> v2:172.21.15.98:3300/0 conn(0x7fcff806b1f0 msgr2=0x7fcff806b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 --2- 172.21.15.70:0/1933256801 >> v2:172.21.15.98:3300/0 conn(0x7fcff806b1f0 0x7fcff806b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 -- 172.21.15.70:0/1933256801 >> v1:172.21.15.98:6789/0 conn(0x7fcff80cdfe0 legacy=0x7fcff80b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffc81c640 1 -- 172.21.15.70:0/1933256801 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff815c710 con 0x7fcff815bd00 2024-04-22T01:18:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.288+0000 7fcffd01d640 1 --2- 172.21.15.70:0/1933256801 >> v2:172.21.15.98:3300/0 conn(0x7fcff806b1f0 0x7fcff806b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffc81c640 1 --2- 172.21.15.70:0/1933256801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcff80cafd0 tx=0x7fcfe801ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356ac679946e45b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcff77fe640 1 -- 172.21.15.70:0/1933256801 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe80ab640 con 0x7fcff815bd00 2024-04-22T01:18:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcff77fe640 1 -- 172.21.15.70:0/1933256801 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe80b0070 con 0x7fcff815bd00 2024-04-22T01:18:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcff77fe640 1 -- 172.21.15.70:0/1933256801 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe80aba00 con 0x7fcff815bd00 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- 172.21.15.70:0/1933256801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff815bd00 msgr2=0x7fcff815cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 --2- 172.21.15.70:0/1933256801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcff80cafd0 tx=0x7fcfe801ce80 comp rx=0 tx=0).stop 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- 172.21.15.70:0/1933256801 shutdown_connections 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 --2- 172.21.15.70:0/1933256801 >> v2:172.21.15.98:3300/0 conn(0x7fcff806b1f0 0x7fcff806b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 --2- 172.21.15.70:0/1933256801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- 172.21.15.70:0/1933256801 >> 172.21.15.70:0/1933256801 conn(0x7fcff8099a60 msgr2=0x7fcff8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- 172.21.15.70:0/1933256801 shutdown_connections 2024-04-22T01:18:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- 172.21.15.70:0/1933256801 wait complete. 2024-04-22T01:18:07.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 Processor -- start 2024-04-22T01:18:07.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- start start 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806b1f0 0x7fcff8159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 0x7fcff8165970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81602e0 con 0x7fcff80cdfe0 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8160160 con 0x7fcff815bd00 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.292+0000 7fcffea80640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcff8160460 con 0x7fcff806b1f0 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 0x7fcff8165970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 0x7fcff8165970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59540/0 (socket says 172.21.15.70:59540) 2024-04-22T01:18:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 -- 172.21.15.70:0/964845104 learned_addr learned my addr 172.21.15.70:0/964845104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffd01d640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffc81c640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806b1f0 0x7fcff8159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 -- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806b1f0 msgr2=0x7fcff8159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806b1f0 0x7fcff8159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 -- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff815bd00 msgr2=0x7fcff815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 -- 172.21.15.70:0/964845104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff80cc010 con 0x7fcff80cdfe0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffc81c640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806b1f0 0x7fcff8159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff7fff640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 0x7fcff8165970 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00c410 tx=0x7fcfec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec011790 con 0x7fcff80cdfe0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfec017070 con 0x7fcff80cdfe0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff80b1310 con 0x7fcff80cdfe0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec011bb0 con 0x7fcff80cdfe0 2024-04-22T01:18:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff815a360 con 0x7fcff80cdfe0 2024-04-22T01:18:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff804f510 con 0x7fcff80cdfe0 2024-04-22T01:18:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcfec01c460 con 0x7fcff80cdfe0 2024-04-22T01:18:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff57fa640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfe40678f0 0x7fcfe4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffc81c640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfe40678f0 0x7fcfe4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fcfec085a50 con 0x7fcff80cdfe0 2024-04-22T01:18:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.296+0000 7fcffc81c640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfe40678f0 0x7fcfe4069db0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80af000 tx=0x7fcfe8004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.300+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcfec021020 con 0x7fcff80cdfe0 2024-04-22T01:18:07.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.472+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcff806b890 con 0x7fcff80cdfe0 2024-04-22T01:18:07.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.472+0000 7fcff57fa640 1 -- 172.21.15.70:0/964845104 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v78) ==== 80+0+0 (secure 0 0 0) 0x7fcfec056400 con 0x7fcff80cdfe0 2024-04-22T01:18:07.476 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:18:07.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfe40678f0 msgr2=0x7fcfe4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfe40678f0 0x7fcfe4069db0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80af000 tx=0x7fcfe8004740 comp rx=0 tx=0).stop 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 msgr2=0x7fcff8165970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 0x7fcff8165970 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00c410 tx=0x7fcfec00c8d0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 shutdown_connections 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcfe40678f0 0x7fcfe4069db0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff815bd00 0x7fcff815cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcff80cdfe0 0x7fcff8165970 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 --2- 172.21.15.70:0/964845104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806b1f0 0x7fcff8159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 >> 172.21.15.70:0/964845104 conn(0x7fcff8099a60 msgr2=0x7fcff806f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 shutdown_connections 2024-04-22T01:18:07.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.476+0000 7fcffea80640 1 -- 172.21.15.70:0/964845104 wait complete. 2024-04-22T01:18:07.496 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:18:07.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:07.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:07.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34471]: fuse finished with error 0 and tester_r 0 2024-04-22T01:18:07.511 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:18:07.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:07.511 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:18:07.679 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:07.679 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:07.679 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:07.685 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:18:07.685 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:18:07.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:07.686 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:18:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-22T01:18:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:18:07.734 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:07.734 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:07.735 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:07.741 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:18:07.741 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:18:07.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:07.741 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:18:07.794 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-22T01:18:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:18:07.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:07.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 Processor -- start 2024-04-22T01:18:07.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 -- start start 2024-04-22T01:18:07.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efca40cea80 0x7efca40ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efca40d2110 0x7efca40d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca415e390 con 0x7efca415dfb0 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efca40cf420 con 0x7efca40cea80 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca9e7f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efca4161e00 con 0x7efca40d2110 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efca40cea80 0x7efca40ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efca40cea80 0x7efca40ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59546/0 (socket says 172.21.15.70:59546) 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca37fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efca415dfb0 0x7efca40b95d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56770/0 (socket says 172.21.15.70:56770) 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 -- 172.21.15.70:0/3159561482 learned_addr learned my addr 172.21.15.70:0/3159561482 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca3fff640 1 --2- 172.21.15.70:0/3159561482 >> v2:172.21.15.98:3300/0 conn(0x7efca40d2110 0x7efca40d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 -- 172.21.15.70:0/3159561482 >> v2:172.21.15.98:3300/0 conn(0x7efca40d2110 msgr2=0x7efca40d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 --2- 172.21.15.70:0/3159561482 >> v2:172.21.15.98:3300/0 conn(0x7efca40d2110 0x7efca40d25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 -- 172.21.15.70:0/3159561482 >> v1:172.21.15.70:6789/0 conn(0x7efca415dfb0 legacy=0x7efca40b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca27fc640 1 -- 172.21.15.70:0/3159561482 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1068952612 0 0) 0x7efca415e390 con 0x7efca415dfb0 2024-04-22T01:18:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca2ffd640 1 -- 172.21.15.70:0/3159561482 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca41620e0 con 0x7efca40cea80 2024-04-22T01:18:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.924+0000 7efca3fff640 1 --2- 172.21.15.70:0/3159561482 >> v2:172.21.15.98:3300/0 conn(0x7efca40d2110 0x7efca40d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:07.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca2ffd640 1 --2- 172.21.15.70:0/3159561482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca40cea80 0x7efca40ceee0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7efca40ce4b0 tx=0x7efc9801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66926a071295ec7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca27fc640 1 -- 172.21.15.70:0/3159561482 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc980ab430 con 0x7efca40cea80 2024-04-22T01:18:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca27fc640 1 -- 172.21.15.70:0/3159561482 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc980b0070 con 0x7efca40cea80 2024-04-22T01:18:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca27fc640 1 -- 172.21.15.70:0/3159561482 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc980ab7f0 con 0x7efca40cea80 2024-04-22T01:18:07.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 -- 172.21.15.70:0/3159561482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca40cea80 msgr2=0x7efca40ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 --2- 172.21.15.70:0/3159561482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca40cea80 0x7efca40ceee0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7efca40ce4b0 tx=0x7efc9801c520 comp rx=0 tx=0).stop 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 -- 172.21.15.70:0/3159561482 shutdown_connections 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 --2- 172.21.15.70:0/3159561482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca40cea80 0x7efca40ceee0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 --2- 172.21.15.70:0/3159561482 >> v2:172.21.15.98:3300/0 conn(0x7efca40d2110 0x7efca40d25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 -- 172.21.15.70:0/3159561482 >> 172.21.15.70:0/3159561482 conn(0x7efca406c9b0 msgr2=0x7efca406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 -- 172.21.15.70:0/3159561482 shutdown_connections 2024-04-22T01:18:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 -- 172.21.15.70:0/3159561482 wait complete. 2024-04-22T01:18:07.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.928+0000 7efca9e7f640 1 Processor -- start 2024-04-22T01:18:07.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- start start 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca40cea80 0x7efca4154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40d2110 0x7efca4154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 0x7efca40b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efca416af50 con 0x7efca415dfb0 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efca416add0 con 0x7efca40cea80 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efca416b0d0 con 0x7efca40d2110 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 0x7efca40b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 0x7efca40b83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59552/0 (socket says 172.21.15.70:59552) 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 -- 172.21.15.70:0/1008605574 learned_addr learned my addr 172.21.15.70:0/1008605574 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca37fe640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca40cea80 0x7efca4154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 -- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40d2110 msgr2=0x7efca4154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40d2110 0x7efca4154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 -- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca40cea80 msgr2=0x7efca4154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca40cea80 0x7efca4154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 -- 172.21.15.70:0/1008605574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca41554c0 con 0x7efca415dfb0 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca3fff640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 0x7efca40b83c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00cd50 tx=0x7efc9c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00f680 con 0x7efca415dfb0 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9c00f840 con 0x7efca415dfb0 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00fc30 con 0x7efca415dfb0 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca41557a0 con 0x7efca415dfb0 2024-04-22T01:18:07.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca4155d60 con 0x7efca415dfb0 2024-04-22T01:18:07.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca4155fa0 con 0x7efca415dfb0 2024-04-22T01:18:07.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc9c0213f0 con 0x7efca415dfb0 2024-04-22T01:18:07.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efc8bfff640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc900678c0 0x7efc90069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:07.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7efc9c086c80 con 0x7efca415dfb0 2024-04-22T01:18:07.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca37fe640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc900678c0 0x7efc90069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:07.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.932+0000 7efca37fe640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc900678c0 0x7efc90069d80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efc94000f70 tx=0x7efc94009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:07.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:07.936+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7efca4155fa0 con 0x7efca415dfb0 2024-04-22T01:18:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.128+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efca4068210 con 0x7efca415dfb0 2024-04-22T01:18:08.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.128+0000 7efc8bfff640 1 -- 172.21.15.70:0/1008605574 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5174 (secure 0 0 0) 0x7efc9c014030 con 0x7efca415dfb0 2024-04-22T01:18:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:08.132 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:41.174335+0000","modified":"2024-04-22T01:17:43.233600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6759":{"gid":6759,"name":"d","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6833/258015518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":258015518},{"type":"v1","addr":"172.21.15.98:6833","nonce":258015518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6759,"qdb_cluster":[6759]},"id":5}]} 2024-04-22T01:18:08.132 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 37 2024-04-22T01:18:08.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc900678c0 msgr2=0x7efc90069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc900678c0 0x7efc90069d80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efc94000f70 tx=0x7efc94009040 comp rx=0 tx=0).stop 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 msgr2=0x7efca40b83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 0x7efca40b83c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00cd50 tx=0x7efc9c00a650 comp rx=0 tx=0).stop 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 shutdown_connections 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc900678c0 0x7efc90069d80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efca415dfb0 0x7efca40b83c0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40d2110 0x7efca4154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 --2- 172.21.15.70:0/1008605574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca40cea80 0x7efca4154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 >> 172.21.15.70:0/1008605574 conn(0x7efca406c9b0 msgr2=0x7efca4156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 shutdown_connections 2024-04-22T01:18:08.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.132+0000 7efca9e7f640 1 -- 172.21.15.70:0/1008605574 wait complete. 2024-04-22T01:18:08.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:08.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 Processor -- start 2024-04-22T01:18:08.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 -- start start 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab015dfb0 0x7f3ab00b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3ab00cea80 0x7f3ab00ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ab00b5730 0x7f3ab00b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3ab00cf420 con 0x7f3ab00cea80 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3ab015e390 con 0x7f3ab00b5730 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3ab55f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab0161e00 con 0x7f3ab015dfb0 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3ab00cea80 0x7f3ab00ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3ab00cea80 0x7f3ab00ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59564/0 (socket says 172.21.15.70:59564) 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 -- 172.21.15.70:0/3709375453 learned_addr learned my addr 172.21.15.70:0/3709375453 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:08.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aaf7fe640 1 --2- 172.21.15.70:0/3709375453 >> v2:172.21.15.98:3300/0 conn(0x7f3ab00b5730 0x7f3ab00b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aaeffd640 1 --2- 172.21.15.70:0/3709375453 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab015dfb0 0x7f3ab00b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 -- 172.21.15.70:0/3709375453 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab015dfb0 msgr2=0x7f3ab00b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:08.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 --2- 172.21.15.70:0/3709375453 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab015dfb0 0x7f3ab00b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 -- 172.21.15.70:0/3709375453 >> v2:172.21.15.98:3300/0 conn(0x7f3ab00b5730 msgr2=0x7f3ab00b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:08.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 --2- 172.21.15.70:0/3709375453 >> v2:172.21.15.98:3300/0 conn(0x7f3ab00b5730 0x7f3ab00b5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.256+0000 7f3aae7fc640 1 -- 172.21.15.70:0/3709375453 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab01620e0 con 0x7f3ab00cea80 2024-04-22T01:18:08.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 --2- 172.21.15.70:0/3709375453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab00ceee0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00b6880 tx=0x7f3aa001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5e38bd60461f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aadffb640 1 -- 172.21.15.70:0/3709375453 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00a5780 con 0x7f3ab00cea80 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aadffb640 1 -- 172.21.15.70:0/3709375453 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa00a5920 con 0x7f3ab00cea80 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aadffb640 1 -- 172.21.15.70:0/3709375453 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00a5cf0 con 0x7f3ab00cea80 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- 172.21.15.70:0/3709375453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 msgr2=0x7f3ab00ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/3709375453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab00ceee0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00b6880 tx=0x7f3aa001c9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- 172.21.15.70:0/3709375453 shutdown_connections 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/3709375453 >> v2:172.21.15.98:3300/0 conn(0x7f3ab00b5730 0x7f3ab00b5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/3709375453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab00ceee0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/3709375453 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab015dfb0 0x7f3ab00b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- 172.21.15.70:0/3709375453 >> 172.21.15.70:0/3709375453 conn(0x7f3ab006c9b0 msgr2=0x7f3ab006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- 172.21.15.70:0/3709375453 shutdown_connections 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- 172.21.15.70:0/3709375453 wait complete. 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 Processor -- start 2024-04-22T01:18:08.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- start start 2024-04-22T01:18:08.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab00b5730 0x7f3ab0157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab01575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab015dfb0 0x7f3ab01533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0161a90 con 0x7f3ab00cea80 2024-04-22T01:18:08.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00b9b10 con 0x7f3ab015dfb0 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3ab55f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab00d2900 con 0x7f3ab00b5730 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab01575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab01575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59578/0 (socket says 172.21.15.70:59578) 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 -- 172.21.15.70:0/2757349774 learned_addr learned my addr 172.21.15.70:0/2757349774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aaf7fe640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab015dfb0 0x7f3ab01533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aaeffd640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab00b5730 0x7f3ab0157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 -- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab00b5730 msgr2=0x7f3ab0157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab00b5730 0x7f3ab0157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 -- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab015dfb0 msgr2=0x7f3ab01533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab015dfb0 0x7f3ab01533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aae7fc640 1 -- 172.21.15.70:0/2757349774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab0156170 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.260+0000 7f3aaf7fe640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab015dfb0 0x7f3ab01533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aae7fc640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab01575d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f3aa001d290 tx=0x7f3aa0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00bc020 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa00b8670 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00b8a40 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab0156450 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab015b7e0 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab0068210 con 0x7f3ab00cea80 2024-04-22T01:18:08.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3aa00bf020 con 0x7f3ab00cea80 2024-04-22T01:18:08.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aa77fe640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3a900678c0 0x7f3a90069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f3aa0120e40 con 0x7f3ab00cea80 2024-04-22T01:18:08.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aaeffd640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3a900678c0 0x7f3a90069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.264+0000 7f3aaeffd640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3a900678c0 0x7f3a90069d80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ab015a310 tx=0x7f3aa8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.268+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3aa00c6030 con 0x7f3ab00cea80 2024-04-22T01:18:08.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.416+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ab00d1950 con 0x7f3ab00cea80 2024-04-22T01:18:08.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3aa77fe640 1 -- 172.21.15.70:0/2757349774 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7f3aa00165e0 con 0x7f3ab00cea80 2024-04-22T01:18:08.421 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:08.422 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3a900678c0 msgr2=0x7f3a90069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3a900678c0 0x7f3a90069d80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ab015a310 tx=0x7f3aa8008040 comp rx=0 tx=0).stop 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 msgr2=0x7f3ab01575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab01575d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f3aa001d290 tx=0x7f3aa0019790 comp rx=0 tx=0).stop 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 shutdown_connections 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3a900678c0 0x7f3a90069d80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab015dfb0 0x7f3ab01533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab00cea80 0x7f3ab01575d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 --2- 172.21.15.70:0/2757349774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ab00b5730 0x7f3ab0157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 >> 172.21.15.70:0/2757349774 conn(0x7f3ab006c9b0 msgr2=0x7f3ab00b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 shutdown_connections 2024-04-22T01:18:08.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.420+0000 7f3ab55f7640 1 -- 172.21.15.70:0/2757349774 wait complete. 2024-04-22T01:18:08.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:17:39.433363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:17:40.173738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:08.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.560+0000 7f8f01365640 1 Processor -- start 2024-04-22T01:18:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.560+0000 7f8f01365640 1 -- start start 2024-04-22T01:18:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8f01365640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8f01365640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8efc0cd050 0x7f8efc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8f01365640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc15c290 con 0x7f8efc0b4450 2024-04-22T01:18:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8f01365640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8efc15c570 con 0x7f8efc0cd050 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8f01365640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8efc15c850 con 0x7f8efc072280 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8efc0cd050 0x7f8efc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8efc0cd050 0x7f8efc0b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59590/0 (socket says 172.21.15.70:59590) 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 -- 172.21.15.70:0/1078302072 learned_addr learned my addr 172.21.15.70:0/1078302072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efa7fc640 1 --2- 172.21.15.70:0/1078302072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 -- 172.21.15.70:0/1078302072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 msgr2=0x7f8efc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 --2- 172.21.15.70:0/1078302072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 -- 172.21.15.70:0/1078302072 >> v1:172.21.15.98:6789/0 conn(0x7f8efc0b4450 legacy=0x7f8efc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 -- 172.21.15.70:0/1078302072 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc15cb30 con 0x7f8efc0cd050 2024-04-22T01:18:08.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.564+0000 7f8efb7fe640 1 --2- 172.21.15.70:0/1078302072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc0b2a40 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009e50 tx=0x7f8ef401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c399bc7ab5e68409 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8ef9ffb640 1 -- 172.21.15.70:0/1078302072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40a5800 con 0x7f8efc0cd050 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8ef9ffb640 1 -- 172.21.15.70:0/1078302072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef40a59a0 con 0x7f8efc0cd050 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8ef9ffb640 1 -- 172.21.15.70:0/1078302072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40a5d70 con 0x7f8efc0cd050 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/1078302072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 msgr2=0x7f8efc0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 --2- 172.21.15.70:0/1078302072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc0b2a40 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009e50 tx=0x7f8ef401c960 comp rx=0 tx=0).stop 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/1078302072 shutdown_connections 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 --2- 172.21.15.70:0/1078302072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc0b2a40 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 --2- 172.21.15.70:0/1078302072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/1078302072 >> 172.21.15.70:0/1078302072 conn(0x7f8efc0b1d40 msgr2=0x7f8efc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/1078302072 shutdown_connections 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/1078302072 wait complete. 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 Processor -- start 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- start start 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc1661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc0b4450 0x7f8efc15f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc15fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc0cd980 con 0x7f8efc0cd050 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc0cd800 con 0x7f8efc0b4450 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8efc0b3790 con 0x7f8efc072280 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc15fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc15fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59602/0 (socket says 172.21.15.70:59602) 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 -- 172.21.15.70:0/2577738123 learned_addr learned my addr 172.21.15.70:0/2577738123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efaffd640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc1661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efa7fc640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc0b4450 0x7f8efc15f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 -- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 msgr2=0x7f8efc1661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc1661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 -- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc0b4450 msgr2=0x7f8efc15f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc0b4450 0x7f8efc15f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 -- 172.21.15.70:0/2577738123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc06f450 con 0x7f8efc0cd050 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8efb7fe640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc15fc10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40af040 tx=0x7f8ef4019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40bc020 con 0x7f8efc0cd050 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef40b8670 con 0x7f8efc0cd050 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8efc06f730 con 0x7f8efc0cd050 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40b8a80 con 0x7f8efc0cd050 2024-04-22T01:18:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.568+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8efc166720 con 0x7f8efc0cd050 2024-04-22T01:18:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.572+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8efc068210 con 0x7f8efc0cd050 2024-04-22T01:18:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.572+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ef40bf020 con 0x7f8efc0cd050 2024-04-22T01:18:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.572+0000 7f8ee37fe640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ee80678c0 0x7f8ee8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.572+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f8ef4120ea0 con 0x7f8efc0cd050 2024-04-22T01:18:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.572+0000 7f8efaffd640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ee80678c0 0x7f8ee8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.572+0000 7f8efaffd640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ee80678c0 0x7f8ee8069d80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fd0 tx=0x7f8eec007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.576+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8ef40c6030 con 0x7f8efc0cd050 2024-04-22T01:18:08.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.740+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8efc0d01e0 con 0x7f8efc0cd050 2024-04-22T01:18:08.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.740+0000 7f8ee37fe640 1 -- 172.21.15.70:0/2577738123 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f8ef40165e0 con 0x7f8efc0cd050 2024-04-22T01:18:08.745 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ee80678c0 msgr2=0x7f8ee8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ee80678c0 0x7f8ee8069d80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fd0 tx=0x7f8eec007e40 comp rx=0 tx=0).stop 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 msgr2=0x7f8efc15fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc15fc10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40af040 tx=0x7f8ef4019700 comp rx=0 tx=0).stop 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 shutdown_connections 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ee80678c0 0x7f8ee8069d80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8efc0cd050 0x7f8efc15fc10 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc0b4450 0x7f8efc15f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 --2- 172.21.15.70:0/2577738123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc072280 0x7f8efc1661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 >> 172.21.15.70:0/2577738123 conn(0x7f8efc0b1d40 msgr2=0x7f8efc0cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:08.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 shutdown_connections 2024-04-22T01:18:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.744+0000 7f8f01365640 1 -- 172.21.15.70:0/2577738123 wait complete. 2024-04-22T01:18:08.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:18:08.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 Processor -- start 2024-04-22T01:18:08.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 -- start start 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa86815de20 0x7fa8680b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8680cea10 0x7fa8680cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8680b75e0 con 0x7fa8680b6db0 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa8680b78c0 con 0x7fa86815de20 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa86db84640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa8680b7ba0 con 0x7fa8680cea10 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa8677fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa86815de20 0x7fa8680b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa866ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa8680b6db0 0x7fa8680d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56786/0 (socket says 172.21.15.70:56786) 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa866ffd640 1 -- 172.21.15.70:0/2081505640 learned_addr learned my addr 172.21.15.70:0/2081505640 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa867fff640 1 --2- 172.21.15.70:0/2081505640 >> v2:172.21.15.98:3300/0 conn(0x7fa8680cea10 0x7fa8680cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa8677fe640 1 -- 172.21.15.70:0/2081505640 >> v2:172.21.15.98:3300/0 conn(0x7fa8680cea10 msgr2=0x7fa8680cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:08.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa8677fe640 1 --2- 172.21.15.70:0/2081505640 >> v2:172.21.15.98:3300/0 conn(0x7fa8680cea10 0x7fa8680cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa8677fe640 1 -- 172.21.15.70:0/2081505640 >> v1:172.21.15.70:6789/0 conn(0x7fa8680b6db0 legacy=0x7fa8680d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa8677fe640 1 -- 172.21.15.70:0/2081505640 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8680b7e80 con 0x7fa86815de20 2024-04-22T01:18:08.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.876+0000 7fa8667fc640 1 -- 172.21.15.70:0/2081505640 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510495945 0 0) 0x7fa8680b75e0 con 0x7fa8680b6db0 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 --2- 172.21.15.70:0/2081505640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86815de20 0x7fa8680b98a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa8680b6d00 tx=0x7fa85801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2974b10eb2564783 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8667fc640 1 -- 172.21.15.70:0/2081505640 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8580ab430 con 0x7fa86815de20 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8667fc640 1 -- 172.21.15.70:0/2081505640 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8580b0070 con 0x7fa86815de20 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8667fc640 1 -- 172.21.15.70:0/2081505640 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8580ab7f0 con 0x7fa86815de20 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/2081505640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86815de20 msgr2=0x7fa8680b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 --2- 172.21.15.70:0/2081505640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86815de20 0x7fa8680b98a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa8680b6d00 tx=0x7fa85801c450 comp rx=0 tx=0).stop 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/2081505640 shutdown_connections 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 --2- 172.21.15.70:0/2081505640 >> v2:172.21.15.98:3300/0 conn(0x7fa8680cea10 0x7fa8680cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 --2- 172.21.15.70:0/2081505640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86815de20 0x7fa8680b98a0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/2081505640 >> 172.21.15.70:0/2081505640 conn(0x7fa868072d20 msgr2=0x7fa868073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/2081505640 shutdown_connections 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/2081505640 wait complete. 2024-04-22T01:18:08.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 Processor -- start 2024-04-22T01:18:08.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- start start 2024-04-22T01:18:08.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 0x7fa868159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8680cea10 0x7fa86814cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa86815de20 0x7fa868157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8680b7330 con 0x7fa8680b6db0 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8680b71b0 con 0x7fa8680cea10 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa8680b8e90 con 0x7fa86815de20 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 0x7fa868159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 0x7fa868159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59626/0 (socket says 172.21.15.70:59626) 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 -- 172.21.15.70:0/540668098 learned_addr learned my addr 172.21.15.70:0/540668098 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa866ffd640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8680cea10 0x7fa86814cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 -- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa86815de20 msgr2=0x7fa868157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa86815de20 0x7fa868157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 -- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8680cea10 msgr2=0x7fa86814cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8680cea10 0x7fa86814cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 -- 172.21.15.70:0/540668098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8681560f0 con 0x7fa8680b6db0 2024-04-22T01:18:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa8677fe640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 0x7fa868159760 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa858019060 tx=0x7fa85801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858004230 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8580043d0 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8681563d0 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858003820 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.880+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa868160480 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.884+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa86815ffd0 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.884+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8580039c0 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.884+0000 7fa84ffff640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8540678c0 0x7fa854069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.884+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fa8580a8030 con 0x7fa8680b6db0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.884+0000 7fa866ffd640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8540678c0 0x7fa854069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:08.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.884+0000 7fa866ffd640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8540678c0 0x7fa854069d80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa86806c9c0 tx=0x7fa85c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:08.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:08.888+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa8580c4060 con 0x7fa8680b6db0 2024-04-22T01:18:09.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.036+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8680ce6c0 con 0x7fa8680b6db0 2024-04-22T01:18:09.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.036+0000 7fa84ffff640 1 -- 172.21.15.70:0/540668098 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7fa858016070 con 0x7fa8680b6db0 2024-04-22T01:18:09.040 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8540678c0 msgr2=0x7fa854069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8540678c0 0x7fa854069d80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa86806c9c0 tx=0x7fa85c009040 comp rx=0 tx=0).stop 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 msgr2=0x7fa868159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 0x7fa868159760 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa858019060 tx=0x7fa85801ac90 comp rx=0 tx=0).stop 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 shutdown_connections 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8540678c0 0x7fa854069d80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa86815de20 0x7fa868157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8680cea10 0x7fa86814cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 --2- 172.21.15.70:0/540668098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8680b6db0 0x7fa868159760 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 >> 172.21.15.70:0/540668098 conn(0x7fa868072d20 msgr2=0x7fa868157280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 shutdown_connections 2024-04-22T01:18:09.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.040+0000 7fa86db84640 1 -- 172.21.15.70:0/540668098 wait complete. 2024-04-22T01:18:09.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:18:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 Processor -- start 2024-04-22T01:18:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 -- start start 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f881015e1e0 0x7f88100b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8810070670 0x7f8810070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f88100d0cc0 0x7f88100d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8810071010 con 0x7f88100d0cc0 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f881015e5c0 con 0x7f881015e1e0 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f88185ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88100ada50 con 0x7f8810070670 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f8816b87640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f88100d0cc0 0x7f88100d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f8816b87640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f88100d0cc0 0x7f88100d1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59636/0 (socket says 172.21.15.70:59636) 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.168+0000 7f8816b87640 1 -- 172.21.15.70:0/884686060 learned_addr learned my addr 172.21.15.70:0/884686060 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816386640 1 --2- 172.21.15.70:0/884686060 >> v2:172.21.15.98:3300/0 conn(0x7f881015e1e0 0x7f88100b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8815b85640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8810070670 0x7f8810070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816b87640 1 -- 172.21.15.70:0/884686060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8810070670 msgr2=0x7f8810070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816b87640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8810070670 0x7f8810070ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816b87640 1 -- 172.21.15.70:0/884686060 >> v2:172.21.15.98:3300/0 conn(0x7f881015e1e0 msgr2=0x7f88100b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816b87640 1 --2- 172.21.15.70:0/884686060 >> v2:172.21.15.98:3300/0 conn(0x7f881015e1e0 0x7f88100b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816b87640 1 -- 172.21.15.70:0/884686060 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88100adca0 con 0x7f88100d0cc0 2024-04-22T01:18:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8815b85640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8810070670 0x7f8810070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8816b87640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88100d0cc0 0x7f88100d1150 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f88100d19b0 tx=0x7f880801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8171d8ceadb724 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8815384640 1 -- 172.21.15.70:0/884686060 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88080a5760 con 0x7f88100d0cc0 2024-04-22T01:18:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8815384640 1 -- 172.21.15.70:0/884686060 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88080a5900 con 0x7f88100d0cc0 2024-04-22T01:18:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f8815384640 1 -- 172.21.15.70:0/884686060 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88080a5cd0 con 0x7f88100d0cc0 2024-04-22T01:18:09.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 -- 172.21.15.70:0/884686060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88100d0cc0 msgr2=0x7f88100d1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88100d0cc0 0x7f88100d1150 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f88100d19b0 tx=0x7f880801c9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 -- 172.21.15.70:0/884686060 shutdown_connections 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88100d0cc0 0x7f88100d1150 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 --2- 172.21.15.70:0/884686060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8810070670 0x7f8810070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 --2- 172.21.15.70:0/884686060 >> v2:172.21.15.98:3300/0 conn(0x7f881015e1e0 0x7f88100b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 -- 172.21.15.70:0/884686060 >> 172.21.15.70:0/884686060 conn(0x7f88100733e0 msgr2=0x7f8810099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 -- 172.21.15.70:0/884686060 shutdown_connections 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.172+0000 7f88185ea640 1 -- 172.21.15.70:0/884686060 wait complete. 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 Processor -- start 2024-04-22T01:18:09.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- start start 2024-04-22T01:18:09.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 0x7f88100b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88100d0cc0 0x7f88101552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f881015e1e0 0x7f881014fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88100af550 con 0x7f8810070670 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88100ae820 con 0x7f881015e1e0 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88100af6d0 con 0x7f88100d0cc0 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 0x7f88100b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 0x7f88100b5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59644/0 (socket says 172.21.15.70:59644) 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 -- 172.21.15.70:0/1487260605 learned_addr learned my addr 172.21.15.70:0/1487260605 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816b87640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f881015e1e0 0x7f881014fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8815b85640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88100d0cc0 0x7f88101552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 -- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88100d0cc0 msgr2=0x7f88101552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88100d0cc0 0x7f88101552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 -- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f881015e1e0 msgr2=0x7f881014fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f881015e1e0 0x7f881014fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 -- 172.21.15.70:0/1487260605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88100b45a0 con 0x7f8810070670 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8816386640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 0x7f88100b5a80 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f880c00d620 tx=0x7f880c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c00b620 con 0x7f8810070670 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c019070 con 0x7f8810070670 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88100b4880 con 0x7f8810070670 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c00b9e0 con 0x7f8810070670 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88100b4e40 con 0x7f8810070670 2024-04-22T01:18:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8810068920 con 0x7f8810070670 2024-04-22T01:18:09.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.176+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f880c00bb80 con 0x7f8810070670 2024-04-22T01:18:09.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.180+0000 7f8806ffd640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87f40678c0 0x7f87f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.180+0000 7f8815b85640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87f40678c0 0x7f87f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.180+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f880c086140 con 0x7f8810070670 2024-04-22T01:18:09.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.180+0000 7f8815b85640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87f40678c0 0x7f87f4069d80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8800006fd0 tx=0x7f8800008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.184+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f880c056b80 con 0x7f8810070670 2024-04-22T01:18:09.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.328+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f88100d1350 con 0x7f8810070670 2024-04-22T01:18:09.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.328+0000 7f8806ffd640 1 -- 172.21.15.70:0/1487260605 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f880c056480 con 0x7f8810070670 2024-04-22T01:18:09.331 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87f40678c0 msgr2=0x7f87f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87f40678c0 0x7f87f4069d80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8800006fd0 tx=0x7f8800008040 comp rx=0 tx=0).stop 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 msgr2=0x7f88100b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 0x7f88100b5a80 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f880c00d620 tx=0x7f880c00dae0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 shutdown_connections 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f87f40678c0 0x7f87f4069d80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f881015e1e0 0x7f881014fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88100d0cc0 0x7f88101552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 --2- 172.21.15.70:0/1487260605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810070670 0x7f88100b5a80 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 >> 172.21.15.70:0/1487260605 conn(0x7f88100733e0 msgr2=0x7f881006b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 shutdown_connections 2024-04-22T01:18:09.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.332+0000 7f88185ea640 1 -- 172.21.15.70:0/1487260605 wait complete. 2024-04-22T01:18:09.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:09.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 Processor -- start 2024-04-22T01:18:09.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 -- start start 2024-04-22T01:18:09.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015aa30 0x7f93d00b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f93d015bb50 0x7f93d00ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d006dbd0 con 0x7f93d00aec70 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f93d00af0a0 con 0x7f93d015bb50 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d6fc0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f93d006de70 con 0x7f93d015aa30 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f93d015bb50 0x7f93d00ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f93d015bb50 0x7f93d00ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59658/0 (socket says 172.21.15.70:59658) 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 -- 172.21.15.70:0/1237490666 learned_addr learned my addr 172.21.15.70:0/1237490666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d4d5c640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015aa30 0x7f93d00b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 -- 172.21.15.70:0/1237490666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015aa30 msgr2=0x7f93d00b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015aa30 0x7f93d00b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 -- 172.21.15.70:0/1237490666 >> v1:172.21.15.98:6789/0 conn(0x7f93d00aec70 legacy=0x7f93d015e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93cffff640 1 -- 172.21.15.70:0/1237490666 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d006e150 con 0x7f93d015bb50 2024-04-22T01:18:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.460+0000 7f93d4d5c640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015aa30 0x7f93d00b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93cffff640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d015bb50 0x7f93d00ae730 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009870 tx=0x7f93bc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbac50922d0db3ab server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93cf7fe640 1 -- 172.21.15.70:0/1237490666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0b1070 con 0x7f93d015bb50 2024-04-22T01:18:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93cf7fe640 1 -- 172.21.15.70:0/1237490666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93bc0ac640 con 0x7f93d015bb50 2024-04-22T01:18:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93cf7fe640 1 -- 172.21.15.70:0/1237490666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0aca10 con 0x7f93d015bb50 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- 172.21.15.70:0/1237490666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d015bb50 msgr2=0x7f93d00ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d015bb50 0x7f93d00ae730 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009870 tx=0x7f93bc01c520 comp rx=0 tx=0).stop 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- 172.21.15.70:0/1237490666 shutdown_connections 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d015bb50 0x7f93d00ae730 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/1237490666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015aa30 0x7f93d00b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- 172.21.15.70:0/1237490666 >> 172.21.15.70:0/1237490666 conn(0x7f93d006b1f0 msgr2=0x7f93d006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- 172.21.15.70:0/1237490666 shutdown_connections 2024-04-22T01:18:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- 172.21.15.70:0/1237490666 wait complete. 2024-04-22T01:18:09.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 Processor -- start 2024-04-22T01:18:09.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- start start 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 0x7f93d014cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d015aa30 0x7f93d0164640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015bb50 0x7f93d0150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93d015af90 con 0x7f93d00aec70 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93d015ae10 con 0x7f93d015aa30 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d6fc0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f93d006e950 con 0x7f93d015bb50 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d4d5c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 0x7f93d014cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d4d5c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 0x7f93d014cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59660/0 (socket says 172.21.15.70:59660) 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.464+0000 7f93d4d5c640 1 -- 172.21.15.70:0/3903699525 learned_addr learned my addr 172.21.15.70:0/3903699525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d555d640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015bb50 0x7f93d0150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d4d5c640 1 -- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015bb50 msgr2=0x7f93d0150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cffff640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d015aa30 0x7f93d0164640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d4d5c640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015bb50 0x7f93d0150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d4d5c640 1 -- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d015aa30 msgr2=0x7f93d0164640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d4d5c640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d015aa30 0x7f93d0164640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d4d5c640 1 -- 172.21.15.70:0/3903699525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d0164b80 con 0x7f93d00aec70 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d4d5c640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 0x7f93d014cc70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f93c400c7f0 tx=0x7f93c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4015070 con 0x7f93d00aec70 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c4007820 con 0x7f93d00aec70 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4007c10 con 0x7f93d00aec70 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d01651f0 con 0x7f93d00aec70 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d0150150 con 0x7f93d00aec70 2024-04-22T01:18:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d0165710 con 0x7f93d00aec70 2024-04-22T01:18:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f93c4018070 con 0x7f93d00aec70 2024-04-22T01:18:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cd7fa640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f93b40678c0 0x7f93b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f93c4085ec0 con 0x7f93d00aec70 2024-04-22T01:18:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cffff640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f93b40678c0 0x7f93b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.468+0000 7f93cffff640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f93b40678c0 0x7f93b4069d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0ae000 tx=0x7f93bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.472+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f93d0165710 con 0x7f93d00aec70 2024-04-22T01:18:09.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.652+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93d0068210 con 0x7f93d00aec70 2024-04-22T01:18:09.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.652+0000 7f93cd7fa640 1 -- 172.21.15.70:0/3903699525 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5174 (secure 0 0 0) 0x7f93d0164b80 con 0x7f93d00aec70 2024-04-22T01:18:09.656 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:09.656 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:41.174335+0000","modified":"2024-04-22T01:17:43.233600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6759":{"gid":6759,"name":"d","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6833/258015518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":258015518},{"type":"v1","addr":"172.21.15.98:6833","nonce":258015518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6759,"qdb_cluster":[6759]},"id":5}]} 2024-04-22T01:18:09.656 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 37 2024-04-22T01:18:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f93b40678c0 msgr2=0x7f93b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f93b40678c0 0x7f93b4069d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0ae000 tx=0x7f93bc008040 comp rx=0 tx=0).stop 2024-04-22T01:18:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 msgr2=0x7f93d014cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 0x7f93d014cc70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f93c400c7f0 tx=0x7f93c400ccb0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 shutdown_connections 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f93b40678c0 0x7f93b4069d80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93d015bb50 0x7f93d0150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d015aa30 0x7f93d0164640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 --2- 172.21.15.70:0/3903699525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93d00aec70 0x7f93d014cc70 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 >> 172.21.15.70:0/3903699525 conn(0x7f93d006b1f0 msgr2=0x7f93d006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 shutdown_connections 2024-04-22T01:18:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.656+0000 7f93d6fc0640 1 -- 172.21.15.70:0/3903699525 wait complete. 2024-04-22T01:18:09.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:09.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.776+0000 7f607e027640 1 Processor -- start 2024-04-22T01:18:09.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.776+0000 7f607e027640 1 -- start start 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f607e027640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f607815aa30 0x7f60780b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f607e027640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60780aee40 0x7f60780b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f607e027640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607806dbd0 con 0x7f607815bb50 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f607e027640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f607806de70 con 0x7f60780aee40 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f607e027640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f607806e150 con 0x7f607815aa30 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60780aee40 0x7f60780b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60780aee40 0x7f60780b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59668/0 (socket says 172.21.15.70:59668) 2024-04-22T01:18:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 -- 172.21.15.70:0/21328015 learned_addr learned my addr 172.21.15.70:0/21328015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:09.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f60777fe640 1 --2- 172.21.15.70:0/21328015 >> v2:172.21.15.98:3300/0 conn(0x7f607815aa30 0x7f60780b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 -- 172.21.15.70:0/21328015 >> v2:172.21.15.98:3300/0 conn(0x7f607815aa30 msgr2=0x7f60780b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:09.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 --2- 172.21.15.70:0/21328015 >> v2:172.21.15.98:3300/0 conn(0x7f607815aa30 0x7f60780b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 -- 172.21.15.70:0/21328015 >> v1:172.21.15.98:6789/0 conn(0x7f607815bb50 legacy=0x7f60780ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:09.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.780+0000 7f6077fff640 1 -- 172.21.15.70:0/21328015 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607806e430 con 0x7f60780aee40 2024-04-22T01:18:09.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f6077fff640 1 --2- 172.21.15.70:0/21328015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60780aee40 0x7f60780b28f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f60780b4680 tx=0x7f607001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219fb2b1f09b2198 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f60767fc640 1 -- 172.21.15.70:0/21328015 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700a5840 con 0x7f60780aee40 2024-04-22T01:18:09.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f60767fc640 1 -- 172.21.15.70:0/21328015 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60700a59e0 con 0x7f60780aee40 2024-04-22T01:18:09.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f60767fc640 1 -- 172.21.15.70:0/21328015 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700a5db0 con 0x7f60780aee40 2024-04-22T01:18:09.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 -- 172.21.15.70:0/21328015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60780aee40 msgr2=0x7f60780b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 --2- 172.21.15.70:0/21328015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60780aee40 0x7f60780b28f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f60780b4680 tx=0x7f607001c960 comp rx=0 tx=0).stop 2024-04-22T01:18:09.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 -- 172.21.15.70:0/21328015 shutdown_connections 2024-04-22T01:18:09.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 --2- 172.21.15.70:0/21328015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60780aee40 0x7f60780b28f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 --2- 172.21.15.70:0/21328015 >> v2:172.21.15.98:3300/0 conn(0x7f607815aa30 0x7f60780b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 -- 172.21.15.70:0/21328015 >> 172.21.15.70:0/21328015 conn(0x7f607806b1f0 msgr2=0x7f607806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 -- 172.21.15.70:0/21328015 shutdown_connections 2024-04-22T01:18:09.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.784+0000 7f607e027640 1 -- 172.21.15.70:0/21328015 wait complete. 2024-04-22T01:18:09.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 Processor -- start 2024-04-22T01:18:09.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- start start 2024-04-22T01:18:09.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60780aee40 0x7f607814f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 0x7f60781422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607815bb50 0x7f60781513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f607806ead0 con 0x7f607815aa30 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f607806e950 con 0x7f607815bb50 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6078167a90 con 0x7f60780aee40 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 0x7f60781422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 0x7f60781422f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59676/0 (socket says 172.21.15.70:59676) 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 -- 172.21.15.70:0/1767349651 learned_addr learned my addr 172.21.15.70:0/1767349651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f60777fe640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60780aee40 0x7f607814f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6077fff640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607815bb50 0x7f60781513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 -- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60780aee40 msgr2=0x7f607814f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60780aee40 0x7f607814f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 -- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607815bb50 msgr2=0x7f60781513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607815bb50 0x7f60781513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 -- 172.21.15.70:0/1767349651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6078142830 con 0x7f607815aa30 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6077fff640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607815bb50 0x7f60781513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f6076ffd640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 0x7f60781422f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f606c00c450 tx=0x7f606c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c012830 con 0x7f607815aa30 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f606c017070 con 0x7f607815aa30 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60780cdd40 con 0x7f607815aa30 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c012c30 con 0x7f607815aa30 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60780ce300 con 0x7f607815aa30 2024-04-22T01:18:09.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6078068210 con 0x7f607815aa30 2024-04-22T01:18:09.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.788+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f606c01c460 con 0x7f607815aa30 2024-04-22T01:18:09.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.792+0000 7f605ffff640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60640678c0 0x7f6064069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:09.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.792+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f606c085b00 con 0x7f607815aa30 2024-04-22T01:18:09.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.792+0000 7f60777fe640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60640678c0 0x7f6064069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:09.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.792+0000 7f60777fe640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60640678c0 0x7f6064069d80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f60680096f0 tx=0x7f6068008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:09.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.796+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f606c056540 con 0x7f607815aa30 2024-04-22T01:18:09.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.952+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f607806f190 con 0x7f607815aa30 2024-04-22T01:18:09.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.952+0000 7f605ffff640 1 -- 172.21.15.70:0/1767349651 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7f606c055e40 con 0x7f607815aa30 2024-04-22T01:18:09.956 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:09.956 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60640678c0 msgr2=0x7f6064069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60640678c0 0x7f6064069d80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f60680096f0 tx=0x7f6068008040 comp rx=0 tx=0).stop 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 msgr2=0x7f60781422f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 0x7f60781422f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f606c00c450 tx=0x7f606c00c910 comp rx=0 tx=0).stop 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 shutdown_connections 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60640678c0 0x7f6064069d80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607815bb50 0x7f60781513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f607815aa30 0x7f60781422f0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 --2- 172.21.15.70:0/1767349651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60780aee40 0x7f607814f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 >> 172.21.15.70:0/1767349651 conn(0x7f607806b1f0 msgr2=0x7f607806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 shutdown_connections 2024-04-22T01:18:09.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:09.956+0000 7f607e027640 1 -- 172.21.15.70:0/1767349651 wait complete. 2024-04-22T01:18:09.974 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:18:09.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:10.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 Processor -- start 2024-04-22T01:18:10.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 -- start start 2024-04-22T01:18:10.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92280cc150 0x7f9228071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9228072060 0x7f922806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92280cd7b0 con 0x7f92281455b0 2024-04-22T01:18:10.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9228072490 con 0x7f92280cc150 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922faa0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f92280cc710 con 0x7f9228072060 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92280cc150 0x7f9228071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92280cc150 0x7f9228071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59680/0 (socket says 172.21.15.70:59680) 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 -- 172.21.15.70:0/738468392 learned_addr learned my addr 172.21.15.70:0/738468392 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922e03d640 1 --2- 172.21.15.70:0/738468392 >> v2:172.21.15.98:3300/0 conn(0x7f9228072060 0x7f922806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 -- 172.21.15.70:0/738468392 >> v2:172.21.15.98:3300/0 conn(0x7f9228072060 msgr2=0x7f922806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 --2- 172.21.15.70:0/738468392 >> v2:172.21.15.98:3300/0 conn(0x7f9228072060 0x7f922806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 -- 172.21.15.70:0/738468392 >> v1:172.21.15.98:6789/0 conn(0x7f92281455b0 legacy=0x7f92280cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:10.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 -- 172.21.15.70:0/738468392 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92280cc9f0 con 0x7f92280cc150 2024-04-22T01:18:10.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.092+0000 7f922d03b640 1 --2- 172.21.15.70:0/738468392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92280cc150 0x7f9228071b20 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f922806ffb0 tx=0x7f921c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953aed09ca570782 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922c83a640 1 -- 172.21.15.70:0/738468392 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c0a5550 con 0x7f92280cc150 2024-04-22T01:18:10.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922c83a640 1 -- 172.21.15.70:0/738468392 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f921c0a56f0 con 0x7f92280cc150 2024-04-22T01:18:10.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922c83a640 1 -- 172.21.15.70:0/738468392 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c0a5ac0 con 0x7f92280cc150 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/738468392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92280cc150 msgr2=0x7f9228071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 --2- 172.21.15.70:0/738468392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92280cc150 0x7f9228071b20 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f922806ffb0 tx=0x7f921c01c890 comp rx=0 tx=0).stop 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/738468392 shutdown_connections 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 --2- 172.21.15.70:0/738468392 >> v2:172.21.15.98:3300/0 conn(0x7f9228072060 0x7f922806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 --2- 172.21.15.70:0/738468392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92280cc150 0x7f9228071b20 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/738468392 >> 172.21.15.70:0/738468392 conn(0x7f922806b1f0 msgr2=0x7f922806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/738468392 shutdown_connections 2024-04-22T01:18:10.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/738468392 wait complete. 2024-04-22T01:18:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 Processor -- start 2024-04-22T01:18:10.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- start start 2024-04-22T01:18:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 0x7f922815eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92280cc150 0x7f92280b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92281455b0 0x7f922813c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f922806f2e0 con 0x7f9228072060 2024-04-22T01:18:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f922806f160 con 0x7f92280cc150 2024-04-22T01:18:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9228168b40 con 0x7f92281455b0 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 0x7f922815eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 0x7f922815eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59682/0 (socket says 172.21.15.70:59682) 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 -- 172.21.15.70:0/3724544463 learned_addr learned my addr 172.21.15.70:0/3724544463 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 -- 172.21.15.70:0/3724544463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92281455b0 msgr2=0x7f922813c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92281455b0 0x7f922813c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 -- 172.21.15.70:0/3724544463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92280cc150 msgr2=0x7f92280b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92280cc150 0x7f92280b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 -- 172.21.15.70:0/3724544463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92280b3ca0 con 0x7f9228072060 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922d83c640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 0x7f922815eb00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f921800b440 tx=0x7f921800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218015020 con 0x7f9228072060 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9218011890 con 0x7f9228072060 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218011c80 con 0x7f9228072060 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9228141020 con 0x7f9228072060 2024-04-22T01:18:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.096+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f922813c6b0 con 0x7f9228072060 2024-04-22T01:18:10.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.100+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92280b3e70 con 0x7f9228072060 2024-04-22T01:18:10.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.100+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9218018020 con 0x7f9228072060 2024-04-22T01:18:10.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.100+0000 7f92267fc640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92140678c0 0x7f9214069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.100+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f9218085c60 con 0x7f9228072060 2024-04-22T01:18:10.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.100+0000 7f922d03b640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92140678c0 0x7f9214069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.100+0000 7f922d03b640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92140678c0 0x7f9214069d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f921c0ae000 tx=0x7f921c004ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.104+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f921801f030 con 0x7f9228072060 2024-04-22T01:18:10.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9228068210 con 0x7f9228072060 2024-04-22T01:18:10.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f92267fc640 1 -- 172.21.15.70:0/3724544463 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5174 (secure 0 0 0) 0x7f921800d040 con 0x7f9228072060 2024-04-22T01:18:10.287 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:10.287 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:41.174335+0000","modified":"2024-04-22T01:17:43.233600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6759":{"gid":6759,"name":"d","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6833/258015518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":258015518},{"type":"v1","addr":"172.21.15.98:6833","nonce":258015518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6759,"qdb_cluster":[6759]},"id":5}]} 2024-04-22T01:18:10.287 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 37 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92140678c0 msgr2=0x7f9214069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92140678c0 0x7f9214069d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f921c0ae000 tx=0x7f921c004ac0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 msgr2=0x7f922815eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 0x7f922815eb00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f921800b440 tx=0x7f921800b900 comp rx=0 tx=0).stop 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 shutdown_connections 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f92140678c0 0x7f9214069d80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92281455b0 0x7f922813c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92280cc150 0x7f92280b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 --2- 172.21.15.70:0/3724544463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9228072060 0x7f922815eb00 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 >> 172.21.15.70:0/3724544463 conn(0x7f922806b1f0 msgr2=0x7f922813f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:10.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 shutdown_connections 2024-04-22T01:18:10.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.284+0000 7f922faa0640 1 -- 172.21.15.70:0/3724544463 wait complete. 2024-04-22T01:18:10.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:10.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.404+0000 7f496df9e640 1 Processor -- start 2024-04-22T01:18:10.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.404+0000 7f496df9e640 1 -- start start 2024-04-22T01:18:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f496df9e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f496809a3d0 0x7f49680cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f496df9e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f49680c6ce0 0x7f49680cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f496df9e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49680cd890 con 0x7f496815f360 2024-04-22T01:18:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f496df9e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f49680c5840 con 0x7f49680c6ce0 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f496df9e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f49680cdb70 con 0x7f496809a3d0 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f49680c6ce0 0x7f49680cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f49680c6ce0 0x7f49680cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59688/0 (socket says 172.21.15.70:59688) 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 -- 172.21.15.70:0/44387994 learned_addr learned my addr 172.21.15.70:0/44387994 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4966ffd640 1 --2- 172.21.15.70:0/44387994 >> v2:172.21.15.98:3300/0 conn(0x7f496809a3d0 0x7f49680cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 -- 172.21.15.70:0/44387994 >> v2:172.21.15.98:3300/0 conn(0x7f496809a3d0 msgr2=0x7f49680cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 --2- 172.21.15.70:0/44387994 >> v2:172.21.15.98:3300/0 conn(0x7f496809a3d0 0x7f49680cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 -- 172.21.15.70:0/44387994 >> v1:172.21.15.70:6789/0 conn(0x7f496815f360 legacy=0x7f4968099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f49667fc640 1 -- 172.21.15.70:0/44387994 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727901389 0 0) 0x7f49680cd890 con 0x7f496815f360 2024-04-22T01:18:10.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 -- 172.21.15.70:0/44387994 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49680cde50 con 0x7f49680c6ce0 2024-04-22T01:18:10.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f4967fff640 1 --2- 172.21.15.70:0/44387994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f49680cf6b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4968073620 tx=0x7f496001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6142df995663ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f49667fc640 1 -- 172.21.15.70:0/44387994 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496815f950 con 0x7f49680c6ce0 2024-04-22T01:18:10.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f49667fc640 1 -- 172.21.15.70:0/44387994 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49600a58e0 con 0x7f49680c6ce0 2024-04-22T01:18:10.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.408+0000 7f49667fc640 1 -- 172.21.15.70:0/44387994 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49600a5cb0 con 0x7f49680c6ce0 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/44387994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 msgr2=0x7f49680cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 --2- 172.21.15.70:0/44387994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f49680cf6b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4968073620 tx=0x7f496001c9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/44387994 shutdown_connections 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 --2- 172.21.15.70:0/44387994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f49680cf6b0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 --2- 172.21.15.70:0/44387994 >> v2:172.21.15.98:3300/0 conn(0x7f496809a3d0 0x7f49680cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/44387994 >> 172.21.15.70:0/44387994 conn(0x7f49680b35d0 msgr2=0x7f49680b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/44387994 shutdown_connections 2024-04-22T01:18:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/44387994 wait complete. 2024-04-22T01:18:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 Processor -- start 2024-04-22T01:18:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- start start 2024-04-22T01:18:10.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f496809a3d0 0x7f4968158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f496809d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f496815f360 0x7f496809dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49680d0790 con 0x7f49680c6ce0 2024-04-22T01:18:10.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49680d0610 con 0x7f496815f360 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f49680d0910 con 0x7f496809a3d0 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f496809d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f496809d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59696/0 (socket says 172.21.15.70:59696) 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 -- 172.21.15.70:0/1996057480 learned_addr learned my addr 172.21.15.70:0/1996057480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4967fff640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f496815f360 0x7f496809dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f49677fe640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f496809a3d0 0x7f4968158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 -- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f496809a3d0 msgr2=0x7f4968158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f496809a3d0 0x7f4968158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 -- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f496815f360 msgr2=0x7f496809dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f496815f360 0x7f496809dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 -- 172.21.15.70:0/1996057480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496809e460 con 0x7f49680c6ce0 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f4966ffd640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f496809d690 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f495000c7f0 tx=0x7f495000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950015070 con 0x7f49680c6ce0 2024-04-22T01:18:10.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4950007820 con 0x7f49680c6ce0 2024-04-22T01:18:10.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496809e740 con 0x7f49680c6ce0 2024-04-22T01:18:10.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950007c10 con 0x7f49680c6ce0 2024-04-22T01:18:10.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496809ed00 con 0x7f49680c6ce0 2024-04-22T01:18:10.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.412+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49680c8c70 con 0x7f49680c6ce0 2024-04-22T01:18:10.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.416+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4950018070 con 0x7f49680c6ce0 2024-04-22T01:18:10.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.416+0000 7f495ffff640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49480678c0 0x7f4948069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.416+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f4950085e70 con 0x7f49680c6ce0 2024-04-22T01:18:10.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.416+0000 7f49677fe640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49480678c0 0x7f4948069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.416+0000 7f49677fe640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49480678c0 0x7f4948069d80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4958000f70 tx=0x7f4958009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.420+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f49680c8c70 con 0x7f49680c6ce0 2024-04-22T01:18:10.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.608+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f496809ef00 con 0x7f49680c6ce0 2024-04-22T01:18:10.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.608+0000 7f495ffff640 1 -- 172.21.15.70:0/1996057480 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5174 (secure 0 0 0) 0x7f49500568b0 con 0x7f49680c6ce0 2024-04-22T01:18:10.611 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:10.611 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:17:41.174335+0000","modified":"2024-04-22T01:17:43.233600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6759":{"gid":6759,"name":"d","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6833/258015518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":258015518},{"type":"v1","addr":"172.21.15.98:6833","nonce":258015518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6759,"qdb_cluster":[6759]},"id":5}]} 2024-04-22T01:18:10.611 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 37 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.608+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49480678c0 msgr2=0x7f4948069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.608+0000 7f496df9e640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49480678c0 0x7f4948069d80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4958000f70 tx=0x7f4958009040 comp rx=0 tx=0).stop 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.608+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 msgr2=0x7f496809d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.608+0000 7f496df9e640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f496809d690 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f495000c7f0 tx=0x7f495000ccb0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 shutdown_connections 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f49480678c0 0x7f4948069d80 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f496815f360 0x7f496809dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49680c6ce0 0x7f496809d690 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 --2- 172.21.15.70:0/1996057480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f496809a3d0 0x7f4968158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 >> 172.21.15.70:0/1996057480 conn(0x7f49680b35d0 msgr2=0x7f4968159210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:10.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 shutdown_connections 2024-04-22T01:18:10.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.612+0000 7f496df9e640 1 -- 172.21.15.70:0/1996057480 wait complete. 2024-04-22T01:18:10.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.732+0000 7fe54ec93640 1 Processor -- start 2024-04-22T01:18:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.732+0000 7fe54ec93640 1 -- start start 2024-04-22T01:18:10.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ec93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe54815a550 0x7fe5480aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ec93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe548071b20 0x7fe548071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ec93640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5480cf4a0 con 0x7fe5480aeae0 2024-04-22T01:18:10.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ec93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe54806f010 con 0x7fe548071b20 2024-04-22T01:18:10.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ec93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe54806f260 con 0x7fe54815a550 2024-04-22T01:18:10.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ca2f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe54815a550 0x7fe5480aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54d230640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe548071b20 0x7fe548071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe53ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe5480aeae0 0x7fe5480cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:36998/0 (socket says 172.21.15.70:36998) 2024-04-22T01:18:10.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe53ffff640 1 -- 172.21.15.70:0/1984826791 learned_addr learned my addr 172.21.15.70:0/1984826791 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:10.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe53f7fe640 1 -- 172.21.15.70:0/1984826791 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384238510 0 0) 0x7fe5480cf4a0 con 0x7fe5480aeae0 2024-04-22T01:18:10.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe53f7fe640 1 -- 172.21.15.70:0/1984826791 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe530003530 con 0x7fe5480aeae0 2024-04-22T01:18:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ca2f640 1 -- 172.21.15.70:0/1984826791 >> v2:172.21.15.98:3300/0 conn(0x7fe548071b20 msgr2=0x7fe548071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ca2f640 1 --2- 172.21.15.70:0/1984826791 >> v2:172.21.15.98:3300/0 conn(0x7fe548071b20 0x7fe548071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ca2f640 1 -- 172.21.15.70:0/1984826791 >> v1:172.21.15.98:6789/0 conn(0x7fe5480aeae0 legacy=0x7fe5480cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54ca2f640 1 -- 172.21.15.70:0/1984826791 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54806f540 con 0x7fe54815a550 2024-04-22T01:18:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.736+0000 7fe54d230640 1 --2- 172.21.15.70:0/1984826791 >> v2:172.21.15.98:3300/0 conn(0x7fe548071b20 0x7fe548071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:18:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ca2f640 1 --2- 172.21.15.70:0/1984826791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe54815a550 0x7fe5480aef50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe5440097c0 tx=0x7fe54401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40f58e006edaa59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe53f7fe640 1 -- 172.21.15.70:0/1984826791 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5440b0070 con 0x7fe54815a550 2024-04-22T01:18:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe53f7fe640 1 -- 172.21.15.70:0/1984826791 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5440ab600 con 0x7fe54815a550 2024-04-22T01:18:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe53f7fe640 1 -- 172.21.15.70:0/1984826791 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5440ab9b0 con 0x7fe54815a550 2024-04-22T01:18:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- 172.21.15.70:0/1984826791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe54815a550 msgr2=0x7fe5480aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 --2- 172.21.15.70:0/1984826791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe54815a550 0x7fe5480aef50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe5440097c0 tx=0x7fe54401c4a0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- 172.21.15.70:0/1984826791 shutdown_connections 2024-04-22T01:18:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 --2- 172.21.15.70:0/1984826791 >> v2:172.21.15.98:3300/0 conn(0x7fe548071b20 0x7fe548071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 --2- 172.21.15.70:0/1984826791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe54815a550 0x7fe5480aef50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- 172.21.15.70:0/1984826791 >> 172.21.15.70:0/1984826791 conn(0x7fe54806b1f0 msgr2=0x7fe54806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- 172.21.15.70:0/1984826791 shutdown_connections 2024-04-22T01:18:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- 172.21.15.70:0/1984826791 wait complete. 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 Processor -- start 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- start start 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe548071b20 0x7fe5480aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480aeae0 0x7fe548147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 0x7fe54814b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480b30d0 con 0x7fe54815a550 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480b2f50 con 0x7fe5480aeae0 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ec93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe5480b3250 con 0x7fe548071b20 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 0x7fe54814b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 0x7fe54814b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59710/0 (socket says 172.21.15.70:59710) 2024-04-22T01:18:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 -- 172.21.15.70:0/4265176446 learned_addr learned my addr 172.21.15.70:0/4265176446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54ca2f640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe548071b20 0x7fe5480aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe53ffff640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480aeae0 0x7fe548147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 -- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe548071b20 msgr2=0x7fe5480aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe548071b20 0x7fe5480aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 -- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480aeae0 msgr2=0x7fe548147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480aeae0 0x7fe548147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 -- 172.21.15.70:0/4265176446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe548072700 con 0x7fe54815a550 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.740+0000 7fe54d230640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 0x7fe54814b160 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fe540009f00 tx=0x7fe54000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540018030 con 0x7fe54815a550 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5480af710 con 0x7fe54815a550 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe54000ce50 con 0x7fe54815a550 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540003bc0 con 0x7fe54815a550 2024-04-22T01:18:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54815ad30 con 0x7fe54815a550 2024-04-22T01:18:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe548068210 con 0x7fe54815a550 2024-04-22T01:18:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe54001b070 con 0x7fe54815a550 2024-04-22T01:18:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe53d7fa640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5340678c0 0x7fe534069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fe540086930 con 0x7fe54815a550 2024-04-22T01:18:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe54ca2f640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5340678c0 0x7fe534069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.744+0000 7fe54ca2f640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5340678c0 0x7fe534069d80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe5440ad000 tx=0x7fe5440a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.748+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe540010040 con 0x7fe54815a550 2024-04-22T01:18:10.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.896+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe54806b6e0 con 0x7fe54815a550 2024-04-22T01:18:10.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.900+0000 7fe53d7fa640 1 -- 172.21.15.70:0/4265176446 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16909 (secure 0 0 0) 0x7fe540012340 con 0x7fe54815a550 2024-04-22T01:18:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:10.904 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":78,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:17:42.835983+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:17:39.433363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-22T01:17:40.173738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5340678c0 msgr2=0x7fe534069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5340678c0 0x7fe534069d80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe5440ad000 tx=0x7fe5440a6000 comp rx=0 tx=0).stop 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 msgr2=0x7fe54814b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 0x7fe54814b160 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fe540009f00 tx=0x7fe54000c490 comp rx=0 tx=0).stop 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 shutdown_connections 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe5340678c0 0x7fe534069d80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54815a550 0x7fe54814b160 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480aeae0 0x7fe548147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 --2- 172.21.15.70:0/4265176446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe548071b20 0x7fe5480aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:10.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 >> 172.21.15.70:0/4265176446 conn(0x7fe54806b1f0 msgr2=0x7fe5480cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:10.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 shutdown_connections 2024-04-22T01:18:10.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:10.904+0000 7fe54ec93640 1 -- 172.21.15.70:0/4265176446 wait complete. 2024-04-22T01:18:10.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:18:11.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.028+0000 7fc0f6fdd640 1 Processor -- start 2024-04-22T01:18:11.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.028+0000 7fc0f6fdd640 1 -- start start 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f6fdd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc0f00ce880 0x7fc0f00d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f6fdd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f00cee40 con 0x7fc0f00ce3f0 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f6fdd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f00cf120 con 0x7fc0f00d17b0 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f6fdd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc0f015c150 con 0x7fc0f00ce880 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e7fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc0f00ce3f0 0x7fc0f00cc610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56808/0 (socket says 172.21.15.70:56808) 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e7fff640 1 -- 172.21.15.70:0/2302213083 learned_addr learned my addr 172.21.15.70:0/2302213083 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112732589 0 0) 0x7fc0f00cee40 con 0x7fc0f00ce3f0 2024-04-22T01:18:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 --2- 172.21.15.70:0/2302213083 >> v2:172.21.15.98:3300/0 conn(0x7fc0f00ce880 0x7fc0f00d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0d8003530 con 0x7fc0f00ce3f0 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 326686008 0 0) 0x7fc0d8003530 con 0x7fc0f00ce3f0 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 >> v2:172.21.15.98:3300/0 conn(0x7fc0f00ce880 msgr2=0x7fc0f00d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 --2- 172.21.15.70:0/2302213083 >> v2:172.21.15.98:3300/0 conn(0x7fc0f00ce880 0x7fc0f00d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 >> v1:172.21.15.98:6789/0 conn(0x7fc0f00d17b0 legacy=0x7fc0f00d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f00cf870 con 0x7fc0f00ce3f0 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fc0e0001680 con 0x7fc0f00ce3f0 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 >> v1:172.21.15.70:6789/0 conn(0x7fc0f00ce3f0 legacy=0x7fc0f00cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d8003a30 0x7fc0d8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0d8004810 0x7fc0d8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0d80051d0 0x7fc0d800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f015c150 con 0x7fc0d80051d0 2024-04-22T01:18:11.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d800c100 con 0x7fc0d8003a30 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d800c300 con 0x7fc0d8004810 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0d80051d0 0x7fc0d800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f4d79640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d8003a30 0x7fc0d8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e7fff640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0d8004810 0x7fc0d8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 -- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0d8004810 msgr2=0x7fc0d8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0d8004810 0x7fc0d8004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 -- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d8003a30 msgr2=0x7fc0d8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d8003a30 0x7fc0d8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 -- 172.21.15.70:0/2302213083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d8003530 con 0x7fc0d80051d0 2024-04-22T01:18:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f4d79640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d8003a30 0x7fc0d8003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 -- 172.21.15.70:0/2302213083 reap_dead start 2024-04-22T01:18:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0f557a640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0d80051d0 0x7fc0d800b9d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009fc0 tx=0x7fc0e801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2becce42e03a0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8009c10 con 0x7fc0d80051d0 2024-04-22T01:18:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0e80a5d10 con 0x7fc0d80051d0 2024-04-22T01:18:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.032+0000 7fc0e77fe640 1 -- 172.21.15.70:0/2302213083 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8003be0 con 0x7fc0d80051d0 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/2302213083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0d80051d0 msgr2=0x7fc0d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0d80051d0 0x7fc0d800b9d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009fc0 tx=0x7fc0e801ccc0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/2302213083 shutdown_connections 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/2302213083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0d80051d0 0x7fc0d800b9d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/2302213083 >> 172.21.15.70:0/2302213083 conn(0x7fc0f008b1b0 msgr2=0x7fc0f008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/2302213083 shutdown_connections 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/2302213083 wait complete. 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 Processor -- start 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- start start 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d80051d0 0x7fc0f015dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 0x7fc0f00b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f015c980 0x7fc0f015cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00cfbe0 con 0x7fc0f00afef0 2024-04-22T01:18:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00d1b90 con 0x7fc0d80051d0 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f00d1d10 con 0x7fc0f015c980 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 0x7fc0f00b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 0x7fc0f00b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59732/0 (socket says 172.21.15.70:59732) 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 -- 172.21.15.70:0/1214963220 learned_addr learned my addr 172.21.15.70:0/1214963220 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f4d79640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d80051d0 0x7fc0f015dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f557a640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f015c980 0x7fc0f015cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 -- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f015c980 msgr2=0x7fc0f015cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f015c980 0x7fc0f015cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 -- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d80051d0 msgr2=0x7fc0f015dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d80051d0 0x7fc0f015dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 -- 172.21.15.70:0/1214963220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f015e5a0 con 0x7fc0f00afef0 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f557a640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f015c980 0x7fc0f015cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f4d79640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d80051d0 0x7fc0f015dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e7fff640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 0x7fc0f00b0370 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000d7a0 tx=0x7fc0e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0019070 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0e00072d0 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f00b5030 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00047f0 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f00b4d20 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.036+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f004f510 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.040+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc0e001c070 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.040+0000 7fc0e57fa640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0dc0678c0 0x7fc0dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.040+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fc0e0085700 con 0x7fc0f00afef0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.040+0000 7fc0f4d79640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0dc0678c0 0x7fc0dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.040+0000 7fc0f4d79640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0dc0678c0 0x7fc0dc069d80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec001700 tx=0x7fc0ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:11.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.044+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc0e00560b0 con 0x7fc0f00afef0 2024-04-22T01:18:11.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.232+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc0f00ce000 con 0x7fc0f00afef0 2024-04-22T01:18:11.237 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:18:11.236+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:18:11.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.236+0000 7fc0e57fa640 1 -- 172.21.15.70:0/1214963220 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7fc0e00559b0 con 0x7fc0f00afef0 2024-04-22T01:18:11.239 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:18:11.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0dc0678c0 msgr2=0x7fc0dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0dc0678c0 0x7fc0dc069d80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec001700 tx=0x7fc0ec009040 comp rx=0 tx=0).stop 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 msgr2=0x7fc0f00b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 0x7fc0f00b0370 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000d7a0 tx=0x7fc0e000dc60 comp rx=0 tx=0).stop 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 shutdown_connections 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc0dc0678c0 0x7fc0dc069d80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0f015c980 0x7fc0f015cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0f00afef0 0x7fc0f00b0370 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 --2- 172.21.15.70:0/1214963220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0d80051d0 0x7fc0f015dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 >> 172.21.15.70:0/1214963220 conn(0x7fc0f008b1b0 msgr2=0x7fc0f015fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 shutdown_connections 2024-04-22T01:18:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.240+0000 7fc0f6fdd640 1 -- 172.21.15.70:0/1214963220 wait complete. 2024-04-22T01:18:11.253 INFO:tasks.ceph.mds.d.smithi098.stderr: -3275> 2024-04-22T01:17:47.076+0000 7f47cdab5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:18:11.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:18:11.305 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:18:11.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 Processor -- start 2024-04-22T01:18:11.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 -- start start 2024-04-22T01:18:11.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f95100b98a0 0x7f95100d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100cea10 0x7f95100cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95100b9ce0 con 0x7f951015de20 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f95100cf3c0 con 0x7f95100b98a0 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f9517f2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f95100b75e0 con 0x7f95100cea10 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f95100b98a0 0x7f95100d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f95100b98a0 0x7f95100d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59738/0 (socket says 172.21.15.70:59738) 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 -- 172.21.15.70:0/3256804832 learned_addr learned my addr 172.21.15.70:0/3256804832 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95164c8640 1 --2- 172.21.15.70:0/3256804832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100cea10 0x7f95100cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 -- 172.21.15.70:0/3256804832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100cea10 msgr2=0x7f95100cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 --2- 172.21.15.70:0/3256804832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100cea10 0x7f95100cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 -- 172.21.15.70:0/3256804832 >> v1:172.21.15.98:6789/0 conn(0x7f951015de20 legacy=0x7f95100b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.372+0000 7f95154c6640 1 -- 172.21.15.70:0/3256804832 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100b78c0 con 0x7f95100b98a0 2024-04-22T01:18:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f95154c6640 1 --2- 172.21.15.70:0/3256804832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95100b98a0 0x7f95100d1d80 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f95100d4e10 tx=0x7f950001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149d825e0f3e1ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9514cc5640 1 -- 172.21.15.70:0/3256804832 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000b5070 con 0x7f95100b98a0 2024-04-22T01:18:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9514cc5640 1 -- 172.21.15.70:0/3256804832 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95000a8990 con 0x7f95100b98a0 2024-04-22T01:18:11.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9514cc5640 1 -- 172.21.15.70:0/3256804832 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000a8d60 con 0x7f95100b98a0 2024-04-22T01:18:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 -- 172.21.15.70:0/3256804832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95100b98a0 msgr2=0x7f95100d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 --2- 172.21.15.70:0/3256804832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95100b98a0 0x7f95100d1d80 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f95100d4e10 tx=0x7f950001ecd0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 -- 172.21.15.70:0/3256804832 shutdown_connections 2024-04-22T01:18:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 --2- 172.21.15.70:0/3256804832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100cea10 0x7f95100cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 --2- 172.21.15.70:0/3256804832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95100b98a0 0x7f95100d1d80 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 -- 172.21.15.70:0/3256804832 >> 172.21.15.70:0/3256804832 conn(0x7f9510072d20 msgr2=0x7f9510073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:11.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 -- 172.21.15.70:0/3256804832 shutdown_connections 2024-04-22T01:18:11.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.376+0000 7f9517f2b640 1 -- 172.21.15.70:0/3256804832 wait complete. 2024-04-22T01:18:11.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 Processor -- start 2024-04-22T01:18:11.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- start start 2024-04-22T01:18:11.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100b98a0 0x7f95101524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95100cea10 0x7f95100c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 0x7f951014d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95100b8160 con 0x7f951015de20 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95100b7e80 con 0x7f95100cea10 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f951015e200 con 0x7f95100b98a0 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 0x7f951014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 0x7f951014d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59740/0 (socket says 172.21.15.70:59740) 2024-04-22T01:18:11.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 -- 172.21.15.70:0/4057526879 learned_addr learned my addr 172.21.15.70:0/4057526879 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9515cc7640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100b98a0 0x7f95101524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95154c6640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95100cea10 0x7f95100c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 -- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100b98a0 msgr2=0x7f95101524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100b98a0 0x7f95101524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 -- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95100cea10 msgr2=0x7f95100c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95100cea10 0x7f95100c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 -- 172.21.15.70:0/4057526879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95101606a0 con 0x7f951015de20 2024-04-22T01:18:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95164c8640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 0x7f951014d890 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f950800cd70 tx=0x7f950800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508003ba0 con 0x7f951015de20 2024-04-22T01:18:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9508003d40 con 0x7f951015de20 2024-04-22T01:18:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508004240 con 0x7f951015de20 2024-04-22T01:18:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95100c9090 con 0x7f951015de20 2024-04-22T01:18:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9510151cb0 con 0x7f951015de20 2024-04-22T01:18:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9510068210 con 0x7f951015de20 2024-04-22T01:18:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.380+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f95080043e0 con 0x7f951015de20 2024-04-22T01:18:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.384+0000 7f95067fc640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f94f40678c0 0x7f94f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.384+0000 7f9515cc7640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f94f40678c0 0x7f94f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.384+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6187+0+0 (secure 0 0 0) 0x7f9508086980 con 0x7f951015de20 2024-04-22T01:18:11.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.384+0000 7f9515cc7640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f94f40678c0 0x7f94f4069d80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b810 tx=0x7f950c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:11.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.388+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f950801d340 con 0x7f951015de20 2024-04-22T01:18:11.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:11.564+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9510068920 con 0x7f951015de20 2024-04-22T01:18:12.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.240+0000 7f95067fc640 1 -- 172.21.15.70:0/4057526879 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7f95080572e0 con 0x7f951015de20 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f94f40678c0 msgr2=0x7f94f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f94f40678c0 0x7f94f4069d80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b810 tx=0x7f950c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 msgr2=0x7f951014d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 0x7f951014d890 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f950800cd70 tx=0x7f950800a660 comp rx=0 tx=0).stop 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 shutdown_connections 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f94f40678c0 0x7f94f4069d80 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f951015de20 0x7f951014d890 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95100cea10 0x7f95100c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 --2- 172.21.15.70:0/4057526879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95100b98a0 0x7f95101524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 >> 172.21.15.70:0/4057526879 conn(0x7f9510072d20 msgr2=0x7f95100b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 shutdown_connections 2024-04-22T01:18:12.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.244+0000 7f9517f2b640 1 -- 172.21.15.70:0/4057526879 wait complete. 2024-04-22T01:18:12.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:18:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 Processor -- start 2024-04-22T01:18:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 -- start start 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f806b900 0x7f97f80b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f97f80b1df0 0x7f97f80cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f80b2280 con 0x7f97f815e030 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f97f80b0420 con 0x7f97f80b1df0 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97fd923640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f97f80cb0b0 con 0x7f97f806b900 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f97f80b1df0 0x7f97f80cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f97f80b1df0 0x7f97f80cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59742/0 (socket says 172.21.15.70:59742) 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 -- 172.21.15.70:0/780025929 learned_addr learned my addr 172.21.15.70:0/780025929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f67fc640 1 --2- 172.21.15.70:0/780025929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f806b900 0x7f97f80b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 -- 172.21.15.70:0/780025929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f806b900 msgr2=0x7f97f80b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 --2- 172.21.15.70:0/780025929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f806b900 0x7f97f80b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 -- 172.21.15.70:0/780025929 >> v1:172.21.15.98:6789/0 conn(0x7f97f815e030 legacy=0x7f97f806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:12.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.388+0000 7f97f77fe640 1 -- 172.21.15.70:0/780025929 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f80b0670 con 0x7f97f80b1df0 2024-04-22T01:18:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97f77fe640 1 --2- 172.21.15.70:0/780025929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f80b1df0 0x7f97f80cd470 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009e50 tx=0x7f97f001c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fc8d6b39624c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97f5ffb640 1 -- 172.21.15.70:0/780025929 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00ac020 con 0x7f97f80b1df0 2024-04-22T01:18:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97f5ffb640 1 -- 172.21.15.70:0/780025929 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f00a64a0 con 0x7f97f80b1df0 2024-04-22T01:18:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97f5ffb640 1 -- 172.21.15.70:0/780025929 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00a6870 con 0x7f97f80b1df0 2024-04-22T01:18:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 -- 172.21.15.70:0/780025929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f80b1df0 msgr2=0x7f97f80cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 --2- 172.21.15.70:0/780025929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f80b1df0 0x7f97f80cd470 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009e50 tx=0x7f97f001c810 comp rx=0 tx=0).stop 2024-04-22T01:18:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 -- 172.21.15.70:0/780025929 shutdown_connections 2024-04-22T01:18:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 --2- 172.21.15.70:0/780025929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f80b1df0 0x7f97f80cd470 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 --2- 172.21.15.70:0/780025929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f806b900 0x7f97f80b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 -- 172.21.15.70:0/780025929 >> 172.21.15.70:0/780025929 conn(0x7f97f8099a60 msgr2=0x7f97f8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 -- 172.21.15.70:0/780025929 shutdown_connections 2024-04-22T01:18:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.392+0000 7f97fd923640 1 -- 172.21.15.70:0/780025929 wait complete. 2024-04-22T01:18:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 Processor -- start 2024-04-22T01:18:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- start start 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97f806b900 0x7f97f81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f80b1df0 0x7f97f8157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 0x7f97f8158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97f80ce340 con 0x7f97f815e030 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97f80ce1c0 con 0x7f97f806b900 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f97f816aa90 con 0x7f97f80b1df0 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 0x7f97f8158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 0x7f97f8158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59752/0 (socket says 172.21.15.70:59752) 2024-04-22T01:18:12.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 -- 172.21.15.70:0/3440434626 learned_addr learned my addr 172.21.15.70:0/3440434626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f6ffd640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97f806b900 0x7f97f81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f67fc640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f80b1df0 0x7f97f8157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 -- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f80b1df0 msgr2=0x7f97f8157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f80b1df0 0x7f97f8157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 -- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97f806b900 msgr2=0x7f97f81542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97f806b900 0x7f97f81542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 -- 172.21.15.70:0/3440434626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f81547f0 con 0x7f97f815e030 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97f77fe640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 0x7f97f8158010 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f97f001aec0 tx=0x7f97f00195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0009510 con 0x7f97f815e030 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f00a7680 con 0x7f97f815e030 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00a7a50 con 0x7f97f815e030 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f8158640 con 0x7f97f815e030 2024-04-22T01:18:12.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f80c95b0 con 0x7f97f815e030 2024-04-22T01:18:12.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.396+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f8068920 con 0x7f97f815e030 2024-04-22T01:18:12.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.400+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f97f00af020 con 0x7f97f815e030 2024-04-22T01:18:12.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.400+0000 7f97dffff640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f97e40678f0 0x7f97e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:12.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.400+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6187+0+0 (secure 0 0 0) 0x7f97f011fba0 con 0x7f97f815e030 2024-04-22T01:18:12.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.400+0000 7f97f6ffd640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f97e40678f0 0x7f97e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:12.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.400+0000 7f97f6ffd640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f97e40678f0 0x7f97e4069db0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f97e800b810 tx=0x7f97e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:12.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.404+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f97f80b2280 con 0x7f97f815e030 2024-04-22T01:18:12.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:12.552+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f97f80cab60 con 0x7f97f815e030 2024-04-22T01:18:13.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:18:13.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:18:13.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.248+0000 7f97dffff640 1 -- 172.21.15.70:0/3440434626 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v80) ==== 181+0+0 (secure 0 0 0) 0x7f97f00f0590 con 0x7f97f815e030 2024-04-22T01:18:13.253 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:18:13.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.252+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f97e40678f0 msgr2=0x7f97e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:13.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.252+0000 7f97fd923640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f97e40678f0 0x7f97e4069db0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f97e800b810 tx=0x7f97e8008040 comp rx=0 tx=0).stop 2024-04-22T01:18:13.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.252+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 msgr2=0x7f97f8158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:13.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.252+0000 7f97fd923640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 0x7f97f8158010 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f97f001aec0 tx=0x7f97f00195b0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 shutdown_connections 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f97e40678f0 0x7f97e4069db0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97f815e030 0x7f97f8158010 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97f80b1df0 0x7f97f8157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 --2- 172.21.15.70:0/3440434626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97f806b900 0x7f97f81542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 >> 172.21.15.70:0/3440434626 conn(0x7f97f8099a60 msgr2=0x7f97f806b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 shutdown_connections 2024-04-22T01:18:13.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.256+0000 7f97fd923640 1 -- 172.21.15.70:0/3440434626 wait complete. 2024-04-22T01:18:13.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:18:13.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 Processor -- start 2024-04-22T01:18:13.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 -- start start 2024-04-22T01:18:13.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8cc415df10 0x7f8cc406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:13.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40b21a0 0x7f8cc4162540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:13.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cc406b860 con 0x7f8cc40cdd70 2024-04-22T01:18:13.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8cc406bb40 con 0x7f8cc415df10 2024-04-22T01:18:13.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8ccc628640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc4161a60 con 0x7f8cc40b21a0 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cca3c4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8cc415df10 0x7f8cc406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cc9bc3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8cc40cdd70 0x7f8cc40b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56822/0 (socket says 172.21.15.70:56822) 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cc9bc3640 1 -- 172.21.15.70:0/1093168979 learned_addr learned my addr 172.21.15.70:0/1093168979 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1093168979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40b21a0 msgr2=0x7f8cc4162540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cca3c4640 1 --2- 172.21.15.70:0/1093168979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40b21a0 0x7f8cc4162540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1093168979 >> v1:172.21.15.70:6789/0 conn(0x7f8cc40cdd70 legacy=0x7f8cc40b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1093168979 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc4161d40 con 0x7f8cc415df10 2024-04-22T01:18:13.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.400+0000 7f8cc93c2640 1 -- 172.21.15.70:0/1093168979 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571495901 0 0) 0x7f8cc406b860 con 0x7f8cc40cdd70 2024-04-22T01:18:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8cca3c4640 1 --2- 172.21.15.70:0/1093168979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc415df10 0x7f8cc406b1f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00097c0 tx=0x7f8cc001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd32df0338d7175e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8cc93c2640 1 -- 172.21.15.70:0/1093168979 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00b1070 con 0x7f8cc415df10 2024-04-22T01:18:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8cc93c2640 1 -- 172.21.15.70:0/1093168979 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc00ac620 con 0x7f8cc415df10 2024-04-22T01:18:13.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8cc93c2640 1 -- 172.21.15.70:0/1093168979 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00ac9f0 con 0x7f8cc415df10 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 -- 172.21.15.70:0/1093168979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc415df10 msgr2=0x7f8cc406b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1093168979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc415df10 0x7f8cc406b1f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00097c0 tx=0x7f8cc001c3e0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 -- 172.21.15.70:0/1093168979 shutdown_connections 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1093168979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40b21a0 0x7f8cc4162540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1093168979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc415df10 0x7f8cc406b1f0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 -- 172.21.15.70:0/1093168979 >> 172.21.15.70:0/1093168979 conn(0x7f8cc4099a60 msgr2=0x7f8cc4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 -- 172.21.15.70:0/1093168979 shutdown_connections 2024-04-22T01:18:13.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.404+0000 7f8ccc628640 1 -- 172.21.15.70:0/1093168979 wait complete. 2024-04-22T01:18:13.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 Processor -- start 2024-04-22T01:18:13.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- start start 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 0x7f8cc4160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40cdd70 0x7f8cc41fb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cc415df10 0x7f8cc40d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40cbac0 con 0x7f8cc40b21a0 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40cb940 con 0x7f8cc415df10 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc40cbc40 con 0x7f8cc40cdd70 2024-04-22T01:18:13.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 0x7f8cc4160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 0x7f8cc4160060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59780/0 (socket says 172.21.15.70:59780) 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1745654898 learned_addr learned my addr 172.21.15.70:0/1745654898 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1745654898 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40cdd70 msgr2=0x7f8cc41fb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40cdd70 0x7f8cc41fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1745654898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cc415df10 msgr2=0x7f8cc40d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cc415df10 0x7f8cc40d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 -- 172.21.15.70:0/1745654898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc4155f50 con 0x7f8cc40b21a0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cca3c4640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 0x7f8cc4160060 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00097c0 tx=0x7f8cc001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00bc070 con 0x7f8cc40b21a0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc0004020 con 0x7f8cc40b21a0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00b1040 con 0x7f8cc40b21a0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc4156230 con 0x7f8cc40b21a0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc415e7b0 con 0x7f8cc40b21a0 2024-04-22T01:18:13.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc4068920 con 0x7f8cc40b21a0 2024-04-22T01:18:13.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.408+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8cc0003620 con 0x7f8cc40b21a0 2024-04-22T01:18:13.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.412+0000 7f8cbaffd640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ca80678c0 0x7f8ca8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:13.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.412+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5749+0+0 (secure 0 0 0) 0x7f8cc0120580 con 0x7f8cc40b21a0 2024-04-22T01:18:13.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.412+0000 7f8cc9bc3640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ca80678c0 0x7f8ca8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:13.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.412+0000 7f8cc9bc3640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ca80678c0 0x7f8ca8069d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4005890 tx=0x7f8cb4007960 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:13.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.416+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8cc0016090 con 0x7f8cc40b21a0 2024-04-22T01:18:13.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:13.564+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8cc406c0e0 con 0x7f8cc40b21a0 2024-04-22T01:18:14.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.252+0000 7f8cbaffd640 1 -- 172.21.15.70:0/1745654898 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v81) ==== 169+0+0 (secure 0 0 0) 0x7f8cc00f1090 con 0x7f8cc40b21a0 2024-04-22T01:18:14.255 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:18:14.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ca80678c0 msgr2=0x7f8ca8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ca80678c0 0x7f8ca8069d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4005890 tx=0x7f8cb4007960 comp rx=0 tx=0).stop 2024-04-22T01:18:14.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 msgr2=0x7f8cc4160060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 0x7f8cc4160060 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00097c0 tx=0x7f8cc001ac20 comp rx=0 tx=0).stop 2024-04-22T01:18:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 shutdown_connections 2024-04-22T01:18:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ca80678c0 0x7f8ca8069d80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cc415df10 0x7f8cc40d0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cc40cdd70 0x7f8cc41fb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 --2- 172.21.15.70:0/1745654898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc40b21a0 0x7f8cc4160060 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.256+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 >> 172.21.15.70:0/1745654898 conn(0x7f8cc4099a60 msgr2=0x7f8cc40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:14.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.260+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 shutdown_connections 2024-04-22T01:18:14.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.260+0000 7f8ccc628640 1 -- 172.21.15.70:0/1745654898 wait complete. 2024-04-22T01:18:14.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-22T01:18:14.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.404+0000 7fed59cf7640 1 Processor -- start 2024-04-22T01:18:14.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.404+0000 7fed59cf7640 1 -- start start 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed59cf7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fed54158650 0x7fed5415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed59cf7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed5406d8f0 0x7fed5406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed59cf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406f440 0x7fed5415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed59cf7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fed5406de60 con 0x7fed5406d8f0 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed59cf7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fed5406e140 con 0x7fed54158650 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed59cf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed5406e420 con 0x7fed5406f440 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed5406d8f0 0x7fed5406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed5406d8f0 0x7fed5406ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59794/0 (socket says 172.21.15.70:59794) 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 -- 172.21.15.70:0/3691735612 learned_addr learned my addr 172.21.15.70:0/3691735612 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed537fe640 1 --2- 172.21.15.70:0/3691735612 >> v2:172.21.15.98:3300/0 conn(0x7fed54158650 0x7fed5415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed53fff640 1 --2- 172.21.15.70:0/3691735612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406f440 0x7fed5415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 -- 172.21.15.70:0/3691735612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406f440 msgr2=0x7fed5415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 --2- 172.21.15.70:0/3691735612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406f440 0x7fed5415f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 -- 172.21.15.70:0/3691735612 >> v2:172.21.15.98:3300/0 conn(0x7fed54158650 msgr2=0x7fed5415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 --2- 172.21.15.70:0/3691735612 >> v2:172.21.15.98:3300/0 conn(0x7fed54158650 0x7fed5415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 -- 172.21.15.70:0/3691735612 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5406e700 con 0x7fed5406d8f0 2024-04-22T01:18:14.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed537fe640 1 --2- 172.21.15.70:0/3691735612 >> v2:172.21.15.98:3300/0 conn(0x7fed54158650 0x7fed5415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:14.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed4affd640 1 --2- 172.21.15.70:0/3691735612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed5406d8f0 0x7fed5406ef00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fed54073010 tx=0x7fed3c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4e5681ceec56f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:14.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed52ffd640 1 -- 172.21.15.70:0/3691735612 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0b0070 con 0x7fed5406d8f0 2024-04-22T01:18:14.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed52ffd640 1 -- 172.21.15.70:0/3691735612 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed3c0ab600 con 0x7fed5406d8f0 2024-04-22T01:18:14.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.408+0000 7fed52ffd640 1 -- 172.21.15.70:0/3691735612 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0ab9d0 con 0x7fed5406d8f0 2024-04-22T01:18:14.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/3691735612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed5406d8f0 msgr2=0x7fed5406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- 172.21.15.70:0/3691735612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed5406d8f0 0x7fed5406ef00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fed54073010 tx=0x7fed3c01c490 comp rx=0 tx=0).stop 2024-04-22T01:18:14.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/3691735612 shutdown_connections 2024-04-22T01:18:14.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- 172.21.15.70:0/3691735612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406f440 0x7fed5415f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- 172.21.15.70:0/3691735612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed5406d8f0 0x7fed5406ef00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- 172.21.15.70:0/3691735612 >> v2:172.21.15.98:3300/0 conn(0x7fed54158650 0x7fed5415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/3691735612 >> 172.21.15.70:0/3691735612 conn(0x7fed540b1d40 msgr2=0x7fed540cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/3691735612 shutdown_connections 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/3691735612 wait complete. 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 Processor -- start 2024-04-22T01:18:14.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- start start 2024-04-22T01:18:14.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406d8f0 0x7fed541604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5406f440 0x7fed54160a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 0x7fed5414a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed540b3870 con 0x7fed54158650 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed540b36f0 con 0x7fed5406f440 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed540b39f0 con 0x7fed5406d8f0 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 0x7fed5414a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 0x7fed5414a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59800/0 (socket says 172.21.15.70:59800) 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 -- 172.21.15.70:0/542195886 learned_addr learned my addr 172.21.15.70:0/542195886 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed537fe640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406d8f0 0x7fed541604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed4affd640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5406f440 0x7fed54160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 -- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406d8f0 msgr2=0x7fed541604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406d8f0 0x7fed541604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 -- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5406f440 msgr2=0x7fed54160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5406f440 0x7fed54160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 -- 172.21.15.70:0/542195886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5414ea30 con 0x7fed54158650 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed4affd640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5406f440 0x7fed54160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed53fff640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 0x7fed5414a480 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fed4c007e60 tx=0x7fed4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c014620 con 0x7fed54158650 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed4c019070 con 0x7fed54158650 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5414d9b0 con 0x7fed54158650 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c014a20 con 0x7fed54158650 2024-04-22T01:18:14.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.412+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed5414df70 con 0x7fed54158650 2024-04-22T01:18:14.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.416+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed54068210 con 0x7fed54158650 2024-04-22T01:18:14.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.416+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed4c014bc0 con 0x7fed54158650 2024-04-22T01:18:14.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.416+0000 7fed50ff9640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed340678c0 0x7fed34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.416+0000 7fed537fe640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed340678c0 0x7fed34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.416+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7fed4c0863a0 con 0x7fed54158650 2024-04-22T01:18:14.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.416+0000 7fed537fe640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed340678c0 0x7fed34069d80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fed4400ba10 tx=0x7fed44008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:14.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.420+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fed5414ea30 con 0x7fed54158650 2024-04-22T01:18:14.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.580+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fed540b40e0 con 0x7fed54158650 2024-04-22T01:18:14.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.588+0000 7fed50ff9640 1 -- 172.21.15.70:0/542195886 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7fed4c057050 con 0x7fed54158650 2024-04-22T01:18:14.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed340678c0 msgr2=0x7fed34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed340678c0 0x7fed34069d80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fed4400ba10 tx=0x7fed44008040 comp rx=0 tx=0).stop 2024-04-22T01:18:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 msgr2=0x7fed5414a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 0x7fed5414a480 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fed4c007e60 tx=0x7fed4c00e4b0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 shutdown_connections 2024-04-22T01:18:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed340678c0 0x7fed34069d80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed54158650 0x7fed5414a480 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5406f440 0x7fed54160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 --2- 172.21.15.70:0/542195886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed5406d8f0 0x7fed541604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 >> 172.21.15.70:0/542195886 conn(0x7fed540b1d40 msgr2=0x7fed54150d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 shutdown_connections 2024-04-22T01:18:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.592+0000 7fed59cf7640 1 -- 172.21.15.70:0/542195886 wait complete. 2024-04-22T01:18:14.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df' 2024-04-22T01:18:14.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 Processor -- start 2024-04-22T01:18:14.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 -- start start 2024-04-22T01:18:14.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b980cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b980cc150 0x7f3b98071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3b98072060 0x7f3b9815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3b980724d0 con 0x7f3b98072060 2024-04-22T01:18:14.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3b980d0250 con 0x7f3b980cc150 2024-04-22T01:18:14.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ec72640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3b980cd7b0 con 0x7f3b981455b0 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3b98072060 0x7f3b9815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3b98072060 0x7f3b9815f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59804/0 (socket says 172.21.15.70:59804) 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 -- 172.21.15.70:0/2943991838 learned_addr learned my addr 172.21.15.70:0/2943991838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9ca0e640 1 --2- 172.21.15.70:0/2943991838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b980cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b97fff640 1 --2- 172.21.15.70:0/2943991838 >> v2:172.21.15.98:3300/0 conn(0x7f3b980cc150 0x7f3b98071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 -- 172.21.15.70:0/2943991838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 msgr2=0x7f3b980cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 --2- 172.21.15.70:0/2943991838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b980cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 -- 172.21.15.70:0/2943991838 >> v2:172.21.15.98:3300/0 conn(0x7f3b980cc150 msgr2=0x7f3b98071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 --2- 172.21.15.70:0/2943991838 >> v2:172.21.15.98:3300/0 conn(0x7f3b980cc150 0x7f3b98071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b9d20f640 1 -- 172.21.15.70:0/2943991838 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980cc710 con 0x7f3b98072060 2024-04-22T01:18:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.724+0000 7f3b97fff640 1 --2- 172.21.15.70:0/2943991838 >> v2:172.21.15.98:3300/0 conn(0x7f3b980cc150 0x7f3b98071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:14.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9d20f640 1 --2- 172.21.15.70:0/2943991838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b98072060 0x7f3b9815f7a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b4d80 tx=0x7f3b9001c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aff1b70f8c97111 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:14.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b977fe640 1 -- 172.21.15.70:0/2943991838 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b900a6040 con 0x7f3b98072060 2024-04-22T01:18:14.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b977fe640 1 -- 172.21.15.70:0/2943991838 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b900af070 con 0x7f3b98072060 2024-04-22T01:18:14.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b977fe640 1 -- 172.21.15.70:0/2943991838 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b900aa660 con 0x7f3b98072060 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- 172.21.15.70:0/2943991838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b98072060 msgr2=0x7f3b9815f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/2943991838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b98072060 0x7f3b9815f7a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b4d80 tx=0x7f3b9001c610 comp rx=0 tx=0).stop 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- 172.21.15.70:0/2943991838 shutdown_connections 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/2943991838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b98072060 0x7f3b9815f7a0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/2943991838 >> v2:172.21.15.98:3300/0 conn(0x7f3b980cc150 0x7f3b98071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/2943991838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b980cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- 172.21.15.70:0/2943991838 >> 172.21.15.70:0/2943991838 conn(0x7f3b9806b1f0 msgr2=0x7f3b9806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- 172.21.15.70:0/2943991838 shutdown_connections 2024-04-22T01:18:14.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- 172.21.15.70:0/2943991838 wait complete. 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 Processor -- start 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- start start 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b98072060 0x7f3b98159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 0x7f3b9815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b98141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9806e380 con 0x7f3b980cc150 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9806e200 con 0x7f3b98072060 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.728+0000 7f3b9ec72640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3b9806e500 con 0x7f3b981455b0 2024-04-22T01:18:14.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 0x7f3b9815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 0x7f3b9815b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59812/0 (socket says 172.21.15.70:59812) 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 -- 172.21.15.70:0/433475828 learned_addr learned my addr 172.21.15.70:0/433475828 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9ca0e640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b98072060 0x7f3b98159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9d20f640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b98141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 -- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 msgr2=0x7f3b98141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b98141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 -- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b98072060 msgr2=0x7f3b98159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b98072060 0x7f3b98159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 -- 172.21.15.70:0/433475828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9813cad0 con 0x7f3b980cc150 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b97fff640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 0x7f3b9815b3e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c7c0 tx=0x7f3b8800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88015070 con 0x7f3b980cc150 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b88007a30 con 0x7f3b980cc150 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b98156610 con 0x7f3b980cc150 2024-04-22T01:18:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88007e00 con 0x7f3b980cc150 2024-04-22T01:18:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b98156eb0 con 0x7f3b980cc150 2024-04-22T01:18:14.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b98068210 con 0x7f3b980cc150 2024-04-22T01:18:14.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b88018070 con 0x7f3b980cc150 2024-04-22T01:18:14.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b957fa640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3b840678c0 0x7f3b84069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:14.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7f3b88085980 con 0x7f3b980cc150 2024-04-22T01:18:14.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9ca0e640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3b840678c0 0x7f3b84069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:14.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.732+0000 7f3b9ca0e640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3b840678c0 0x7f3b84069d80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3b98139b20 tx=0x7f3b8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:14.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.736+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3b880566c0 con 0x7f3b980cc150 2024-04-22T01:18:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:14.956+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]} v 0) -- 0x7f3b9813bb20 con 0x7f3b980cc150 2024-04-22T01:18:15.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b957fa640 1 -- 172.21.15.70:0/433475828 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]}]=0 v210) ==== 134+0+0 (secure 0 0 0) 0x7f3b88055fc0 con 0x7f3b980cc150 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3b840678c0 msgr2=0x7f3b84069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3b840678c0 0x7f3b84069d80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3b98139b20 tx=0x7f3b8c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 msgr2=0x7f3b9815b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 0x7f3b9815b3e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c7c0 tx=0x7f3b8800cc80 comp rx=0 tx=0).stop 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 shutdown_connections 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3b840678c0 0x7f3b84069d80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b981455b0 0x7f3b98141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b980cc150 0x7f3b9815b3e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.256+0000 7f3b9ec72640 1 --2- 172.21.15.70:0/433475828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b98072060 0x7f3b98159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.260+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 >> 172.21.15.70:0/433475828 conn(0x7f3b9806b1f0 msgr2=0x7f3b9813ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:15.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.260+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 shutdown_connections 2024-04-22T01:18:15.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.260+0000 7f3b9ec72640 1 -- 172.21.15.70:0/433475828 wait complete. 2024-04-22T01:18:15.283 INFO:tasks.cephfs_test_runner:That when a client modifies the quota on a directory used ... ok 2024-04-22T01:18:15.284 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_getfattr (tasks.cephfs.test_quota.TestQuota) That quota changes made from one client are visible to another 2024-04-22T01:18:15.284 INFO:tasks.cephfs_test_runner:test_remote_update_getfattr (tasks.cephfs.test_quota.TestQuota) 2024-04-22T01:18:15.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:15.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.404+0000 7faf583a6640 1 Processor -- start 2024-04-22T01:18:15.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.404+0000 7faf583a6640 1 -- start start 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf583a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 0x7faf500cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf583a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faf500cc150 0x7faf50071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf583a6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf500d0250 con 0x7faf50072060 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf583a6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faf500724d0 con 0x7faf500cc150 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf583a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faf500cd7b0 con 0x7faf501455b0 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf56943640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7faf50072060 0x7faf5015f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56832/0 (socket says 172.21.15.70:56832) 2024-04-22T01:18:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf56943640 1 -- 172.21.15.70:0/769392301 learned_addr learned my addr 172.21.15.70:0/769392301 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 --2- 172.21.15.70:0/769392301 >> v2:172.21.15.98:3300/0 conn(0x7faf500cc150 0x7faf50071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf56142640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 0x7faf500cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3201397731 0 0) 0x7faf500d0250 con 0x7faf50072060 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf3c003530 con 0x7faf50072060 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3773947192 0 0) 0x7faf3c003530 con 0x7faf50072060 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 msgr2=0x7faf500cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 0x7faf500cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 >> v2:172.21.15.98:3300/0 conn(0x7faf500cc150 msgr2=0x7faf50071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 --2- 172.21.15.70:0/769392301 >> v2:172.21.15.98:3300/0 conn(0x7faf500cc150 0x7faf50071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf500cc710 con 0x7faf50072060 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7faf48001a00 con 0x7faf50072060 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 >> v1:172.21.15.70:6789/0 conn(0x7faf50072060 legacy=0x7faf5015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf3c0039c0 0x7faf3c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf3c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 0x7faf3c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf500724d0 con 0x7faf3c004810 2024-04-22T01:18:15.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf500cd7b0 con 0x7faf3c0039c0 2024-04-22T01:18:15.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c00bab0 con 0x7faf501455b0 2024-04-22T01:18:15.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf3c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf56142640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf3c0039c0 0x7faf3c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 -- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 msgr2=0x7faf3c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:15.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf501455b0 0x7faf3c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 -- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf3c0039c0 msgr2=0x7faf3c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:15.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf3c0039c0 0x7faf3c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.408+0000 7faf55941640 1 -- 172.21.15.70:0/769392301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c003530 con 0x7faf3c004810 2024-04-22T01:18:15.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf55941640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf3c004c90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ab0 tx=0x7faf4c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3325a7f58dd5e53a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0096b0 con 0x7faf3c004810 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c0a6450 con 0x7faf3c004810 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf55140640 1 -- 172.21.15.70:0/769392301 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0a6840 con 0x7faf3c004810 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/769392301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 msgr2=0x7faf3c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 --2- 172.21.15.70:0/769392301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf3c004c90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ab0 tx=0x7faf4c01ca70 comp rx=0 tx=0).stop 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56943640 1 -- 172.21.15.70:0/769392301 reap_dead start 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/769392301 shutdown_connections 2024-04-22T01:18:15.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/769392301 >> 172.21.15.70:0/769392301 conn(0x7faf5006b1f0 msgr2=0x7faf5006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/769392301 shutdown_connections 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/769392301 wait complete. 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 Processor -- start 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- start start 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf500aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5015b730 0x7faf500ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf500ad550 0x7faf501f8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf50166b30 con 0x7faf3c004810 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf5006f210 con 0x7faf5015b730 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faf50166cb0 con 0x7faf500ad550 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf500aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf500aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59822/0 (socket says 172.21.15.70:59822) 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 -- 172.21.15.70:0/1580285255 learned_addr learned my addr 172.21.15.70:0/1580285255 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 -- 172.21.15.70:0/1580285255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf500ad550 msgr2=0x7faf501f8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf500ad550 0x7faf501f8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 -- 172.21.15.70:0/1580285255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5015b730 msgr2=0x7faf500ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5015b730 0x7faf500ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 -- 172.21.15.70:0/1580285255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf500aac10 con 0x7faf3c004810 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf56142640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf500aa6d0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7faf400115e0 tx=0x7faf40011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4000f8f0 con 0x7faf3c004810 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4000fa90 con 0x7faf3c004810 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf500abdc0 con 0x7faf3c004810 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4000fe40 con 0x7faf3c004810 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.412+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf500a8560 con 0x7faf3c004810 2024-04-22T01:18:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.416+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf50068210 con 0x7faf3c004810 2024-04-22T01:18:15.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.416+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf40015460 con 0x7faf3c004810 2024-04-22T01:18:15.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.416+0000 7faf46ffd640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faf340678f0 0x7faf34069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.416+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7faf400863f0 con 0x7faf3c004810 2024-04-22T01:18:15.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.416+0000 7faf55941640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faf340678f0 0x7faf34069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.416+0000 7faf55941640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faf340678f0 0x7faf34069db0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0ad000 tx=0x7faf4c004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.420+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7faf40019030 con 0x7faf3c004810 2024-04-22T01:18:15.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.580+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf5015f510 con 0x7faf3c004810 2024-04-22T01:18:15.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf46ffd640 1 -- 172.21.15.70:0/1580285255 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+13256 (secure 0 0 0) 0x7faf400570a0 con 0x7faf3c004810 2024-04-22T01:18:15.585 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:15.585 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:14.255172+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6833/258015518":"2024-04-23T01:18:11.239499+0000","172.21.15.98:6832/258015518":"2024-04-23T01:18:11.239499+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faf340678f0 msgr2=0x7faf34069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faf340678f0 0x7faf34069db0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0ad000 tx=0x7faf4c004740 comp rx=0 tx=0).stop 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 msgr2=0x7faf500aa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf500aa6d0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7faf400115e0 tx=0x7faf40011aa0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 shutdown_connections 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faf340678f0 0x7faf34069db0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf500ad550 0x7faf501f8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5015b730 0x7faf500ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 --2- 172.21.15.70:0/1580285255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf3c004810 0x7faf500aa6d0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 >> 172.21.15.70:0/1580285255 conn(0x7faf5006b1f0 msgr2=0x7faf500af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 shutdown_connections 2024-04-22T01:18:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.584+0000 7faf583a6640 1 -- 172.21.15.70:0/1580285255 wait complete. 2024-04-22T01:18:15.605 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:15.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:15.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 Processor -- start 2024-04-22T01:18:15.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 -- start start 2024-04-22T01:18:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c15e6e0 0x7fba0c15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba0c0ce920 0x7fba0c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0c15f000 con 0x7fba0c0ceda0 2024-04-22T01:18:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fba0c0b3800 con 0x7fba0c0ce920 2024-04-22T01:18:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba13e66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fba0c0b3ae0 con 0x7fba0c15e6e0 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba11c02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c15e6e0 0x7fba0c15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba11401640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba0c0ce920 0x7fba0c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba12403640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba0c0ceda0 0x7fba0c06d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:37040/0 (socket says 172.21.15.70:37040) 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba12403640 1 -- 172.21.15.70:0/157000575 learned_addr learned my addr 172.21.15.70:0/157000575 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba11c02640 1 -- 172.21.15.70:0/157000575 >> v2:172.21.15.98:3300/0 conn(0x7fba0c0ce920 msgr2=0x7fba0c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba11c02640 1 --2- 172.21.15.70:0/157000575 >> v2:172.21.15.98:3300/0 conn(0x7fba0c0ce920 0x7fba0c154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba11c02640 1 -- 172.21.15.70:0/157000575 >> v1:172.21.15.98:6789/0 conn(0x7fba0c0ceda0 legacy=0x7fba0c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.732+0000 7fba11c02640 1 -- 172.21.15.70:0/157000575 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c0cde40 con 0x7fba0c15e6e0 2024-04-22T01:18:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba11c02640 1 --2- 172.21.15.70:0/157000575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c15e6e0 0x7fba0c15eac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba080097c0 tx=0x7fba0801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b473465b265630f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba10c00640 1 -- 172.21.15.70:0/157000575 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba080a7070 con 0x7fba0c15e6e0 2024-04-22T01:18:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba10c00640 1 -- 172.21.15.70:0/157000575 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba080b0070 con 0x7fba0c15e6e0 2024-04-22T01:18:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba10c00640 1 -- 172.21.15.70:0/157000575 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba080ab890 con 0x7fba0c15e6e0 2024-04-22T01:18:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 -- 172.21.15.70:0/157000575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c15e6e0 msgr2=0x7fba0c15eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 --2- 172.21.15.70:0/157000575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c15e6e0 0x7fba0c15eac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba080097c0 tx=0x7fba0801c470 comp rx=0 tx=0).stop 2024-04-22T01:18:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 -- 172.21.15.70:0/157000575 shutdown_connections 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 --2- 172.21.15.70:0/157000575 >> v2:172.21.15.98:3300/0 conn(0x7fba0c0ce920 0x7fba0c154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 --2- 172.21.15.70:0/157000575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c15e6e0 0x7fba0c15eac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 -- 172.21.15.70:0/157000575 >> 172.21.15.70:0/157000575 conn(0x7fba0c0b1d40 msgr2=0x7fba0c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 -- 172.21.15.70:0/157000575 shutdown_connections 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 -- 172.21.15.70:0/157000575 wait complete. 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 Processor -- start 2024-04-22T01:18:15.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.736+0000 7fba13e66640 1 -- start start 2024-04-22T01:18:15.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c0ce920 0x7fba0c165850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 0x7fba0c159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba0c15e6e0 0x7fba0c1589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c0b32c0 con 0x7fba0c0ceda0 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1555b0 con 0x7fba0c15e6e0 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fba0c0b3440 con 0x7fba0c0ce920 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 0x7fba0c159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 0x7fba0c159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59834/0 (socket says 172.21.15.70:59834) 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 -- 172.21.15.70:0/504253407 learned_addr learned my addr 172.21.15.70:0/504253407 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11c02640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c0ce920 0x7fba0c165850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba12403640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba0c15e6e0 0x7fba0c1589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 -- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c0ce920 msgr2=0x7fba0c165850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c0ce920 0x7fba0c165850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 -- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba0c15e6e0 msgr2=0x7fba0c1589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba0c15e6e0 0x7fba0c1589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 -- 172.21.15.70:0/504253407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c15ec90 con 0x7fba0c0ceda0 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11401640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 0x7fba0c159d00 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00e8a0 tx=0x7fb9fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc017070 con 0x7fba0c0ceda0 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9fc00c940 con 0x7fba0c0ceda0 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c0b0910 con 0x7fba0c0ceda0 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00ccf0 con 0x7fba0c0ceda0 2024-04-22T01:18:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c156500 con 0x7fba0c0ceda0 2024-04-22T01:18:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba0c068210 con 0x7fba0c0ceda0 2024-04-22T01:18:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9fc01a070 con 0x7fba0c0ceda0 2024-04-22T01:18:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba027fc640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9f00678c0 0x7fb9f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7fb9fc0860c0 con 0x7fba0c0ceda0 2024-04-22T01:18:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.740+0000 7fba11c02640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9f00678c0 0x7fb9f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.744+0000 7fba11c02640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9f00678c0 0x7fb9f0069d80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba080af000 tx=0x7fba080a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:15.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.744+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb9fc056ee0 con 0x7fba0c0ceda0 2024-04-22T01:18:15.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.908+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fba0c0d08e0 con 0x7fba0c0ceda0 2024-04-22T01:18:15.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.908+0000 7fba027fc640 1 -- 172.21.15.70:0/504253407 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb9fc0567e0 con 0x7fba0c0ceda0 2024-04-22T01:18:15.910 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.908+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9f00678c0 msgr2=0x7fb9f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.908+0000 7fba13e66640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9f00678c0 0x7fb9f0069d80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba080af000 tx=0x7fba080a7000 comp rx=0 tx=0).stop 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.908+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 msgr2=0x7fba0c159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.908+0000 7fba13e66640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 0x7fba0c159d00 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00e8a0 tx=0x7fb9fc00eca0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 shutdown_connections 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb9f00678c0 0x7fb9f0069d80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba0c15e6e0 0x7fba0c1589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba0c0ceda0 0x7fba0c159d00 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 --2- 172.21.15.70:0/504253407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fba0c0ce920 0x7fba0c165850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:15.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 >> 172.21.15.70:0/504253407 conn(0x7fba0c0b1d40 msgr2=0x7fba0c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:15.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 shutdown_connections 2024-04-22T01:18:15.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:15.912+0000 7fba13e66640 1 -- 172.21.15.70:0/504253407 wait complete. 2024-04-22T01:18:15.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr' 2024-04-22T01:18:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f19811640 1 Processor -- start 2024-04-22T01:18:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f19811640 1 -- start start 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f19811640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 0x7f2f140ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f19811640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f1415e390 con 0x7f2f140d2110 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f19811640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f14161e00 con 0x7f2f1415dfb0 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f19811640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2f140cf420 con 0x7f2f140cea80 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f137fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2f140d2110 0x7f2f140d2570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56842/0 (socket says 172.21.15.70:56842) 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f137fe640 1 -- 172.21.15.70:0/3580624268 learned_addr learned my addr 172.21.15.70:0/3580624268 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 0x7f2f140ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372149363 0 0) 0x7f2f1415e390 con 0x7f2f140d2110 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2efc003530 con 0x7f2f140d2110 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3915418548 0 0) 0x7f2efc003530 con 0x7f2f140d2110 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 msgr2=0x7f2f140ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:16.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 0x7f2f140ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 >> v1:172.21.15.98:6789/0 conn(0x7f2f1415dfb0 legacy=0x7f2f140b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f141620e0 con 0x7f2f140d2110 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f2f0c001a00 con 0x7f2f140d2110 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 >> v1:172.21.15.70:6789/0 conn(0x7f2f140d2110 legacy=0x7f2f140d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2efc0039f0 0x7f2efc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2efc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 0x7f2efc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc00ba80 con 0x7f2efc004810 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc00bc00 con 0x7f2efc0039f0 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2efc00be00 con 0x7f2f140cea80 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2efc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f137fe640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 0x7f2efc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f12ffd640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2efc0039f0 0x7f2efc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 -- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 msgr2=0x7f2efc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:16.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cea80 0x7f2efc0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:16.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 -- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2efc0039f0 msgr2=0x7f2efc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:16.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2efc0039f0 0x7f2efc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:16.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f127fc640 1 -- 172.21.15.70:0/3580624268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2efc003530 con 0x7f2efc004810 2024-04-22T01:18:16.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.044+0000 7f2f12ffd640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2efc0039f0 0x7f2efc003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f127fc640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2efc004c90 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009a90 tx=0x7f2f0801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232e25ae489f95ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080b2070 con 0x7f2efc004810 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f080a5600 con 0x7f2efc004810 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f11ffb640 1 -- 172.21.15.70:0/3580624268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08009690 con 0x7f2efc004810 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 -- 172.21.15.70:0/3580624268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 msgr2=0x7f2efc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 --2- 172.21.15.70:0/3580624268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2efc004c90 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009a90 tx=0x7f2f0801ca10 comp rx=0 tx=0).stop 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f137fe640 1 -- 172.21.15.70:0/3580624268 reap_dead start 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 -- 172.21.15.70:0/3580624268 shutdown_connections 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 -- 172.21.15.70:0/3580624268 >> 172.21.15.70:0/3580624268 conn(0x7f2f1406c9b0 msgr2=0x7f2f1406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:16.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 -- 172.21.15.70:0/3580624268 shutdown_connections 2024-04-22T01:18:16.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 -- 172.21.15.70:0/3580624268 wait complete. 2024-04-22T01:18:16.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 Processor -- start 2024-04-22T01:18:16.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 -- start start 2024-04-22T01:18:16.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.048+0000 7f2f19811640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2f1414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1415b1d0 0x7f2f140cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cd9c0 0x7f2f140d2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f14161a90 con 0x7f2efc004810 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f141626a0 con 0x7f2f1415b1d0 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2f140ceee0 con 0x7f2f140cd9c0 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2f1414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2f1414e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59844/0 (socket says 172.21.15.70:59844) 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 -- 172.21.15.70:0/794217752 learned_addr learned my addr 172.21.15.70:0/794217752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f127fc640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1415b1d0 0x7f2f140cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f137fe640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cd9c0 0x7f2f140d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 -- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cd9c0 msgr2=0x7f2f140d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cd9c0 0x7f2f140d2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 -- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1415b1d0 msgr2=0x7f2f140cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1415b1d0 0x7f2f140cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 -- 172.21.15.70:0/794217752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f140cc920 con 0x7f2efc004810 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f137fe640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cd9c0 0x7f2f140d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:16.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f12ffd640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2f1414e050 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400d7a0 tx=0x7f2f0400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:16.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04005270 con 0x7f2efc004810 2024-04-22T01:18:16.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f04017070 con 0x7f2efc004810 2024-04-22T01:18:16.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f140ccc00 con 0x7f2efc004810 2024-04-22T01:18:16.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0400b6b0 con 0x7f2efc004810 2024-04-22T01:18:16.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f140cd1c0 con 0x7f2efc004810 2024-04-22T01:18:16.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f14068210 con 0x7f2efc004810 2024-04-22T01:18:16.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2f0400b850 con 0x7f2efc004810 2024-04-22T01:18:16.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2efbfff640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f000678c0 0x7f2f00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:16.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7f2f04086540 con 0x7f2efc004810 2024-04-22T01:18:16.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f127fc640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f000678c0 0x7f2f00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:16.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.052+0000 7f2f127fc640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f000678c0 0x7f2f00069d80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2f080af000 tx=0x7f2f080046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:16.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.056+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2f04057260 con 0x7f2efc004810 2024-04-22T01:18:16.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:16.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]} v 0) -- 0x7f2f140b6970 con 0x7f2efc004810 2024-04-22T01:18:17.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.268+0000 7f2efbfff640 1 -- 172.21.15.70:0/794217752 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]}]=0 v212) ==== 143+0+0 (secure 0 0 0) 0x7f2f04056b60 con 0x7f2efc004810 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f000678c0 msgr2=0x7f2f00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f000678c0 0x7f2f00069d80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2f080af000 tx=0x7f2f080046d0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 msgr2=0x7f2f1414e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2f1414e050 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400d7a0 tx=0x7f2f0400dc60 comp rx=0 tx=0).stop 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 shutdown_connections 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2f000678c0 0x7f2f00069d80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f140cd9c0 0x7f2f140d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1415b1d0 0x7f2f140cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 --2- 172.21.15.70:0/794217752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2efc004810 0x7f2f1414e050 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 >> 172.21.15.70:0/794217752 conn(0x7f2f1406c9b0 msgr2=0x7f2f140cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:17.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 shutdown_connections 2024-04-22T01:18:17.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.272+0000 7f2f19811640 1 -- 172.21.15.70:0/794217752 wait complete. 2024-04-22T01:18:17.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-22T01:18:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c2c47640 1 Processor -- start 2024-04-22T01:18:17.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c2c47640 1 -- start start 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c2c47640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb1bc15e140 0x7fb1bc0cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c2c47640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1bc15e520 con 0x7fb1bc0cf1c0 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c2c47640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1bc0b8d70 con 0x7fb1bc0d1400 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c2c47640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb1bc0b9330 con 0x7fb1bc15e140 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c09e3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb1bc15e140 0x7fb1bc0cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1bbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb1bc0cf1c0 0x7fb1bc0d0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43334/0 (socket says 172.21.15.70:43334) 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1bbfff640 1 -- 172.21.15.70:0/3449498788 learned_addr learned my addr 172.21.15.70:0/3449498788 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:17.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c09e3640 1 --2- 172.21.15.70:0/3449498788 >> v2:172.21.15.70:3300/0 conn(0x7fb1bc15e140 0x7fb1bc0cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55070/0 (socket says 172.21.15.70:55070) 2024-04-22T01:18:17.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c09e3640 1 -- 172.21.15.70:0/3449498788 >> v1:172.21.15.98:6789/0 conn(0x7fb1bc0d1400 legacy=0x7fb1bc0d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:17.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c09e3640 1 -- 172.21.15.70:0/3449498788 >> v1:172.21.15.70:6789/0 conn(0x7fb1bc0cf1c0 legacy=0x7fb1bc0d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1c09e3640 1 -- 172.21.15.70:0/3449498788 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc0b9050 con 0x7fb1bc15e140 2024-04-22T01:18:17.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.416+0000 7fb1bb7fe640 1 -- 172.21.15.70:0/3449498788 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4178864511 0 0) 0x7fb1bc15e520 con 0x7fb1bc0cf1c0 2024-04-22T01:18:17.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c09e3640 1 --2- 172.21.15.70:0/3449498788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc0cec80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0097c0 tx=0x7fb1ac01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cbffd9ace476803 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:17.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1bb7fe640 1 -- 172.21.15.70:0/3449498788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac0b0070 con 0x7fb1bc15e140 2024-04-22T01:18:17.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1bb7fe640 1 -- 172.21.15.70:0/3449498788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1ac0ab600 con 0x7fb1bc15e140 2024-04-22T01:18:17.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1bb7fe640 1 -- 172.21.15.70:0/3449498788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac0ab9d0 con 0x7fb1bc15e140 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- 172.21.15.70:0/3449498788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 msgr2=0x7fb1bc0cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/3449498788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc0cec80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0097c0 tx=0x7fb1ac01c3d0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- 172.21.15.70:0/3449498788 shutdown_connections 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/3449498788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc0cec80 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- 172.21.15.70:0/3449498788 >> 172.21.15.70:0/3449498788 conn(0x7fb1bc072d20 msgr2=0x7fb1bc073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- 172.21.15.70:0/3449498788 shutdown_connections 2024-04-22T01:18:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- 172.21.15.70:0/3449498788 wait complete. 2024-04-22T01:18:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 Processor -- start 2024-04-22T01:18:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- start start 2024-04-22T01:18:17.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1bc0cf1c0 0x7fb1bc0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1bc0d1400 0x7fb1bc0b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc0d3ae0 con 0x7fb1bc15e140 2024-04-22T01:18:17.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc0d3960 con 0x7fb1bc0d1400 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c2c47640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb1bc16bc80 con 0x7fb1bc0cf1c0 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55080/0 (socket says 172.21.15.70:55080) 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 -- 172.21.15.70:0/2051212852 learned_addr learned my addr 172.21.15.70:0/2051212852 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c09e3640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1bc0cf1c0 0x7fb1bc0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1bbfff640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1bc0d1400 0x7fb1bc0b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 -- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1bc0cf1c0 msgr2=0x7fb1bc0b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1bc0cf1c0 0x7fb1bc0b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 -- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1bc0d1400 msgr2=0x7fb1bc0b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1bc0d1400 0x7fb1bc0b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 -- 172.21.15.70:0/2051212852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc158b20 con 0x7fb1bc15e140 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c09e3640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1bc0cf1c0 0x7fb1bc0b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1bbfff640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1bc0d1400 0x7fb1bc0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.420+0000 7fb1c11e4640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc157da0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400c780 tx=0x7fb1b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4016020 con 0x7fb1bc15e140 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b400aa90 con 0x7fb1bc15e140 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b400ce30 con 0x7fb1bc15e140 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1bc158e00 con 0x7fb1bc15e140 2024-04-22T01:18:17.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1bc153270 con 0x7fb1bc15e140 2024-04-22T01:18:17.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1bc153470 con 0x7fb1bc15e140 2024-04-22T01:18:17.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb1b4019020 con 0x7fb1bc15e140 2024-04-22T01:18:17.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1b97fa640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1a80678c0 0x7fb1a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7fb1b4085e00 con 0x7fb1bc15e140 2024-04-22T01:18:17.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1c09e3640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1a80678c0 0x7fb1a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.424+0000 7fb1c09e3640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1a80678c0 0x7fb1a8069d80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0ad000 tx=0x7fb1ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:17.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.428+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb1bc153470 con 0x7fb1bc15e140 2024-04-22T01:18:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.584+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb1bc068920 con 0x7fb1bc15e140 2024-04-22T01:18:17.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1b97fa640 1 -- 172.21.15.70:0/2051212852 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fb1b4020020 con 0x7fb1bc15e140 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1a80678c0 msgr2=0x7fb1a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1a80678c0 0x7fb1a8069d80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0ad000 tx=0x7fb1ac008040 comp rx=0 tx=0).stop 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 msgr2=0x7fb1bc157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc157da0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400c780 tx=0x7fb1b400cc40 comp rx=0 tx=0).stop 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 shutdown_connections 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1a80678c0 0x7fb1a8069d80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1bc15e140 0x7fb1bc157da0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1bc0d1400 0x7fb1bc0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 --2- 172.21.15.70:0/2051212852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1bc0cf1c0 0x7fb1bc0b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 >> 172.21.15.70:0/2051212852 conn(0x7fb1bc072d20 msgr2=0x7fb1bc0c70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 shutdown_connections 2024-04-22T01:18:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.592+0000 7fb1c2c47640 1 -- 172.21.15.70:0/2051212852 wait complete. 2024-04-22T01:18:17.611 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:18:17.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:17.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:17.704 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:18:17.704 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:17.704 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:17.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:18:17.709 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:17.709 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:17.715 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:18:17.715 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:17.715 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:17.757 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:18:17.757 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:17.758 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:17.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:18:17.766 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-22T01:18:17.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 Processor -- start 2024-04-22T01:18:17.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 -- start start 2024-04-22T01:18:17.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7facdc15be40 0x7facdc0673a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7facdc0ad7e0 0x7facdc06b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facdc0cffd0 con 0x7facdc0b5d20 2024-04-22T01:18:17.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7facdc0678e0 con 0x7facdc15be40 2024-04-22T01:18:17.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face462c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7facdc0cfd80 con 0x7facdc0ad7e0 2024-04-22T01:18:17.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7facdc15be40 0x7facdc0673a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7facdc15be40 0x7facdc0673a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55092/0 (socket says 172.21.15.70:55092) 2024-04-22T01:18:17.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 -- 172.21.15.70:0/497627624 learned_addr learned my addr 172.21.15.70:0/497627624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:17.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face2bc9640 1 --2- 172.21.15.70:0/497627624 >> v2:172.21.15.98:3300/0 conn(0x7facdc0ad7e0 0x7facdc06b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 -- 172.21.15.70:0/497627624 >> v2:172.21.15.98:3300/0 conn(0x7facdc0ad7e0 msgr2=0x7facdc06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:17.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 --2- 172.21.15.70:0/497627624 >> v2:172.21.15.98:3300/0 conn(0x7facdc0ad7e0 0x7facdc06b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 -- 172.21.15.70:0/497627624 >> v1:172.21.15.98:6789/0 conn(0x7facdc0b5d20 legacy=0x7facdc0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:17.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.924+0000 7face23c8640 1 -- 172.21.15.70:0/497627624 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facdc0d02b0 con 0x7facdc15be40 2024-04-22T01:18:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face23c8640 1 --2- 172.21.15.70:0/497627624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc15be40 0x7facdc0673a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7facdc0b5000 tx=0x7facd801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d316e8edbddd3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face13c6640 1 -- 172.21.15.70:0/497627624 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80ab5f0 con 0x7facdc15be40 2024-04-22T01:18:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face13c6640 1 -- 172.21.15.70:0/497627624 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd80b0070 con 0x7facdc15be40 2024-04-22T01:18:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face13c6640 1 -- 172.21.15.70:0/497627624 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80ab9b0 con 0x7facdc15be40 2024-04-22T01:18:17.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 -- 172.21.15.70:0/497627624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc15be40 msgr2=0x7facdc0673a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 --2- 172.21.15.70:0/497627624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc15be40 0x7facdc0673a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7facdc0b5000 tx=0x7facd801c4e0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 -- 172.21.15.70:0/497627624 shutdown_connections 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 --2- 172.21.15.70:0/497627624 >> v2:172.21.15.98:3300/0 conn(0x7facdc0ad7e0 0x7facdc06b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 --2- 172.21.15.70:0/497627624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc15be40 0x7facdc0673a0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 -- 172.21.15.70:0/497627624 >> 172.21.15.70:0/497627624 conn(0x7facdc099a60 msgr2=0x7facdc0b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 -- 172.21.15.70:0/497627624 shutdown_connections 2024-04-22T01:18:17.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 -- 172.21.15.70:0/497627624 wait complete. 2024-04-22T01:18:17.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 Processor -- start 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 -- start start 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facdc0ad7e0 0x7facdc14ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 0x7facdc15ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.928+0000 7face462c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facdc15be40 0x7facdc151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face462c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7facdc06c2d0 con 0x7facdc0b5d20 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face462c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facdc06c150 con 0x7facdc15be40 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face462c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7facdc06c450 con 0x7facdc0ad7e0 2024-04-22T01:18:17.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 0x7facdc15ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 0x7facdc15ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55108/0 (socket says 172.21.15.70:55108) 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 -- 172.21.15.70:0/1357288944 learned_addr learned my addr 172.21.15.70:0/1357288944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face2bc9640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facdc15be40 0x7facdc151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face23c8640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facdc0ad7e0 0x7facdc14ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 -- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facdc0ad7e0 msgr2=0x7facdc14ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facdc0ad7e0 0x7facdc14ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 -- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facdc15be40 msgr2=0x7facdc151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:17.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facdc15be40 0x7facdc151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 -- 172.21.15.70:0/1357288944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facdc15c9d0 con 0x7facdc0b5d20 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face1bc7640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 0x7facdc15ede0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7faccc00c450 tx=0x7faccc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc012830 con 0x7facdc0b5d20 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faccc017070 con 0x7facdc0b5d20 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc012c30 con 0x7facdc0b5d20 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facdc15c6f0 con 0x7facdc0b5d20 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facdc15fe60 con 0x7facdc0b5d20 2024-04-22T01:18:17.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facdc068210 con 0x7facdc0b5d20 2024-04-22T01:18:17.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faccc01c460 con 0x7facdc0b5d20 2024-04-22T01:18:17.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7facd2ffd640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facc00678c0 0x7facc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:17.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7faccc0862e0 con 0x7facdc0b5d20 2024-04-22T01:18:17.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face23c8640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facc00678c0 0x7facc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:17.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.932+0000 7face23c8640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facc00678c0 0x7facc0069d80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7facd80af000 tx=0x7facd80a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:17.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:17.936+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7faccc057100 con 0x7facdc0b5d20 2024-04-22T01:18:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.132+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facdc1601d0 con 0x7facdc0b5d20 2024-04-22T01:18:18.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.132+0000 7facd2ffd640 1 -- 172.21.15.70:0/1357288944 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+3863 (secure 0 0 0) 0x7faccc056a00 con 0x7facdc0b5d20 2024-04-22T01:18:18.135 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:18.135 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5171,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/826699598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":826699598},{"type":"v1","addr":"172.21.15.70:6837","nonce":826699598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/828618366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":828618366},{"type":"v1","addr":"172.21.15.98:6835","nonce":828618366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5761,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1090906452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1090906452},{"type":"v1","addr":"172.21.15.98:6833","nonce":1090906452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/4259862680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":4259862680},{"type":"v1","addr":"172.21.15.70:6835","nonce":4259862680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[]} 2024-04-22T01:18:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 39 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facc00678c0 msgr2=0x7facc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facc00678c0 0x7facc0069d80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7facd80af000 tx=0x7facd80a7000 comp rx=0 tx=0).stop 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 msgr2=0x7facdc15ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 0x7facdc15ede0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7faccc00c450 tx=0x7faccc00c910 comp rx=0 tx=0).stop 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 shutdown_connections 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facc00678c0 0x7facc0069d80 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facdc15be40 0x7facdc151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facdc0b5d20 0x7facdc15ede0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 --2- 172.21.15.70:0/1357288944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facdc0ad7e0 0x7facdc14ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 >> 172.21.15.70:0/1357288944 conn(0x7facdc099a60 msgr2=0x7facdc155020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 shutdown_connections 2024-04-22T01:18:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.136+0000 7face462c640 1 -- 172.21.15.70:0/1357288944 wait complete. 2024-04-22T01:18:18.155 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:18:18.155 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-22T01:18:18.156 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:18:18.156 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:18.156 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:18:18.156 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-22T01:18:18.157 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:18:18.157 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:18.157 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:18:18.157 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-22T01:18:18.158 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:18:18.158 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:18.158 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:18:18.158 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-22T01:18:18.159 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:18:18.159 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:18.170 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:18:18.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:18:18.172 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:18:18.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:18:18.174 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:18:18.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:18:18.175 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:18:18.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:18:18.215 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:18:18.215 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:18.216 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:18:18.216 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:18.219 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:18:18.219 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:18.220 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:18:18.221 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:18.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-22T01:18:18.277 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:18:18.281 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:18:18.293 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:18:18.294 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:18:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 Processor -- start 2024-04-22T01:18:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 -- start start 2024-04-22T01:18:18.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffbac0d17b0 0x7ffbac0d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:18.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0ce880 0x7ffbac0d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:18.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac0cee40 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ffbac15c150 con 0x7ffbac0d17b0 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbb0886640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffbac0cf400 con 0x7ffbac0ce880 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbaa59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffbac0d17b0 0x7ffbac0d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffba1d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ffbac0ce3f0 0x7ffbac0cc610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43354/0 (socket says 172.21.15.70:43354) 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffba1d9b640 1 -- 172.21.15.70:0/2181475614 learned_addr learned my addr 172.21.15.70:0/2181475614 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbaad9d640 1 --2- 172.21.15.70:0/2181475614 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0ce880 0x7ffbac0d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbaa59c640 1 -- 172.21.15.70:0/2181475614 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0ce880 msgr2=0x7ffbac0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbaa59c640 1 --2- 172.21.15.70:0/2181475614 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0ce880 0x7ffbac0d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbaa59c640 1 -- 172.21.15.70:0/2181475614 >> v1:172.21.15.70:6789/0 conn(0x7ffbac0ce3f0 legacy=0x7ffbac0cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:18.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffbaa59c640 1 -- 172.21.15.70:0/2181475614 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac0cf120 con 0x7ffbac0d17b0 2024-04-22T01:18:18.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.344+0000 7ffba9d9b640 1 -- 172.21.15.70:0/2181475614 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940751426 0 0) 0x7ffbac0cee40 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 --2- 172.21.15.70:0/2181475614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0d17b0 0x7ffbac0d1b90 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0097c0 tx=0x7ffb9c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e566b2cd5d3253b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffba9d9b640 1 -- 172.21.15.70:0/2181475614 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0b0070 con 0x7ffbac0d17b0 2024-04-22T01:18:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffba9d9b640 1 -- 172.21.15.70:0/2181475614 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb9c0ab600 con 0x7ffbac0d17b0 2024-04-22T01:18:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffba9d9b640 1 -- 172.21.15.70:0/2181475614 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0ab9d0 con 0x7ffbac0d17b0 2024-04-22T01:18:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- 172.21.15.70:0/2181475614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0d17b0 msgr2=0x7ffbac0d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:18.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 --2- 172.21.15.70:0/2181475614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0d17b0 0x7ffbac0d1b90 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0097c0 tx=0x7ffb9c01c410 comp rx=0 tx=0).stop 2024-04-22T01:18:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- 172.21.15.70:0/2181475614 shutdown_connections 2024-04-22T01:18:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 --2- 172.21.15.70:0/2181475614 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0ce880 0x7ffbac0d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 --2- 172.21.15.70:0/2181475614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0d17b0 0x7ffbac0d1b90 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- 172.21.15.70:0/2181475614 >> 172.21.15.70:0/2181475614 conn(0x7ffbac08b1b0 msgr2=0x7ffbac08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- 172.21.15.70:0/2181475614 shutdown_connections 2024-04-22T01:18:18.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- 172.21.15.70:0/2181475614 wait complete. 2024-04-22T01:18:18.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 Processor -- start 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- start start 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 0x7ffbac153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbac0ce880 0x7ffbac0aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0d17b0 0x7ffbac0af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac0ccea0 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac0ccd20 con 0x7ffbac0ce880 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbb0886640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffbac15f980 con 0x7ffbac0d17b0 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 0x7ffbac153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 0x7ffbac153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55138/0 (socket says 172.21.15.70:55138) 2024-04-22T01:18:18.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 -- 172.21.15.70:0/4005957648 learned_addr learned my addr 172.21.15.70:0/4005957648 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffba1d9b640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbac0ce880 0x7ffbac0aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaad9d640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0d17b0 0x7ffbac0af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 -- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0d17b0 msgr2=0x7ffbac0af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0d17b0 0x7ffbac0af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 -- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbac0ce880 msgr2=0x7ffbac0aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbac0ce880 0x7ffbac0aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 -- 172.21.15.70:0/4005957648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac1548b0 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.348+0000 7ffbaa59c640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 0x7ffbac153de0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0ae040 tx=0x7ffb9c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0b0050 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb9c0bc070 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac0afb00 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0b5870 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac0b00c0 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbac04f510 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb9c0b5a10 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba37fe640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb8c0678c0 0x7ffb8c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:18.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5335+0+0 (secure 0 0 0) 0x7ffb9c1201c0 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba1d9b640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb8c0678c0 0x7ffb8c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.352+0000 7ffba1d9b640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb8c0678c0 0x7ffb8c069d80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffbac071cc0 tx=0x7ffb94007df0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:18.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.356+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ffb9c016090 con 0x7ffbac0ce3f0 2024-04-22T01:18:18.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:18.516+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ffbac166960 con 0x7ffbac0ce3f0 2024-04-22T01:18:19.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.288+0000 7ffba37fe640 1 -- 172.21.15.70:0/4005957648 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v83) ==== 97+0+0 (secure 0 0 0) 0x7ffb9c0f0790 con 0x7ffbac0ce3f0 2024-04-22T01:18:19.292 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-22T01:18:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb8c0678c0 msgr2=0x7ffb8c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb8c0678c0 0x7ffb8c069d80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffbac071cc0 tx=0x7ffb94007df0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 msgr2=0x7ffbac153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 0x7ffbac153de0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0ae040 tx=0x7ffb9c01ac50 comp rx=0 tx=0).stop 2024-04-22T01:18:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 shutdown_connections 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffb8c0678c0 0x7ffb8c069d80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbac0d17b0 0x7ffbac0af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbac0ce880 0x7ffbac0aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 --2- 172.21.15.70:0/4005957648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbac0ce3f0 0x7ffbac153de0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 >> 172.21.15.70:0/4005957648 conn(0x7ffbac08b1b0 msgr2=0x7ffbac0b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 shutdown_connections 2024-04-22T01:18:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.292+0000 7ffbb0886640 1 -- 172.21.15.70:0/4005957648 wait complete. 2024-04-22T01:18:19.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-22T01:18:19.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd24a6640 1 Processor -- start 2024-04-22T01:18:19.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd24a6640 1 -- start start 2024-04-22T01:18:19.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd24a6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbbcc0ceee0 0x7fbbcc0d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd24a6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbcc15e390 con 0x7fbbcc0cea80 2024-04-22T01:18:19.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd24a6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbcc161e00 con 0x7fbbcc15dfb0 2024-04-22T01:18:19.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd24a6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbbcc1620e0 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbcb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbbcc0cea80 0x7fbbcc0d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43370/0 (socket says 172.21.15.70:43370) 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbcb7fe640 1 -- 172.21.15.70:0/4277348611 learned_addr learned my addr 172.21.15.70:0/4277348611 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd0a43640 1 --2- 172.21.15.70:0/4277348611 >> v2:172.21.15.70:3300/0 conn(0x7fbbcc0ceee0 0x7fbbcc0d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd0a43640 1 -- 172.21.15.70:0/4277348611 >> v1:172.21.15.98:6789/0 conn(0x7fbbcc15dfb0 legacy=0x7fbbcc0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbcaffd640 1 -- 172.21.15.70:0/4277348611 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415050038 0 0) 0x7fbbcc15e390 con 0x7fbbcc0cea80 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd0a43640 1 -- 172.21.15.70:0/4277348611 >> v1:172.21.15.70:6789/0 conn(0x7fbbcc0cea80 legacy=0x7fbbcc0d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.432+0000 7fbbd0a43640 1 -- 172.21.15.70:0/4277348611 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc1623c0 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd0a43640 1 --2- 172.21.15.70:0/4277348611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc0d2820 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009d90 tx=0x7fbbbc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b589bc95f7fe2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:19.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbcaffd640 1 -- 172.21.15.70:0/4277348611 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0b2070 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbcaffd640 1 -- 172.21.15.70:0/4277348611 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbbc0a5940 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbcaffd640 1 -- 172.21.15.70:0/4277348611 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0a5d10 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- 172.21.15.70:0/4277348611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 msgr2=0x7fbbcc0d2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/4277348611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc0d2820 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009d90 tx=0x7fbbbc01c970 comp rx=0 tx=0).stop 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- 172.21.15.70:0/4277348611 shutdown_connections 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/4277348611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc0d2820 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- 172.21.15.70:0/4277348611 >> 172.21.15.70:0/4277348611 conn(0x7fbbcc06c9b0 msgr2=0x7fbbcc06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- 172.21.15.70:0/4277348611 shutdown_connections 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- 172.21.15.70:0/4277348611 wait complete. 2024-04-22T01:18:19.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 Processor -- start 2024-04-22T01:18:19.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- start start 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbcc0cea80 0x7fbbcc0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbcc15dfb0 0x7fbbcc0b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc16bf60 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.436+0000 7fbbd24a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc16bde0 con 0x7fbbcc15dfb0 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbd24a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbbcc16c0e0 con 0x7fbbcc0cea80 2024-04-22T01:18:19.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55152/0 (socket says 172.21.15.70:55152) 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 -- 172.21.15.70:0/433530981 learned_addr learned my addr 172.21.15.70:0/433530981 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbd0a43640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbcc15dfb0 0x7fbbcc0b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcbfff640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbcc0cea80 0x7fbbcc0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 -- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbcc0cea80 msgr2=0x7fbbcc0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbcc0cea80 0x7fbbcc0cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 -- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbcc15dfb0 msgr2=0x7fbbcc0b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbcc15dfb0 0x7fbbcc0b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 -- 172.21.15.70:0/433530981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc0cd490 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcb7fe640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc152220 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000c440 tx=0x7fbbc000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0018070 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc00074e0 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc0cd770 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0004a10 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc0cdd30 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc068210 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbbc001b070 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbc8ff9640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbbb80678f0 0x7fbbb8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5249+0+0 (secure 0 0 0) 0x7fbbc0085050 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcbfff640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbbb80678f0 0x7fbbb8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.440+0000 7fbbcbfff640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbbb80678f0 0x7fbbb8069db0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4007fd0 tx=0x7fbbc4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:19.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.444+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbbc0055e30 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fbbcc0d1950 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbc8ff9640 1 -- 172.21.15.70:0/433530981 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7fbbc0055730 con 0x7fbbcc0ceee0 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw==", 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w==", 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ==", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA==", 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA==", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA==", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg==", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg==", 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-22T01:18:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong==", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ==", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA==", 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ==", 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw==", 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow r", 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ==", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA==", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:18:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw==", 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm2O5GBRAA/JxaogwD4Hjmtr9eQvMY0g==", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw==", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:18:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg==", 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-22T01:18:19.678 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:18:19.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbbb80678f0 msgr2=0x7fbbb8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbbb80678f0 0x7fbbb8069db0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4007fd0 tx=0x7fbbc4009040 comp rx=0 tx=0).stop 2024-04-22T01:18:19.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 msgr2=0x7fbbcc152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc152220 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000c440 tx=0x7fbbc000c900 comp rx=0 tx=0).stop 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 shutdown_connections 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbbb80678f0 0x7fbbb8069db0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbcc15dfb0 0x7fbbcc0b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbcc0ceee0 0x7fbbcc152220 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 --2- 172.21.15.70:0/433530981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbcc0cea80 0x7fbbcc0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 >> 172.21.15.70:0/433530981 conn(0x7fbbcc06c9b0 msgr2=0x7fbbcc0b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 shutdown_connections 2024-04-22T01:18:19.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.664+0000 7fbbd24a6640 1 -- 172.21.15.70:0/433530981 wait complete. 2024-04-22T01:18:19.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:19.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.796+0000 7fd6b366e640 1 Processor -- start 2024-04-22T01:18:19.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.796+0000 7fd6b366e640 1 -- start start 2024-04-22T01:18:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b366e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 0x7fd6ac0b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b366e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ac0b6550 con 0x7fd6ac0d3120 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b366e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ac15e390 con 0x7fd6ac0cea80 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b366e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac161e00 con 0x7fd6ac15dfb0 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b1c0b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd6ac0d3120 0x7fd6ac0b5e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43380/0 (socket says 172.21.15.70:43380) 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b1c0b640 1 -- 172.21.15.70:0/2717706050 learned_addr learned my addr 172.21.15.70:0/2717706050 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297277469 0 0) 0x7fd6ac0b6550 con 0x7fd6ac0d3120 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b140a640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 0x7fd6ac0b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd698003530 con 0x7fd6ac0d3120 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4202034669 0 0) 0x7fd698003530 con 0x7fd6ac0d3120 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 msgr2=0x7fd6ac0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 0x7fd6ac0b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 >> v1:172.21.15.98:6789/0 conn(0x7fd6ac0cea80 legacy=0x7fd6ac0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac1620e0 con 0x7fd6ac0d3120 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fd6a4001a00 con 0x7fd6ac0d3120 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 >> v1:172.21.15.70:6789/0 conn(0x7fd6ac0d3120 legacy=0x7fd6ac0b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 0x7fd6980039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd698004020 0x7fd698004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd698004d50 0x7fd6980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac161e00 con 0x7fd698004020 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd69800bb90 con 0x7fd698004d50 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd69800bd90 con 0x7fd6ac15dfb0 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd698004020 0x7fd698004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b140a640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 0x7fd6980039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b1c0b640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd698004d50 0x7fd6980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b0c09640 1 -- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 msgr2=0x7fd6980039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6ac15dfb0 0x7fd6980039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b0c09640 1 -- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd698004d50 msgr2=0x7fd6980051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd698004d50 0x7fd6980051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.800+0000 7fd6b0c09640 1 -- 172.21.15.70:0/2717706050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd698003530 con 0x7fd698004020 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd698004020 0x7fd698004810 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8001780 tx=0x7fd6a801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987f05369464cd38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a80013a0 con 0x7fd698004020 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a80a9070 con 0x7fd698004020 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6a3fff640 1 -- 172.21.15.70:0/2717706050 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a80052d0 con 0x7fd698004020 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- 172.21.15.70:0/2717706050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd698004020 msgr2=0x7fd698004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 --2- 172.21.15.70:0/2717706050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd698004020 0x7fd698004810 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8001780 tx=0x7fd6a801c420 comp rx=0 tx=0).stop 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b1c0b640 1 -- 172.21.15.70:0/2717706050 reap_dead start 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- 172.21.15.70:0/2717706050 shutdown_connections 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- 172.21.15.70:0/2717706050 >> 172.21.15.70:0/2717706050 conn(0x7fd6ac06c9b0 msgr2=0x7fd6ac06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- 172.21.15.70:0/2717706050 shutdown_connections 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- 172.21.15.70:0/2717706050 wait complete. 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 Processor -- start 2024-04-22T01:18:19.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- start start 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd698004020 0x7fd6ac1f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 0x7fd6ac0b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6ac0b4f80 0x7fd6ac1f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac161a90 con 0x7fd6ac1f7030 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac1626a0 con 0x7fd6ac0b4f80 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac0ceee0 con 0x7fd698004020 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 0x7fd6ac0b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 0x7fd6ac0b4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55166/0 (socket says 172.21.15.70:55166) 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 -- 172.21.15.70:0/542033090 learned_addr learned my addr 172.21.15.70:0/542033090 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:19.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b140a640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd698004020 0x7fd6ac1f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b1c0b640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6ac0b4f80 0x7fd6ac1f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 -- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd698004020 msgr2=0x7fd6ac1f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd698004020 0x7fd6ac1f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 -- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6ac0b4f80 msgr2=0x7fd6ac1f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6ac0b4f80 0x7fd6ac1f7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 -- 172.21.15.70:0/542033090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac1f8680 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b0c09640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 0x7fd6ac0b4a40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8005040 tx=0x7fd698005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8007a40 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a8007040 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac15ff10 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.804+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a80071f0 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac1604d0 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ac15fd00 con 0x7fd6ac1f7030 2024-04-22T01:18:19.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6a80be020 con 0x7fd6ac1f7030 2024-04-22T01:18:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6a1ffb640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6900678f0 0x7fd690069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5249+0+0 (secure 0 0 0) 0x7fd6a811ef90 con 0x7fd6ac1f7030 2024-04-22T01:18:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6b140a640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6900678f0 0x7fd690069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:19.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.808+0000 7fd6b140a640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6900678f0 0x7fd690069db0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00b9b0 tx=0x7fd69c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:19.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.812+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd6a80c6060 con 0x7fd6ac1f7030 2024-04-22T01:18:19.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.972+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6ac068210 con 0x7fd6ac1f7030 2024-04-22T01:18:19.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6a1ffb640 1 -- 172.21.15.70:0/542033090 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+13129 (secure 0 0 0) 0x7fd6a8016130 con 0x7fd6ac1f7030 2024-04-22T01:18:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:19.978 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":85,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:19.306487+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:19.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6900678f0 msgr2=0x7fd690069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6900678f0 0x7fd690069db0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00b9b0 tx=0x7fd69c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 msgr2=0x7fd6ac0b4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 0x7fd6ac0b4a40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8005040 tx=0x7fd698005840 comp rx=0 tx=0).stop 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 shutdown_connections 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd6900678f0 0x7fd690069db0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6ac0b4f80 0x7fd6ac1f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6ac1f7030 0x7fd6ac0b4a40 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 --2- 172.21.15.70:0/542033090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd698004020 0x7fd6ac1f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 >> 172.21.15.70:0/542033090 conn(0x7fd6ac06c9b0 msgr2=0x7fd6ac0b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 shutdown_connections 2024-04-22T01:18:19.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:19.976+0000 7fd6b366e640 1 -- 172.21.15.70:0/542033090 wait complete. 2024-04-22T01:18:19.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:19.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:20.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.120+0000 7f6ed5d81640 1 Processor -- start 2024-04-22T01:18:20.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.120+0000 7f6ed5d81640 1 -- start start 2024-04-22T01:18:20.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ed5d81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ed00cd220 0x7f6ed0166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ed5d81640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed015c290 con 0x7f6ed0072280 2024-04-22T01:18:20.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ed5d81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed015c570 con 0x7f6ed00b4450 2024-04-22T01:18:20.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ed5d81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6ed00b3a50 con 0x7f6ed00cd220 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6eceffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6ed0072280 0x7f6ed00ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43390/0 (socket says 172.21.15.70:43390) 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6eceffd640 1 -- 172.21.15.70:0/2324015306 learned_addr learned my addr 172.21.15.70:0/2324015306 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecffff640 1 --2- 172.21.15.70:0/2324015306 >> v2:172.21.15.98:3300/0 conn(0x7f6ed00cd220 0x7f6ed0166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192171 0 0) 0x7f6ed015c290 con 0x7f6ed0072280 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eb8003530 con 0x7f6ed0072280 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2371317653 0 0) 0x7f6eb8003530 con 0x7f6ed0072280 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 >> v2:172.21.15.98:3300/0 conn(0x7f6ed00cd220 msgr2=0x7f6ed0166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 --2- 172.21.15.70:0/2324015306 >> v2:172.21.15.98:3300/0 conn(0x7f6ed00cd220 0x7f6ed0166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 >> v1:172.21.15.98:6789/0 conn(0x7f6ed00b4450 legacy=0x7f6ed0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed015c850 con 0x7f6ed0072280 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6ec4001680 con 0x7f6ed0072280 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 >> v1:172.21.15.70:6789/0 conn(0x7f6ed0072280 legacy=0x7f6ed00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6eb8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb8004810 0x7f6eb8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6eb80051d0 0x7f6eb800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed00b3a50 con 0x7f6eb80039c0 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb800c070 con 0x7f6eb8004810 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6eb800c270 con 0x7f6eb80051d0 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6eb8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6eceffd640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb8004810 0x7f6eb8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6eb80051d0 msgr2=0x7f6eb800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6eb80051d0 0x7f6eb800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb8004810 msgr2=0x7f6eb8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb8004810 0x7f6eb8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/2324015306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb8003530 con 0x7f6eb80039c0 2024-04-22T01:18:20.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.124+0000 7f6ecffff640 1 -- 172.21.15.70:0/2324015306 reap_dead start 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6eb8003de0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001780 tx=0x7f6ec001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d81e44dbf61b188 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00013a0 con 0x7f6eb80039c0 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec00052f0 con 0x7f6eb80039c0 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ece7fc640 1 -- 172.21.15.70:0/2324015306 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0003840 con 0x7f6eb80039c0 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- 172.21.15.70:0/2324015306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 msgr2=0x7f6eb8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6eb8003de0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001780 tx=0x7f6ec001c400 comp rx=0 tx=0).stop 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- 172.21.15.70:0/2324015306 shutdown_connections 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/2324015306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6eb8003de0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- 172.21.15.70:0/2324015306 >> 172.21.15.70:0/2324015306 conn(0x7f6ed00b1d40 msgr2=0x7f6ed00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:20.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- 172.21.15.70:0/2324015306 shutdown_connections 2024-04-22T01:18:20.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- 172.21.15.70:0/2324015306 wait complete. 2024-04-22T01:18:20.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 Processor -- start 2024-04-22T01:18:20.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- start start 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6ed0141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ed0151f50 0x7f6ed0141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ed01591c0 0x7f6ed0156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed0072100 con 0x7f6eb80039c0 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed0071f80 con 0x7f6ed01591c0 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed00b4830 con 0x7f6ed0151f50 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6ed0141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6ed0141390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55188/0 (socket says 172.21.15.70:55188) 2024-04-22T01:18:20.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/1738393972 learned_addr learned my addr 172.21.15.70:0/1738393972 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6eceffd640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ed0151f50 0x7f6ed0141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecffff640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ed01591c0 0x7f6ed0156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ed0151f50 msgr2=0x7f6ed0141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ed0151f50 0x7f6ed0141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ed01591c0 msgr2=0x7f6ed0156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ed01591c0 0x7f6ed0156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 -- 172.21.15.70:0/1738393972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed0159ac0 con 0x7f6eb80039c0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecffff640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ed01591c0 0x7f6ed0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ecf7fe640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6ed0141390 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f6ec001c230 tx=0x7f6eb8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0007a40 con 0x7f6eb80039c0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec00ba070 con 0x7f6eb80039c0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed0159da0 con 0x7f6eb80039c0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.128+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00054a0 con 0x7f6eb80039c0 2024-04-22T01:18:20.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed015a360 con 0x7f6eb80039c0 2024-04-22T01:18:20.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed0068210 con 0x7f6eb80039c0 2024-04-22T01:18:20.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6ec0007030 con 0x7f6eb80039c0 2024-04-22T01:18:20.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6eb7fff640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ebc0678c0 0x7f6ebc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5249+0+0 (secure 0 0 0) 0x7f6ec011fb00 con 0x7f6eb80039c0 2024-04-22T01:18:20.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6eceffd640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ebc0678c0 0x7f6ebc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.132+0000 7f6eceffd640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ebc0678c0 0x7f6ebc069d80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4001000 tx=0x7f6ec4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.136+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6ec0016110 con 0x7f6eb80039c0 2024-04-22T01:18:20.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.296+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ed00d01e0 con 0x7f6eb80039c0 2024-04-22T01:18:20.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.296+0000 7f6eb7fff640 1 -- 172.21.15.70:0/1738393972 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f6ec00f0800 con 0x7f6eb80039c0 2024-04-22T01:18:20.300 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:20.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ebc0678c0 msgr2=0x7f6ebc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ebc0678c0 0x7f6ebc069d80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4001000 tx=0x7f6ec4009040 comp rx=0 tx=0).stop 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 msgr2=0x7f6ed0141390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6ed0141390 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f6ec001c230 tx=0x7f6eb8005840 comp rx=0 tx=0).stop 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 shutdown_connections 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6ebc0678c0 0x7f6ebc069d80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ed01591c0 0x7f6ed0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ed0151f50 0x7f6ed0141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 --2- 172.21.15.70:0/1738393972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eb80039c0 0x7f6ed0141390 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 >> 172.21.15.70:0/1738393972 conn(0x7f6ed00b1d40 msgr2=0x7f6ed015d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 shutdown_connections 2024-04-22T01:18:20.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.300+0000 7f6ed5d81640 1 -- 172.21.15.70:0/1738393972 wait complete. 2024-04-22T01:18:20.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:18:20.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.432+0000 7f4f91d2b640 1 Processor -- start 2024-04-22T01:18:20.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.432+0000 7f4f91d2b640 1 -- start start 2024-04-22T01:18:20.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f91d2b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4f8c1455b0 0x7f4f8c0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f91d2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0cc150 0x7f4f8c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f91d2b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f8c072060 0x7f4f8c15f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f91d2b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8c0724d0 con 0x7f4f8c1455b0 2024-04-22T01:18:20.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f91d2b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8c0d0250 con 0x7f4f8c072060 2024-04-22T01:18:20.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f91d2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c0cd7b0 con 0x7f4f8c0cc150 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4f8c1455b0 0x7f4f8c0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4f8c1455b0 0x7f4f8c0cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55200/0 (socket says 172.21.15.70:55200) 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/4259837460 learned_addr learned my addr 172.21.15.70:0/4259837460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8affd640 1 --2- 172.21.15.70:0/4259837460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0cc150 0x7f4f8c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8bfff640 1 --2- 172.21.15.70:0/4259837460 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c072060 0x7f4f8c15f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/4259837460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0cc150 msgr2=0x7f4f8c071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 --2- 172.21.15.70:0/4259837460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0cc150 0x7f4f8c071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/4259837460 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c072060 msgr2=0x7f4f8c15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 --2- 172.21.15.70:0/4259837460 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c072060 0x7f4f8c15f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/4259837460 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c0cc710 con 0x7f4f8c1455b0 2024-04-22T01:18:20.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8bfff640 1 --2- 172.21.15.70:0/4259837460 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c072060 0x7f4f8c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:20.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8b7fe640 1 --2- 172.21.15.70:0/4259837460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c1455b0 0x7f4f8c0cfd10 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0b1580 tx=0x7f4f7c01c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d8edf4267aeb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8a7fc640 1 -- 172.21.15.70:0/4259837460 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0b1070 con 0x7f4f8c1455b0 2024-04-22T01:18:20.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8a7fc640 1 -- 172.21.15.70:0/4259837460 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0ac440 con 0x7f4f8c1455b0 2024-04-22T01:18:20.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.436+0000 7f4f8a7fc640 1 -- 172.21.15.70:0/4259837460 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0ac810 con 0x7f4f8c1455b0 2024-04-22T01:18:20.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- 172.21.15.70:0/4259837460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c1455b0 msgr2=0x7f4f8c0cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/4259837460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c1455b0 0x7f4f8c0cfd10 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0b1580 tx=0x7f4f7c01c140 comp rx=0 tx=0).stop 2024-04-22T01:18:20.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- 172.21.15.70:0/4259837460 shutdown_connections 2024-04-22T01:18:20.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/4259837460 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c072060 0x7f4f8c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/4259837460 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0cc150 0x7f4f8c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/4259837460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c1455b0 0x7f4f8c0cfd10 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- 172.21.15.70:0/4259837460 >> 172.21.15.70:0/4259837460 conn(0x7f4f8c06b1f0 msgr2=0x7f4f8c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:20.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- 172.21.15.70:0/4259837460 shutdown_connections 2024-04-22T01:18:20.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- 172.21.15.70:0/4259837460 wait complete. 2024-04-22T01:18:20.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 Processor -- start 2024-04-22T01:18:20.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- start start 2024-04-22T01:18:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 0x7f4f8c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f8c0cc150 0x7f4f8c13d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c1455b0 0x7f4f8c144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c06e380 con 0x7f4f8c072060 2024-04-22T01:18:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c06e200 con 0x7f4f8c0cc150 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f91d2b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c06e500 con 0x7f4f8c1455b0 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 0x7f4f8c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 0x7f4f8c0b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55206/0 (socket says 172.21.15.70:55206) 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/2421700432 learned_addr learned my addr 172.21.15.70:0/2421700432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8bfff640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c1455b0 0x7f4f8c144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c1455b0 msgr2=0x7f4f8c144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c1455b0 0x7f4f8c144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8affd640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f8c0cc150 0x7f4f8c13d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f8c0cc150 msgr2=0x7f4f8c13d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f8c0cc150 0x7f4f8c13d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8b7fe640 1 -- 172.21.15.70:0/2421700432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c0b3700 con 0x7f4f8c072060 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.440+0000 7f4f8bfff640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c1455b0 0x7f4f8c144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f8b7fe640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 0x7f4f8c0b3b90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0af040 tx=0x7f4f7c018ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0bc070 con 0x7f4f8c072060 2024-04-22T01:18:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0b7660 con 0x7f4f8c072060 2024-04-22T01:18:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c0b2230 con 0x7f4f8c072060 2024-04-22T01:18:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0b1070 con 0x7f4f8c072060 2024-04-22T01:18:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c13d720 con 0x7f4f8c072060 2024-04-22T01:18:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c068210 con 0x7f4f8c072060 2024-04-22T01:18:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4f7c0bf070 con 0x7f4f8c072060 2024-04-22T01:18:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f73fff640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4f780678c0 0x7f4f78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5249+0+0 (secure 0 0 0) 0x7f4f7c11f460 con 0x7f4f8c072060 2024-04-22T01:18:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f8affd640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4f780678c0 0x7f4f78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.444+0000 7f4f8affd640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4f780678c0 0x7f4f78069d80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c165000 tx=0x7f4f80008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.448+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4f7c016110 con 0x7f4f8c072060 2024-04-22T01:18:20.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.608+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f4f8c15f510 con 0x7f4f8c072060 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.608+0000 7f4f73fff640 1 -- 172.21.15.70:0/2421700432 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) ==== 84+0+65 (secure 0 0 0) 0x7f4f7c0f0160 con 0x7f4f8c072060 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:20.611 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:18:20.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4f780678c0 msgr2=0x7f4f78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4f780678c0 0x7f4f78069d80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c165000 tx=0x7f4f80008040 comp rx=0 tx=0).stop 2024-04-22T01:18:20.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 msgr2=0x7f4f8c0b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 0x7f4f8c0b3b90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0af040 tx=0x7f4f7c018ee0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 shutdown_connections 2024-04-22T01:18:20.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4f780678c0 0x7f4f78069d80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c1455b0 0x7f4f8c144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f8c0cc150 0x7f4f8c13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 --2- 172.21.15.70:0/2421700432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8c072060 0x7f4f8c0b3b90 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 >> 172.21.15.70:0/2421700432 conn(0x7f4f8c06b1f0 msgr2=0x7f4f8c0af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:20.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 shutdown_connections 2024-04-22T01:18:20.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.612+0000 7f4f91d2b640 1 -- 172.21.15.70:0/2421700432 wait complete. 2024-04-22T01:18:20.628 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:18:20.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:18:20.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f27032c5640 1 Processor -- start 2024-04-22T01:18:20.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f27032c5640 1 -- start start 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f27032c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26fc0b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f27032c5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26fc0b75e0 con 0x7f26fc0cea10 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f27032c5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26fc0b78c0 con 0x7f26fc0b6db0 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f27032c5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0b7ba0 con 0x7f26fc15de20 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2701061640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26fc0b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2701862640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f26fc0cea10 0x7f26fc0ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43404/0 (socket says 172.21.15.70:43404) 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2701862640 1 -- 172.21.15.70:0/4059098121 learned_addr learned my addr 172.21.15.70:0/4059098121 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269244593 0 0) 0x7f26fc0b75e0 con 0x7f26fc0cea10 2024-04-22T01:18:20.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e8003530 con 0x7f26fc0cea10 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4218053935 0 0) 0x7f26e8003530 con 0x7f26fc0cea10 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 msgr2=0x7f26fc0b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26fc0b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 >> v1:172.21.15.98:6789/0 conn(0x7f26fc0b6db0 legacy=0x7f26fc0d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc0b7e80 con 0x7f26fc0cea10 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f26f4001a00 con 0x7f26fc0cea10 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2701061640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26fc0b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 >> v1:172.21.15.70:6789/0 conn(0x7f26fc0cea10 legacy=0x7f26fc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80039f0 0x7f26e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26e8004810 0x7f26e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26e80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc0b7ba0 con 0x7f26e8004810 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26e800bab0 con 0x7f26e80039f0 2024-04-22T01:18:20.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f26e800bcb0 con 0x7f26fc15de20 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2700860640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26e8004810 0x7f26e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2701061640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80039f0 0x7f26e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2701862640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26e80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2700860640 1 -- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 msgr2=0x7f26e80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2700860640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc15de20 0x7f26e80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2700860640 1 -- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80039f0 msgr2=0x7f26e8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2700860640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e80039f0 0x7f26e8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.748+0000 7f2700860640 1 -- 172.21.15.70:0/4059098121 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e8003530 con 0x7f26e8004810 2024-04-22T01:18:20.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f2700860640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26e8004810 0x7f26e8004c90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f26ec001750 tx=0x7f26ec01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed02fd770aa82a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0013a0 con 0x7f26e8004810 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec0052f0 con 0x7f26e8004810 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f26f3fff640 1 -- 172.21.15.70:0/4059098121 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec003840 con 0x7f26e8004810 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- 172.21.15.70:0/4059098121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26e8004810 msgr2=0x7f26e8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 --2- 172.21.15.70:0/4059098121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26e8004810 0x7f26e8004c90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f26ec001750 tx=0x7f26ec01c3c0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f2701862640 1 -- 172.21.15.70:0/4059098121 reap_dead start 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- 172.21.15.70:0/4059098121 shutdown_connections 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- 172.21.15.70:0/4059098121 >> 172.21.15.70:0/4059098121 conn(0x7f26fc072d20 msgr2=0x7f26fc073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- 172.21.15.70:0/4059098121 shutdown_connections 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- 172.21.15.70:0/4059098121 wait complete. 2024-04-22T01:18:20.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 Processor -- start 2024-04-22T01:18:20.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- start start 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e8004810 0x7f26fc0c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc151d20 0x7f26fc151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 0x7f26fc15e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc0b71f0 con 0x7f26fc15e410 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc15e200 con 0x7f26e8004810 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f27032c5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0b7370 con 0x7f26fc151d20 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f2701862640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 0x7f26fc15e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f2701862640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 0x7f26fc15e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55210/0 (socket says 172.21.15.70:55210) 2024-04-22T01:18:20.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.752+0000 7f2701862640 1 -- 172.21.15.70:0/441048335 learned_addr learned my addr 172.21.15.70:0/441048335 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2700860640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc151d20 0x7f26fc151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701061640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e8004810 0x7f26fc0c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701862640 1 -- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc151d20 msgr2=0x7f26fc151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701862640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc151d20 0x7f26fc151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701862640 1 -- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e8004810 msgr2=0x7f26fc0c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701862640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e8004810 0x7f26fc0c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701862640 1 -- 172.21.15.70:0/441048335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc1525a0 con 0x7f26fc15e410 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701061640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e8004810 0x7f26fc0c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:20.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701862640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 0x7f26fc15e890 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0b6d00 tx=0x7f26f400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4007aa0 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f400ee20 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc160d40 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4014670 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc0cb0f0 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26fc068210 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f26f4014810 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f26f1ffb640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f26e00678f0 0x7f26e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5249+0+0 (secure 0 0 0) 0x7f26f40858f0 con 0x7f26fc15e410 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701061640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f26e00678f0 0x7f26e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:20.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.756+0000 7f2701061640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f26e00678f0 0x7f26e0069db0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009a20 tx=0x7f26f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:20.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.760+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f26fc1525a0 con 0x7f26fc15e410 2024-04-22T01:18:20.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:20.920+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f26fc0b6610 con 0x7f26fc15e410 2024-04-22T01:18:21.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.300+0000 7f26f1ffb640 1 -- 172.21.15.70:0/441048335 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) ==== 138+0+0 (secure 0 0 0) 0x7f26f40565f0 con 0x7f26fc15e410 2024-04-22T01:18:21.303 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:18:21.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f26e00678f0 msgr2=0x7f26e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:21.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f26e00678f0 0x7f26e0069db0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009a20 tx=0x7f26f8008040 comp rx=0 tx=0).stop 2024-04-22T01:18:21.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 msgr2=0x7f26fc15e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:21.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 0x7f26fc15e890 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0b6d00 tx=0x7f26f400e530 comp rx=0 tx=0).stop 2024-04-22T01:18:21.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 shutdown_connections 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f26e00678f0 0x7f26e0069db0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26fc15e410 0x7f26fc15e890 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26fc151d20 0x7f26fc151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 --2- 172.21.15.70:0/441048335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26e8004810 0x7f26fc0c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 >> 172.21.15.70:0/441048335 conn(0x7f26fc072d20 msgr2=0x7f26fc0d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 shutdown_connections 2024-04-22T01:18:21.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.304+0000 7f27032c5640 1 -- 172.21.15.70:0/441048335 wait complete. 2024-04-22T01:18:21.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:18:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 Processor -- start 2024-04-22T01:18:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 -- start start 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7ab8072280 0x7f7ab80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab80cd050 0x7f7ab80b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ab815c290 con 0x7f7ab80b4450 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7ab815c570 con 0x7f7ab8072280 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abe659640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab815c850 con 0x7f7ab80cd050 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7ab8072280 0x7f7ab80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7ab8072280 0x7f7ab80ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55212/0 (socket says 172.21.15.70:55212) 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 -- 172.21.15.70:0/2099570098 learned_addr learned my addr 172.21.15.70:0/2099570098 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7abcbf6640 1 --2- 172.21.15.70:0/2099570098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab80cd050 0x7f7ab80b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 -- 172.21.15.70:0/2099570098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab80cd050 msgr2=0x7f7ab80b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 --2- 172.21.15.70:0/2099570098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab80cd050 0x7f7ab80b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 -- 172.21.15.70:0/2099570098 >> v1:172.21.15.98:6789/0 conn(0x7f7ab80b4450 legacy=0x7f7ab8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:21.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.464+0000 7f7ab77fe640 1 -- 172.21.15.70:0/2099570098 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab815cb30 con 0x7f7ab8072280 2024-04-22T01:18:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7ab77fe640 1 --2- 172.21.15.70:0/2099570098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab8072280 0x7f7ab80ccb10 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009870 tx=0x7f7aac01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfd1f0a680db7ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7ab6ffd640 1 -- 172.21.15.70:0/2099570098 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac0ab600 con 0x7f7ab8072280 2024-04-22T01:18:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7ab6ffd640 1 -- 172.21.15.70:0/2099570098 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aac0b0070 con 0x7f7ab8072280 2024-04-22T01:18:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7ab6ffd640 1 -- 172.21.15.70:0/2099570098 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac0ab9c0 con 0x7f7ab8072280 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 -- 172.21.15.70:0/2099570098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab8072280 msgr2=0x7f7ab80ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 --2- 172.21.15.70:0/2099570098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab8072280 0x7f7ab80ccb10 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009870 tx=0x7f7aac01c4d0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 -- 172.21.15.70:0/2099570098 shutdown_connections 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 --2- 172.21.15.70:0/2099570098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab80cd050 0x7f7ab80b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 --2- 172.21.15.70:0/2099570098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab8072280 0x7f7ab80ccb10 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 -- 172.21.15.70:0/2099570098 >> 172.21.15.70:0/2099570098 conn(0x7f7ab80b1d40 msgr2=0x7f7ab80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 -- 172.21.15.70:0/2099570098 shutdown_connections 2024-04-22T01:18:21.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 -- 172.21.15.70:0/2099570098 wait complete. 2024-04-22T01:18:21.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 Processor -- start 2024-04-22T01:18:21.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.468+0000 7f7abe659640 1 -- start start 2024-04-22T01:18:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab8072280 0x7f7ab81581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 0x7f7ab813a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab80cd050 0x7f7ab813c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab80cd980 con 0x7f7ab80b4450 2024-04-22T01:18:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab80cd800 con 0x7f7ab80cd050 2024-04-22T01:18:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab80b3790 con 0x7f7ab8072280 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 0x7f7ab813a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 0x7f7ab813a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55228/0 (socket says 172.21.15.70:55228) 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 -- 172.21.15.70:0/126033549 learned_addr learned my addr 172.21.15.70:0/126033549 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab7fff640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab8072280 0x7f7ab81581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 -- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab8072280 msgr2=0x7f7ab81581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab8072280 0x7f7ab81581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abcbf6640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab80cd050 0x7f7ab813c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 -- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab80cd050 msgr2=0x7f7ab813c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab80cd050 0x7f7ab813c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 -- 172.21.15.70:0/126033549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab813a990 con 0x7f7ab80b4450 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab77fe640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 0x7f7ab813a450 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0ad040 tx=0x7f7aac01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac0be5f0 con 0x7f7ab80b4450 2024-04-22T01:18:21.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aac0be790 con 0x7f7ab80b4450 2024-04-22T01:18:21.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab813a160 con 0x7f7ab80b4450 2024-04-22T01:18:21.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac0beb80 con 0x7f7ab80b4450 2024-04-22T01:18:21.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab813ccd0 con 0x7f7ab80b4450 2024-04-22T01:18:21.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab8068210 con 0x7f7ab80b4450 2024-04-22T01:18:21.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7aac0aa070 con 0x7f7ab80b4450 2024-04-22T01:18:21.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab4ff9640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7aa4067910 0x7f7aa4069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:21.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5619+0+0 (secure 0 0 0) 0x7f7aac120380 con 0x7f7ab80b4450 2024-04-22T01:18:21.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.472+0000 7f7ab7fff640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7aa4067910 0x7f7aa4069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:21.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.476+0000 7f7ab7fff640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7aa4067910 0x7f7aa4069dd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8000f70 tx=0x7f7aa8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:21.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.480+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7aac0c4060 con 0x7f7ab80b4450 2024-04-22T01:18:21.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:21.636+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7ab80cc880 con 0x7f7ab80b4450 2024-04-22T01:18:22.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.304+0000 7f7ab4ff9640 1 -- 172.21.15.70:0/126033549 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) ==== 170+0+0 (secure 0 0 0) 0x7f7aac016150 con 0x7f7ab80b4450 2024-04-22T01:18:22.308 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:18:22.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7aa4067910 msgr2=0x7f7aa4069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7aa4067910 0x7f7aa4069dd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8000f70 tx=0x7f7aa8009040 comp rx=0 tx=0).stop 2024-04-22T01:18:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 msgr2=0x7f7ab813a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 0x7f7ab813a450 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0ad040 tx=0x7f7aac01ad10 comp rx=0 tx=0).stop 2024-04-22T01:18:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 shutdown_connections 2024-04-22T01:18:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7aa4067910 0x7f7aa4069dd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab80cd050 0x7f7ab813c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7ab80b4450 0x7f7ab813a450 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 --2- 172.21.15.70:0/126033549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab8072280 0x7f7ab81581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 >> 172.21.15.70:0/126033549 conn(0x7f7ab80b1d40 msgr2=0x7f7ab80cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:22.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 shutdown_connections 2024-04-22T01:18:22.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.308+0000 7f7abe659640 1 -- 172.21.15.70:0/126033549 wait complete. 2024-04-22T01:18:22.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:18:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 Processor -- start 2024-04-22T01:18:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 -- start start 2024-04-22T01:18:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4015de20 0x7f0a400b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a400b6db0 0x7f0a400d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a400b75e0 con 0x7f0a400cea10 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0a400b78c0 con 0x7f0a400b6db0 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a47c00640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a400b7ba0 con 0x7f0a4015de20 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a400b6db0 0x7f0a400d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a400b6db0 0x7f0a400d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55242/0 (socket says 172.21.15.70:55242) 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 -- 172.21.15.70:0/838791328 learned_addr learned my addr 172.21.15.70:0/838791328 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4599c640 1 --2- 172.21.15.70:0/838791328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4015de20 0x7f0a400b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 -- 172.21.15.70:0/838791328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4015de20 msgr2=0x7f0a400b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 --2- 172.21.15.70:0/838791328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4015de20 0x7f0a400b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 -- 172.21.15.70:0/838791328 >> v1:172.21.15.98:6789/0 conn(0x7f0a400cea10 legacy=0x7f0a400ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:22.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.468+0000 7f0a4519b640 1 -- 172.21.15.70:0/838791328 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a400b7e80 con 0x7f0a400b6db0 2024-04-22T01:18:22.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4519b640 1 --2- 172.21.15.70:0/838791328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400b6db0 0x7f0a400d1d80 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f0a300098a0 tx=0x7f0a3001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bc30af0150373 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:22.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4499a640 1 -- 172.21.15.70:0/838791328 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300b5070 con 0x7f0a400b6db0 2024-04-22T01:18:22.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4499a640 1 -- 172.21.15.70:0/838791328 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a300a89b0 con 0x7f0a400b6db0 2024-04-22T01:18:22.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4499a640 1 -- 172.21.15.70:0/838791328 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300a8d80 con 0x7f0a400b6db0 2024-04-22T01:18:22.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- 172.21.15.70:0/838791328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400b6db0 msgr2=0x7f0a400d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:22.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 --2- 172.21.15.70:0/838791328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400b6db0 0x7f0a400d1d80 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f0a300098a0 tx=0x7f0a3001ecf0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- 172.21.15.70:0/838791328 shutdown_connections 2024-04-22T01:18:22.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 --2- 172.21.15.70:0/838791328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400b6db0 0x7f0a400d1d80 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 --2- 172.21.15.70:0/838791328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a4015de20 0x7f0a400b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- 172.21.15.70:0/838791328 >> 172.21.15.70:0/838791328 conn(0x7f0a40072d20 msgr2=0x7f0a40073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- 172.21.15.70:0/838791328 shutdown_connections 2024-04-22T01:18:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- 172.21.15.70:0/838791328 wait complete. 2024-04-22T01:18:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 Processor -- start 2024-04-22T01:18:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- start start 2024-04-22T01:18:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400b6db0 0x7f0a401f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 0x7f0a400c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4015de20 0x7f0a40160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a400d22c0 con 0x7f0a400cea10 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4015e200 con 0x7f0a4015de20 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a47c00640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0a400b71f0 con 0x7f0a400b6db0 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4519b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 0x7f0a400c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4519b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 0x7f0a400c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55250/0 (socket says 172.21.15.70:55250) 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4519b640 1 -- 172.21.15.70:0/2440892518 learned_addr learned my addr 172.21.15.70:0/2440892518 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4599c640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400b6db0 0x7f0a401f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.472+0000 7f0a4619d640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4015de20 0x7f0a40160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4519b640 1 -- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400b6db0 msgr2=0x7f0a401f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4519b640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400b6db0 0x7f0a401f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4519b640 1 -- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4015de20 msgr2=0x7f0a40160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4519b640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4015de20 0x7f0a40160d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4519b640 1 -- 172.21.15.70:0/2440892518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4015f760 con 0x7f0a400cea10 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4519b640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 0x7f0a400c9af0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f0a3001d3a0 tx=0x7f0a3001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300b5070 con 0x7f0a400cea10 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a300b0a90 con 0x7f0a400cea10 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4015fa40 con 0x7f0a400cea10 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300b0e60 con 0x7f0a400cea10 2024-04-22T01:18:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a40160000 con 0x7f0a400cea10 2024-04-22T01:18:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a40068210 con 0x7f0a400cea10 2024-04-22T01:18:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a300c2060 con 0x7f0a400cea10 2024-04-22T01:18:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a367fc640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a240678c0 0x7f0a24069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6001+0+0 (secure 0 0 0) 0x7f0a301232a0 con 0x7f0a400cea10 2024-04-22T01:18:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4599c640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a240678c0 0x7f0a24069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.476+0000 7f0a4599c640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a240678c0 0x7f0a24069d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0a40160770 tx=0x7f0a3c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:22.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.480+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0a30018850 con 0x7f0a400cea10 2024-04-22T01:18:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:22.676+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0a400d3450 con 0x7f0a400cea10 2024-04-22T01:18:23.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.332+0000 7f0a367fc640 1 -- 172.21.15.70:0/2440892518 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v42) ==== 174+0+0 (secure 0 0 0) 0x7f0a300f3d90 con 0x7f0a400cea10 2024-04-22T01:18:23.335 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 13 and data pool 14 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.336+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a240678c0 msgr2=0x7f0a24069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.336+0000 7f0a47c00640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a240678c0 0x7f0a24069d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0a40160770 tx=0x7f0a3c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.336+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 msgr2=0x7f0a400c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.336+0000 7f0a47c00640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 0x7f0a400c9af0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f0a3001d3a0 tx=0x7f0a3001ba90 comp rx=0 tx=0).stop 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 shutdown_connections 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f0a240678c0 0x7f0a24069d80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4015de20 0x7f0a40160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a400cea10 0x7f0a400c9af0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 --2- 172.21.15.70:0/2440892518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400b6db0 0x7f0a401f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 >> 172.21.15.70:0/2440892518 conn(0x7f0a40072d20 msgr2=0x7f0a400b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 shutdown_connections 2024-04-22T01:18:23.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.340+0000 7f0a47c00640 1 -- 172.21.15.70:0/2440892518 wait complete. 2024-04-22T01:18:23.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:23.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.488+0000 7ffbcc3dd640 1 Processor -- start 2024-04-22T01:18:23.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.488+0000 7ffbcc3dd640 1 -- start start 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbcc3dd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffbc40cd740 0x7ffbc406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbcc3dd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffbc406b730 0x7ffbc40d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbcc3dd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbc406bb50 con 0x7ffbc415bc70 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbcc3dd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ffbc40cfd10 con 0x7ffbc406b730 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbcc3dd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ffbc415c050 con 0x7ffbc40cd740 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca97a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffbc406b730 0x7ffbc40d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca179640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ffbc415bc70 0x7ffbc415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43418/0 (socket says 172.21.15.70:43418) 2024-04-22T01:18:23.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca179640 1 -- 172.21.15.70:0/428977703 learned_addr learned my addr 172.21.15.70:0/428977703 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:23.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbc9978640 1 --2- 172.21.15.70:0/428977703 >> v2:172.21.15.98:3300/0 conn(0x7ffbc40cd740 0x7ffbc406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca97a640 1 -- 172.21.15.70:0/428977703 >> v2:172.21.15.98:3300/0 conn(0x7ffbc40cd740 msgr2=0x7ffbc406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:23.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca97a640 1 --2- 172.21.15.70:0/428977703 >> v2:172.21.15.98:3300/0 conn(0x7ffbc40cd740 0x7ffbc406b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca97a640 1 -- 172.21.15.70:0/428977703 >> v1:172.21.15.70:6789/0 conn(0x7ffbc415bc70 legacy=0x7ffbc415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbc9177640 1 -- 172.21.15.70:0/428977703 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302053840 0 0) 0x7ffbc406bb50 con 0x7ffbc415bc70 2024-04-22T01:18:23.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca97a640 1 -- 172.21.15.70:0/428977703 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc415ff90 con 0x7ffbc406b730 2024-04-22T01:18:23.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbca97a640 1 --2- 172.21.15.70:0/428977703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc406b730 0x7ffbc40d0480 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4073920 tx=0x7ffbc001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7e531c010c0b08 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:23.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbc9177640 1 -- 172.21.15.70:0/428977703 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00b2070 con 0x7ffbc406b730 2024-04-22T01:18:23.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbc9177640 1 -- 172.21.15.70:0/428977703 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc00a59c0 con 0x7ffbc406b730 2024-04-22T01:18:23.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.492+0000 7ffbc9177640 1 -- 172.21.15.70:0/428977703 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00a5d90 con 0x7ffbc406b730 2024-04-22T01:18:23.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/428977703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc406b730 msgr2=0x7ffbc40d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/428977703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc406b730 0x7ffbc40d0480 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4073920 tx=0x7ffbc001c900 comp rx=0 tx=0).stop 2024-04-22T01:18:23.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/428977703 shutdown_connections 2024-04-22T01:18:23.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/428977703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc406b730 0x7ffbc40d0480 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/428977703 >> v2:172.21.15.98:3300/0 conn(0x7ffbc40cd740 0x7ffbc406b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/428977703 >> 172.21.15.70:0/428977703 conn(0x7ffbc4099a60 msgr2=0x7ffbc4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:23.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/428977703 shutdown_connections 2024-04-22T01:18:23.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/428977703 wait complete. 2024-04-22T01:18:23.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 Processor -- start 2024-04-22T01:18:23.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- start start 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbc406b730 0x7ffbc413ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc40cd740 0x7ffbc40af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 0x7ffbc414ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc40d1350 con 0x7ffbc415bc70 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc40d11d0 con 0x7ffbc40cd740 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffbc40d14d0 con 0x7ffbc406b730 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 0x7ffbc414ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 0x7ffbc414ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55278/0 (socket says 172.21.15.70:55278) 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 -- 172.21.15.70:0/478855162 learned_addr learned my addr 172.21.15.70:0/478855162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca179640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbc406b730 0x7ffbc413ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 -- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbc406b730 msgr2=0x7ffbc413ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbc9978640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc40cd740 0x7ffbc40af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbc406b730 0x7ffbc413ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 -- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc40cd740 msgr2=0x7ffbc40af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc40cd740 0x7ffbc40af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 -- 172.21.15.70:0/478855162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc4158620 con 0x7ffbc415bc70 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca179640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbc406b730 0x7ffbc413ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbca97a640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 0x7ffbc414ed20 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00b0040 tx=0x7ffbc00196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00bd020 con 0x7ffbc415bc70 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc00ad6c0 con 0x7ffbc415bc70 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc415b220 con 0x7ffbc415bc70 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00b2040 con 0x7ffbc415bc70 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.496+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc4158340 con 0x7ffbc415bc70 2024-04-22T01:18:23.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.500+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc4068920 con 0x7ffbc415bc70 2024-04-22T01:18:23.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.500+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffbc00c0020 con 0x7ffbc415bc70 2024-04-22T01:18:23.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.500+0000 7ffbbaffd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffba80678c0 0x7ffba8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.500+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7ffbc0121450 con 0x7ffbc415bc70 2024-04-22T01:18:23.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.500+0000 7ffbca179640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffba80678c0 0x7ffba8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.500+0000 7ffbca179640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffba80678c0 0x7ffba8069d80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc000f70 tx=0x7ffbbc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:23.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.504+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ffbc00c7030 con 0x7ffbc415bc70 2024-04-22T01:18:23.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.668+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffbc40cd520 con 0x7ffbc415bc70 2024-04-22T01:18:23.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbbaffd640 1 -- 172.21.15.70:0/478855162 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16862 (secure 0 0 0) 0x7ffbc4158620 con 0x7ffbc415bc70 2024-04-22T01:18:23.673 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:23.673 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":88,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:22.683877+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:23.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffba80678c0 msgr2=0x7ffba8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffba80678c0 0x7ffba8069d80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc000f70 tx=0x7ffbbc009040 comp rx=0 tx=0).stop 2024-04-22T01:18:23.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 msgr2=0x7ffbc414ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 0x7ffbc414ed20 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00b0040 tx=0x7ffbc00196a0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 shutdown_connections 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffba80678c0 0x7ffba8069d80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbc415bc70 0x7ffbc414ed20 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc40cd740 0x7ffbc40af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 --2- 172.21.15.70:0/478855162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffbc406b730 0x7ffbc413ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 >> 172.21.15.70:0/478855162 conn(0x7ffbc4099a60 msgr2=0x7ffbc40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 shutdown_connections 2024-04-22T01:18:23.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.672+0000 7ffbcc3dd640 1 -- 172.21.15.70:0/478855162 wait complete. 2024-04-22T01:18:23.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:23.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.804+0000 7f80f5d73640 1 Processor -- start 2024-04-22T01:18:23.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80f5d73640 1 -- start start 2024-04-22T01:18:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80f5d73640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80f0159380 0x7f80f00b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80f5d73640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f80f00cfe60 0x7f80f006e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80f5d73640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f0162a60 con 0x7f80f00b3a50 2024-04-22T01:18:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80f5d73640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f80f00b3f40 con 0x7f80f00cfe60 2024-04-22T01:18:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80f5d73640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f80f0162d00 con 0x7f80f0159380 2024-04-22T01:18:23.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f80f00cfe60 0x7f80f006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f80f00cfe60 0x7f80f006e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55284/0 (socket says 172.21.15.70:55284) 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 -- 172.21.15.70:0/645377776 learned_addr learned my addr 172.21.15.70:0/645377776 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80ef7fe640 1 --2- 172.21.15.70:0/645377776 >> v2:172.21.15.98:3300/0 conn(0x7f80f0159380 0x7f80f00b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 -- 172.21.15.70:0/645377776 >> v2:172.21.15.98:3300/0 conn(0x7f80f0159380 msgr2=0x7f80f00b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 --2- 172.21.15.70:0/645377776 >> v2:172.21.15.98:3300/0 conn(0x7f80f0159380 0x7f80f00b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 -- 172.21.15.70:0/645377776 >> v1:172.21.15.98:6789/0 conn(0x7f80f00b3a50 legacy=0x7f80f00cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 -- 172.21.15.70:0/645377776 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f0162fe0 con 0x7f80f00cfe60 2024-04-22T01:18:23.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80ef7fe640 1 --2- 172.21.15.70:0/645377776 >> v2:172.21.15.98:3300/0 conn(0x7f80f0159380 0x7f80f00b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:23.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.808+0000 7f80effff640 1 --2- 172.21.15.70:0/645377776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00cfe60 0x7f80f006e260 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f80f00d0850 tx=0x7f80e801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327e36deae906ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:23.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80ee7fc640 1 -- 172.21.15.70:0/645377776 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e80a5780 con 0x7f80f00cfe60 2024-04-22T01:18:23.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80ee7fc640 1 -- 172.21.15.70:0/645377776 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e80a5920 con 0x7f80f00cfe60 2024-04-22T01:18:23.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80ee7fc640 1 -- 172.21.15.70:0/645377776 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e80a5cf0 con 0x7f80f00cfe60 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- 172.21.15.70:0/645377776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00cfe60 msgr2=0x7f80f006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 --2- 172.21.15.70:0/645377776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00cfe60 0x7f80f006e260 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f80f00d0850 tx=0x7f80e801c920 comp rx=0 tx=0).stop 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- 172.21.15.70:0/645377776 shutdown_connections 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 --2- 172.21.15.70:0/645377776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00cfe60 0x7f80f006e260 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 --2- 172.21.15.70:0/645377776 >> v2:172.21.15.98:3300/0 conn(0x7f80f0159380 0x7f80f00b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- 172.21.15.70:0/645377776 >> 172.21.15.70:0/645377776 conn(0x7f80f006b1f0 msgr2=0x7f80f006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- 172.21.15.70:0/645377776 shutdown_connections 2024-04-22T01:18:23.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- 172.21.15.70:0/645377776 wait complete. 2024-04-22T01:18:23.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 Processor -- start 2024-04-22T01:18:23.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- start start 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 0x7f80f014ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80f00cfe60 0x7f80f015c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80f0159380 0x7f80f01506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80f016f3d0 con 0x7f80f00b3a50 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80f016f250 con 0x7f80f0159380 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80f5d73640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f80f016f550 con 0x7f80f00cfe60 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80ef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 0x7f80f014ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80ef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 0x7f80f014ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55300/0 (socket says 172.21.15.70:55300) 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.812+0000 7f80ef7fe640 1 -- 172.21.15.70:0/1591704779 learned_addr learned my addr 172.21.15.70:0/1591704779 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80effff640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80f0159380 0x7f80f01506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80eeffd640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80f00cfe60 0x7f80f015c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80ef7fe640 1 -- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80f00cfe60 msgr2=0x7f80f015c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80ef7fe640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80f00cfe60 0x7f80f015c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80ef7fe640 1 -- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80f0159380 msgr2=0x7f80f01506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80ef7fe640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80f0159380 0x7f80f01506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80ef7fe640 1 -- 172.21.15.70:0/1591704779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f0155840 con 0x7f80f00b3a50 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80ef7fe640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 0x7f80f014ff50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f80d800c7f0 tx=0x7f80d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8007a30 con 0x7f80f00b3a50 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d8007bd0 con 0x7f80f00b3a50 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f0155b20 con 0x7f80f00b3a50 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d80113f0 con 0x7f80f00b3a50 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f0159f90 con 0x7f80f00b3a50 2024-04-22T01:18:23.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f015a190 con 0x7f80f00b3a50 2024-04-22T01:18:23.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f80d8011590 con 0x7f80f00b3a50 2024-04-22T01:18:23.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80e7fff640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f80d00678c0 0x7f80d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:23.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f80d8086020 con 0x7f80f00b3a50 2024-04-22T01:18:23.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80eeffd640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f80d00678c0 0x7f80d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:23.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.816+0000 7f80eeffd640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f80d00678c0 0x7f80d0069d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f80f0156300 tx=0x7f80e0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:23.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.820+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f80f015a190 con 0x7f80f00b3a50 2024-04-22T01:18:23.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.980+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80f0068210 con 0x7f80f00b3a50 2024-04-22T01:18:23.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80e7fff640 1 -- 172.21.15.70:0/1591704779 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16862 (secure 0 0 0) 0x7f80d8015020 con 0x7f80f00b3a50 2024-04-22T01:18:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:23.986 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":88,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:22.683877+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:23.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f80d00678c0 msgr2=0x7f80d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f80d00678c0 0x7f80d0069d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f80f0156300 tx=0x7f80e0008040 comp rx=0 tx=0).stop 2024-04-22T01:18:23.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 msgr2=0x7f80f014ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:23.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 0x7f80f014ff50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f80d800c7f0 tx=0x7f80d800ccb0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 shutdown_connections 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f80d00678c0 0x7f80d0069d80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80f0159380 0x7f80f01506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80f00cfe60 0x7f80f015c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 --2- 172.21.15.70:0/1591704779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80f00b3a50 0x7f80f014ff50 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 >> 172.21.15.70:0/1591704779 conn(0x7f80f006b1f0 msgr2=0x7f80f0158ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 shutdown_connections 2024-04-22T01:18:23.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:23.984+0000 7f80f5d73640 1 -- 172.21.15.70:0/1591704779 wait complete. 2024-04-22T01:18:24.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:18:24.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 Processor -- start 2024-04-22T01:18:24.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 -- start start 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb11806d8f0 0x7fb11815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb11815f5c0 0x7fb11815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11806de60 con 0x7fb118158650 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb11806e140 con 0x7fb11815f5c0 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb11d6ef640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb11806e420 con 0x7fb11806d8f0 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb11815f5c0 0x7fb11815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb11815f5c0 0x7fb11815f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55310/0 (socket says 172.21.15.70:55310) 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 -- 172.21.15.70:0/199828479 learned_addr learned my addr 172.21.15.70:0/199828479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1167fc640 1 --2- 172.21.15.70:0/199828479 >> v2:172.21.15.98:3300/0 conn(0x7fb11806d8f0 0x7fb11815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 -- 172.21.15.70:0/199828479 >> v2:172.21.15.98:3300/0 conn(0x7fb11806d8f0 msgr2=0x7fb11815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 --2- 172.21.15.70:0/199828479 >> v2:172.21.15.98:3300/0 conn(0x7fb11806d8f0 0x7fb11815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 -- 172.21.15.70:0/199828479 >> v1:172.21.15.98:6789/0 conn(0x7fb118158650 legacy=0x7fb11806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.124+0000 7fb1177fe640 1 -- 172.21.15.70:0/199828479 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11806e700 con 0x7fb11815f5c0 2024-04-22T01:18:24.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 --2- 172.21.15.70:0/199828479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11815f9d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb118072730 tx=0x7fb11001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5f219d4bb715a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:24.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb115ffb640 1 -- 172.21.15.70:0/199828479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100a5620 con 0x7fb11815f5c0 2024-04-22T01:18:24.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb115ffb640 1 -- 172.21.15.70:0/199828479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1100a57c0 con 0x7fb11815f5c0 2024-04-22T01:18:24.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb115ffb640 1 -- 172.21.15.70:0/199828479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100a5b90 con 0x7fb11815f5c0 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- 172.21.15.70:0/199828479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 msgr2=0x7fb11815f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/199828479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11815f9d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb118072730 tx=0x7fb11001c960 comp rx=0 tx=0).stop 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- 172.21.15.70:0/199828479 shutdown_connections 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/199828479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11815f9d0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/199828479 >> v2:172.21.15.98:3300/0 conn(0x7fb11806d8f0 0x7fb11815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- 172.21.15.70:0/199828479 >> 172.21.15.70:0/199828479 conn(0x7fb1180b1d40 msgr2=0x7fb1180cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- 172.21.15.70:0/199828479 shutdown_connections 2024-04-22T01:18:24.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- 172.21.15.70:0/199828479 wait complete. 2024-04-22T01:18:24.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 Processor -- start 2024-04-22T01:18:24.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- start start 2024-04-22T01:18:24.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb11806d8f0 0x7fb118147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb118158650 0x7fb118148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11814a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb11816add0 con 0x7fb11815f5c0 2024-04-22T01:18:24.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb11816ac50 con 0x7fb118158650 2024-04-22T01:18:24.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb11d6ef640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb11816af50 con 0x7fb11806d8f0 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11814a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11814a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55322/0 (socket says 172.21.15.70:55322) 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 -- 172.21.15.70:0/1460426155 learned_addr learned my addr 172.21.15.70:0/1460426155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb116ffd640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb11806d8f0 0x7fb118147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1167fc640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb118158650 0x7fb118148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 -- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb11806d8f0 msgr2=0x7fb118147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb11806d8f0 0x7fb118147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 -- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb118158650 msgr2=0x7fb118148980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb118158650 0x7fb118148980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1177fe640 1 -- 172.21.15.70:0/1460426155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb118147670 con 0x7fb11815f5c0 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.128+0000 7fb1167fc640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb118158650 0x7fb118148980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb1177fe640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11814a4e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb110019570 tx=0x7fb110019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:24.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100bc020 con 0x7fb11815f5c0 2024-04-22T01:18:24.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1100b8670 con 0x7fb11815f5c0 2024-04-22T01:18:24.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1181566f0 con 0x7fb11815f5c0 2024-04-22T01:18:24.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110009510 con 0x7fb11815f5c0 2024-04-22T01:18:24.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb118156cb0 con 0x7fb11815f5c0 2024-04-22T01:18:24.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb118068210 con 0x7fb11815f5c0 2024-04-22T01:18:24.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb1100bf020 con 0x7fb11815f5c0 2024-04-22T01:18:24.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb0ff7fe640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1040678c0 0x7fb104069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7fb110120cf0 con 0x7fb11815f5c0 2024-04-22T01:18:24.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb116ffd640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1040678c0 0x7fb104069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.132+0000 7fb116ffd640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1040678c0 0x7fb104069d80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb118148200 tx=0x7fb10c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:24.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.136+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb110125070 con 0x7fb11815f5c0 2024-04-22T01:18:24.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.332+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb118148760 con 0x7fb11815f5c0 2024-04-22T01:18:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.356+0000 7fb0ff7fe640 1 -- 172.21.15.70:0/1460426155 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) ==== 118+0+0 (secure 0 0 0) 0x7fb1100c6030 con 0x7fb11815f5c0 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1040678c0 msgr2=0x7fb104069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1040678c0 0x7fb104069d80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb118148200 tx=0x7fb10c009040 comp rx=0 tx=0).stop 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 msgr2=0x7fb11814a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11814a4e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb110019570 tx=0x7fb110019700 comp rx=0 tx=0).stop 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 shutdown_connections 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb1040678c0 0x7fb104069d80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb11815f5c0 0x7fb11814a4e0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb118158650 0x7fb118148980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 --2- 172.21.15.70:0/1460426155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb11806d8f0 0x7fb118147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 >> 172.21.15.70:0/1460426155 conn(0x7fb1180b1d40 msgr2=0x7fb1180b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:24.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 shutdown_connections 2024-04-22T01:18:24.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.360+0000 7fb11d6ef640 1 -- 172.21.15.70:0/1460426155 wait complete. 2024-04-22T01:18:24.380 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-22T01:18:24.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:18:24.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.496+0000 7f3568879640 1 Processor -- start 2024-04-22T01:18:24.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.496+0000 7f3568879640 1 -- start start 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3568879640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3564072280 0x7f35640ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3568879640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f35640cd050 0x7f35640b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3568879640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f356415c290 con 0x7f35640b4450 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3568879640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f356415c570 con 0x7f35640cd050 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3568879640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f356415c850 con 0x7f3564072280 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f35640cd050 0x7f35640b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f35640cd050 0x7f35640b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55334/0 (socket says 172.21.15.70:55334) 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 -- 172.21.15.70:0/3573123793 learned_addr learned my addr 172.21.15.70:0/3573123793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f356259c640 1 --1- 172.21.15.70:0/3573123793 >> v1:172.21.15.70:6789/0 conn(0x7f35640b4450 0x7f3564071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43426/0 (socket says 172.21.15.70:43426) 2024-04-22T01:18:24.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 -- 172.21.15.70:0/3573123793 >> v2:172.21.15.98:3300/0 conn(0x7f3564072280 msgr2=0x7f35640ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:24.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 --2- 172.21.15.70:0/3573123793 >> v2:172.21.15.98:3300/0 conn(0x7f3564072280 0x7f35640ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 -- 172.21.15.70:0/3573123793 >> v1:172.21.15.70:6789/0 conn(0x7f35640b4450 legacy=0x7f3564071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 -- 172.21.15.70:0/3573123793 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356415cb30 con 0x7f35640cd050 2024-04-22T01:18:24.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3562d9d640 1 --2- 172.21.15.70:0/3573123793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f35640b2a40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f35640cf9a0 tx=0x7f355c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d6b939dedca94a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:24.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3561d9b640 1 -- 172.21.15.70:0/3573123793 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c0a5750 con 0x7f35640cd050 2024-04-22T01:18:24.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3561d9b640 1 -- 172.21.15.70:0/3573123793 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355c0a58f0 con 0x7f35640cd050 2024-04-22T01:18:24.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.500+0000 7f3561d9b640 1 -- 172.21.15.70:0/3573123793 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c0a5cc0 con 0x7f35640cd050 2024-04-22T01:18:24.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/3573123793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 msgr2=0x7f35640b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 --2- 172.21.15.70:0/3573123793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f35640b2a40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f35640cf9a0 tx=0x7f355c01c960 comp rx=0 tx=0).stop 2024-04-22T01:18:24.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/3573123793 shutdown_connections 2024-04-22T01:18:24.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 --2- 172.21.15.70:0/3573123793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f35640b2a40 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 --2- 172.21.15.70:0/3573123793 >> v2:172.21.15.98:3300/0 conn(0x7f3564072280 0x7f35640ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/3573123793 >> 172.21.15.70:0/3573123793 conn(0x7f35640b1d40 msgr2=0x7f35640cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:24.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/3573123793 shutdown_connections 2024-04-22T01:18:24.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/3573123793 wait complete. 2024-04-22T01:18:24.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 Processor -- start 2024-04-22T01:18:24.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- start start 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3564072280 0x7f356414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35640b4450 0x7f356414c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f3564146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35640cd980 con 0x7f35640cd050 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35640cd800 con 0x7f3564072280 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f35640b3790 con 0x7f35640b4450 2024-04-22T01:18:24.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f3564146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f3564146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55350/0 (socket says 172.21.15.70:55350) 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 -- 172.21.15.70:0/1782589510 learned_addr learned my addr 172.21.15.70:0/1782589510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 -- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35640b4450 msgr2=0x7f356414c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35640b4450 0x7f356414c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 -- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3564072280 msgr2=0x7f356414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f356259c640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3564072280 0x7f356414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3564072280 0x7f356414e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 -- 172.21.15.70:0/1782589510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356414d1e0 con 0x7f35640cd050 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f356259c640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3564072280 0x7f356414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3562d9d640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f3564146e50 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f355c019570 tx=0x7f355c019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c0b8670 con 0x7f35640cd050 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355c0be070 con 0x7f35640cd050 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c0b8a10 con 0x7f35640cd050 2024-04-22T01:18:24.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356414d4c0 con 0x7f35640cd050 2024-04-22T01:18:24.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.504+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356414da80 con 0x7f35640cd050 2024-04-22T01:18:24.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.508+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3564068210 con 0x7f35640cd050 2024-04-22T01:18:24.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.508+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f355c0b8bb0 con 0x7f35640cd050 2024-04-22T01:18:24.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.508+0000 7f355b7fe640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35440678c0 0x7f3544069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:24.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.508+0000 7f356259c640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35440678c0 0x7f3544069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:24.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.508+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7f355c121040 con 0x7f35640cd050 2024-04-22T01:18:24.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.508+0000 7f356259c640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35440678c0 0x7f3544069d80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3554007fd0 tx=0x7f3554009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:24.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.512+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f355c0165e0 con 0x7f35640cd050 2024-04-22T01:18:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:24.716+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f35640cede0 con 0x7f35640cd050 2024-04-22T01:18:25.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.372+0000 7f355b7fe640 1 -- 172.21.15.70:0/1782589510 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f355c0ac3a0 con 0x7f35640cd050 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35440678c0 msgr2=0x7f3544069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35440678c0 0x7f3544069d80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3554007fd0 tx=0x7f3554009040 comp rx=0 tx=0).stop 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 msgr2=0x7f3564146e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f3564146e50 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f355c019570 tx=0x7f355c019700 comp rx=0 tx=0).stop 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 shutdown_connections 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f35440678c0 0x7f3544069d80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35640cd050 0x7f3564146e50 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35640b4450 0x7f356414c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 --2- 172.21.15.70:0/1782589510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3564072280 0x7f356414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 >> 172.21.15.70:0/1782589510 conn(0x7f35640b1d40 msgr2=0x7f356415f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 shutdown_connections 2024-04-22T01:18:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.376+0000 7f3568879640 1 -- 172.21.15.70:0/1782589510 wait complete. 2024-04-22T01:18:25.396 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:18:25.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 Processor -- start 2024-04-22T01:18:25.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 -- start start 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef2c15aa30 0x7fef2c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fef2c0aee40 0x7fef2c0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef2c06dbd0 con 0x7fef2c15bb50 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fef2c06de70 con 0x7fef2c0aee40 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef32277640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fef2c06e150 con 0x7fef2c15aa30 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fef2c0aee40 0x7fef2c0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fef2c0aee40 0x7fef2c0b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55362/0 (socket says 172.21.15.70:55362) 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 -- 172.21.15.70:0/2539533312 learned_addr learned my addr 172.21.15.70:0/2539533312 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef2bfff640 1 --2- 172.21.15.70:0/2539533312 >> v2:172.21.15.98:3300/0 conn(0x7fef2c15aa30 0x7fef2c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 -- 172.21.15.70:0/2539533312 >> v2:172.21.15.98:3300/0 conn(0x7fef2c15aa30 msgr2=0x7fef2c0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 --2- 172.21.15.70:0/2539533312 >> v2:172.21.15.98:3300/0 conn(0x7fef2c15aa30 0x7fef2c0b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 -- 172.21.15.70:0/2539533312 >> v1:172.21.15.98:6789/0 conn(0x7fef2c15bb50 legacy=0x7fef2c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:25.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.516+0000 7fef30814640 1 -- 172.21.15.70:0/2539533312 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2c06e430 con 0x7fef2c0aee40 2024-04-22T01:18:25.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef30814640 1 --2- 172.21.15.70:0/2539533312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c0aee40 0x7fef2c0b28f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0b4680 tx=0x7fef2401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943fbe2df8c0e6e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:25.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2affd640 1 -- 172.21.15.70:0/2539533312 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef240a5840 con 0x7fef2c0aee40 2024-04-22T01:18:25.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2affd640 1 -- 172.21.15.70:0/2539533312 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef240a59e0 con 0x7fef2c0aee40 2024-04-22T01:18:25.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2affd640 1 -- 172.21.15.70:0/2539533312 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef240a5db0 con 0x7fef2c0aee40 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- 172.21.15.70:0/2539533312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c0aee40 msgr2=0x7fef2c0b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 --2- 172.21.15.70:0/2539533312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c0aee40 0x7fef2c0b28f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0b4680 tx=0x7fef2401c960 comp rx=0 tx=0).stop 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- 172.21.15.70:0/2539533312 shutdown_connections 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 --2- 172.21.15.70:0/2539533312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c0aee40 0x7fef2c0b28f0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 --2- 172.21.15.70:0/2539533312 >> v2:172.21.15.98:3300/0 conn(0x7fef2c15aa30 0x7fef2c0b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- 172.21.15.70:0/2539533312 >> 172.21.15.70:0/2539533312 conn(0x7fef2c06b1f0 msgr2=0x7fef2c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- 172.21.15.70:0/2539533312 shutdown_connections 2024-04-22T01:18:25.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- 172.21.15.70:0/2539533312 wait complete. 2024-04-22T01:18:25.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 Processor -- start 2024-04-22T01:18:25.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- start start 2024-04-22T01:18:25.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef2c0aee40 0x7fef2c165200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 0x7fef2c155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef2c15bb50 0x7fef2c1fa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef2c06ead0 con 0x7fef2c15aa30 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef2c06e950 con 0x7fef2c0aee40 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef32277640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fef2c071b20 con 0x7fef2c15bb50 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 0x7fef2c155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 0x7fef2c155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55368/0 (socket says 172.21.15.70:55368) 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2b7fe640 1 -- 172.21.15.70:0/1895640990 learned_addr learned my addr 172.21.15.70:0/1895640990 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef2bfff640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef2c0aee40 0x7fef2c165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.520+0000 7fef30814640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef2c15bb50 0x7fef2c1fa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2b7fe640 1 -- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef2c15bb50 msgr2=0x7fef2c1fa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2b7fe640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef2c15bb50 0x7fef2c1fa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2b7fe640 1 -- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef2c0aee40 msgr2=0x7fef2c165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2b7fe640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef2c0aee40 0x7fef2c165200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2b7fe640 1 -- 172.21.15.70:0/1895640990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2c164160 con 0x7fef2c15aa30 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2b7fe640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 0x7fef2c155a20 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fef2000b440 tx=0x7fef2000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20015020 con 0x7fef2c15aa30 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef20011890 con 0x7fef2c15aa30 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2c164440 con 0x7fef2c15aa30 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20011c80 con 0x7fef2c15aa30 2024-04-22T01:18:25.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef2c164a00 con 0x7fef2c15aa30 2024-04-22T01:18:25.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef2c068210 con 0x7fef2c15aa30 2024-04-22T01:18:25.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef20018020 con 0x7fef2c15aa30 2024-04-22T01:18:25.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef28ff9640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef180678c0 0x7fef18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fef20085c90 con 0x7fef2c15aa30 2024-04-22T01:18:25.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2bfff640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef180678c0 0x7fef18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.524+0000 7fef2bfff640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef180678c0 0x7fef18069d80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00b9d0 tx=0x7fef1c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:25.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.528+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fef2001f030 con 0x7fef2c15aa30 2024-04-22T01:18:25.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.724+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef2c0cc5d0 con 0x7fef2c15aa30 2024-04-22T01:18:25.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.724+0000 7fef28ff9640 1 -- 172.21.15.70:0/1895640990 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7fef2000d040 con 0x7fef2c15aa30 2024-04-22T01:18:25.727 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:25.727 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:22.684405+0000","modified":"2024-04-22T01:18:24.723325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7215":{"gid":7215,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2337533615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2337533615},{"type":"v1","addr":"172.21.15.70:6837","nonce":2337533615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7215,"qdb_cluster":[7215]},"id":6}]} 2024-04-22T01:18:25.727 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-04-22T01:18:25.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef180678c0 msgr2=0x7fef18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef180678c0 0x7fef18069d80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00b9d0 tx=0x7fef1c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:25.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 msgr2=0x7fef2c155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 0x7fef2c155a20 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fef2000b440 tx=0x7fef2000b900 comp rx=0 tx=0).stop 2024-04-22T01:18:25.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 shutdown_connections 2024-04-22T01:18:25.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fef180678c0 0x7fef18069d80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef2c15bb50 0x7fef2c1fa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef2c15aa30 0x7fef2c155a20 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 --2- 172.21.15.70:0/1895640990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef2c0aee40 0x7fef2c165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 >> 172.21.15.70:0/1895640990 conn(0x7fef2c06b1f0 msgr2=0x7fef2c15db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:25.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 shutdown_connections 2024-04-22T01:18:25.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.728+0000 7fef32277640 1 -- 172.21.15.70:0/1895640990 wait complete. 2024-04-22T01:18:25.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:25.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.864+0000 7f8770b4d640 1 Processor -- start 2024-04-22T01:18:25.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.864+0000 7f8770b4d640 1 -- start start 2024-04-22T01:18:25.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8770b4d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f876c163500 0x7f876c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8770b4d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876c162a60 con 0x7f876c159380 2024-04-22T01:18:25.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8770b4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876c0b3f40 con 0x7f876c0cfc90 2024-04-22T01:18:25.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8770b4d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f876c0d0100 con 0x7f876c163500 2024-04-22T01:18:25.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f876a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f876c159380 0x7f876c0b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43438/0 (socket says 172.21.15.70:43438) 2024-04-22T01:18:25.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f876a59c640 1 -- 172.21.15.70:0/310649712 learned_addr learned my addr 172.21.15.70:0/310649712 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:25.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8769d9b640 1 --2- 172.21.15.70:0/310649712 >> v2:172.21.15.98:3300/0 conn(0x7f876c163500 0x7f876c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f876959a640 1 -- 172.21.15.70:0/310649712 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4163093023 0 0) 0x7f876c162a60 con 0x7f876c159380 2024-04-22T01:18:25.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f876959a640 1 -- 172.21.15.70:0/310649712 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8754003530 con 0x7f876c159380 2024-04-22T01:18:25.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8769d9b640 1 -- 172.21.15.70:0/310649712 >> v1:172.21.15.98:6789/0 conn(0x7f876c0cfc90 legacy=0x7f876c06e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8769d9b640 1 -- 172.21.15.70:0/310649712 >> v1:172.21.15.70:6789/0 conn(0x7f876c159380 legacy=0x7f876c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f876959a640 1 -- 172.21.15.70:0/310649712 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1546396716 0 0) 0x7f8754003530 con 0x7f876c159380 2024-04-22T01:18:25.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.868+0000 7f8769d9b640 1 -- 172.21.15.70:0/310649712 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c162d00 con 0x7f876c163500 2024-04-22T01:18:25.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 --2- 172.21.15.70:0/310649712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c163500 0x7f876c0cf750 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f87600097c0 tx=0x7f876001cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d286852bb1110f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:25.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f876959a640 1 -- 172.21.15.70:0/310649712 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87600b0070 con 0x7f876c163500 2024-04-22T01:18:25.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f876959a640 1 -- 172.21.15.70:0/310649712 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87600ab400 con 0x7f876c163500 2024-04-22T01:18:25.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f876959a640 1 -- 172.21.15.70:0/310649712 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c0016a0 con 0x7f876c163500 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- 172.21.15.70:0/310649712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c163500 msgr2=0x7f876c0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 --2- 172.21.15.70:0/310649712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c163500 0x7f876c0cf750 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f87600097c0 tx=0x7f876001cd60 comp rx=0 tx=0).stop 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- 172.21.15.70:0/310649712 shutdown_connections 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 --2- 172.21.15.70:0/310649712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c163500 0x7f876c0cf750 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- 172.21.15.70:0/310649712 >> 172.21.15.70:0/310649712 conn(0x7f876c06b1f0 msgr2=0x7f876c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- 172.21.15.70:0/310649712 shutdown_connections 2024-04-22T01:18:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- 172.21.15.70:0/310649712 wait complete. 2024-04-22T01:18:25.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 Processor -- start 2024-04-22T01:18:25.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- start start 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c0cfc90 0x7f876c14fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 0x7f876c157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f876c163500 0x7f876c154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f876c166c50 con 0x7f876c159380 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f876c0b4220 con 0x7f876c0cfc90 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8770b4d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f876c166dd0 con 0x7f876c163500 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 0x7f876c157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 0x7f876c157450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55374/0 (socket says 172.21.15.70:55374) 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 -- 172.21.15.70:0/872704447 learned_addr learned my addr 172.21.15.70:0/872704447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 -- 172.21.15.70:0/872704447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f876c163500 msgr2=0x7f876c154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f876c163500 0x7f876c154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 -- 172.21.15.70:0/872704447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c0cfc90 msgr2=0x7f876c14fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c0cfc90 0x7f876c14fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.872+0000 7f8769d9b640 1 -- 172.21.15.70:0/872704447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c150150 con 0x7f876c159380 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f8769d9b640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 0x7f876c157450 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f8760000c10 tx=0x7f87600190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87600b9020 con 0x7f876c159380 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8760003920 con 0x7f876c159380 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876c1563b0 con 0x7f876c159380 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87600b0040 con 0x7f876c159380 2024-04-22T01:18:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876c156970 con 0x7f876c159380 2024-04-22T01:18:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c068210 con 0x7f876c159380 2024-04-22T01:18:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f87600bc020 con 0x7f876c159380 2024-04-22T01:18:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f875affd640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f874c0678f0 0x7f874c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7f876011fc70 con 0x7f876c159380 2024-04-22T01:18:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f876a59c640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f874c0678f0 0x7f874c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:25.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.876+0000 7f876a59c640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f874c0678f0 0x7f874c069db0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f875c001210 tx=0x7f875c001ad0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:25.880+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f87600c3030 con 0x7f876c159380 2024-04-22T01:18:26.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.040+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f876c0af310 con 0x7f876c159380 2024-04-22T01:18:26.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f875affd640 1 -- 172.21.15.70:0/872704447 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16849 (secure 0 0 0) 0x7f8760015fb0 con 0x7f876c159380 2024-04-22T01:18:26.046 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:26.046 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:24.337654+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f874c0678f0 msgr2=0x7f874c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f874c0678f0 0x7f874c069db0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f875c001210 tx=0x7f875c001ad0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 msgr2=0x7f876c157450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 0x7f876c157450 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f8760000c10 tx=0x7f87600190a0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 shutdown_connections 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f874c0678f0 0x7f874c069db0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f876c163500 0x7f876c154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c159380 0x7f876c157450 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 --2- 172.21.15.70:0/872704447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f876c0cfc90 0x7f876c14fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 >> 172.21.15.70:0/872704447 conn(0x7f876c06b1f0 msgr2=0x7f876c0cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:26.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 shutdown_connections 2024-04-22T01:18:26.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.044+0000 7f8770b4d640 1 -- 172.21.15.70:0/872704447 wait complete. 2024-04-22T01:18:26.065 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:18:26.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:18:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.184+0000 7f824b381640 1 Processor -- start 2024-04-22T01:18:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.184+0000 7f824b381640 1 -- start start 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824b381640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f824415aa30 0x7f82440b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824b381640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f824406dbd0 con 0x7f82440aee40 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824b381640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f824406de70 con 0x7f824415bb50 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824b381640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f824406e150 con 0x7f824415aa30 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824991e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f82440aee40 0x7f82440b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44508/0 (socket says 172.21.15.70:44508) 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824991e640 1 -- 172.21.15.70:0/2000578836 learned_addr learned my addr 172.21.15.70:0/2000578836 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 --2- 172.21.15.70:0/2000578836 >> v2:172.21.15.98:3300/0 conn(0x7f824415aa30 0x7f82440b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269555607 0 0) 0x7f824406dbd0 con 0x7f82440aee40 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f822c003530 con 0x7f82440aee40 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3327027765 0 0) 0x7f822c003530 con 0x7f82440aee40 2024-04-22T01:18:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 >> v2:172.21.15.98:3300/0 conn(0x7f824415aa30 msgr2=0x7f82440b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 --2- 172.21.15.70:0/2000578836 >> v2:172.21.15.98:3300/0 conn(0x7f824415aa30 0x7f82440b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 >> v1:172.21.15.98:6789/0 conn(0x7f824415bb50 legacy=0x7f82440ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824406e430 con 0x7f82440aee40 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f8234001a20 con 0x7f82440aee40 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 >> v1:172.21.15.70:6789/0 conn(0x7f82440aee40 legacy=0x7f82440b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f822c0039c0 0x7f822c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f822c004810 0x7f822c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0051d0 0x7f822c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f824406e150 con 0x7f822c0039c0 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f822c00c070 con 0x7f822c0051d0 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f822c00c270 con 0x7f822c004810 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f822c0039c0 0x7f822c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824991e640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0051d0 0x7f822c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824891c640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f822c004810 0x7f822c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 -- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f822c004810 msgr2=0x7f822c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f822c004810 0x7f822c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 -- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0051d0 msgr2=0x7f822c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0051d0 0x7f822c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 -- 172.21.15.70:0/2000578836 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822c003530 con 0x7f822c0039c0 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824991e640 1 -- 172.21.15.70:0/2000578836 reap_dead start 2024-04-22T01:18:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824891c640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f822c004810 0x7f822c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:26.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f824911d640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f822c0039c0 0x7f822c003de0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f823c009ac0 tx=0x7f823c01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793e2aaf5bd1e9d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.188+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c009690 con 0x7f822c0039c0 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823c0a64e0 con 0x7f822c0039c0 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f8243fff640 1 -- 172.21.15.70:0/2000578836 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c0a68b0 con 0x7f822c0039c0 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/2000578836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f822c0039c0 msgr2=0x7f822c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f822c0039c0 0x7f822c003de0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f823c009ac0 tx=0x7f823c01ca20 comp rx=0 tx=0).stop 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/2000578836 shutdown_connections 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 --2- 172.21.15.70:0/2000578836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f822c0039c0 0x7f822c003de0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/2000578836 >> 172.21.15.70:0/2000578836 conn(0x7f824406b1f0 msgr2=0x7f824406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/2000578836 shutdown_connections 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/2000578836 wait complete. 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 Processor -- start 2024-04-22T01:18:26.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- start start 2024-04-22T01:18:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0039c0 0x7f824414f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 0x7f8244155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82440ceba0 0x7f8244155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f824415af90 con 0x7f82440ce720 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f824415ae10 con 0x7f822c0039c0 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f824406e950 con 0x7f82440ceba0 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 0x7f8244155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 0x7f8244155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41782/0 (socket says 172.21.15.70:41782) 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 -- 172.21.15.70:0/1424012057 learned_addr learned my addr 172.21.15.70:0/1424012057 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824911d640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0039c0 0x7f824414f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824991e640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82440ceba0 0x7f8244155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 -- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82440ceba0 msgr2=0x7f8244155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82440ceba0 0x7f8244155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 -- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0039c0 msgr2=0x7f824414f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0039c0 0x7f824414f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 -- 172.21.15.70:0/1424012057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82440cf170 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824911d640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0039c0 0x7f824414f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824891c640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 0x7f8244155990 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f823800d4a0 tx=0x7f823800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238005280 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8238005440 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82440cf450 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238007260 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.192+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82440b0740 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.196+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8244068210 con 0x7f82440ce720 2024-04-22T01:18:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.196+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8238019020 con 0x7f82440ce720 2024-04-22T01:18:26.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.196+0000 7f8241ffb640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82300678f0 0x7f8230069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.196+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7f8238085480 con 0x7f82440ce720 2024-04-22T01:18:26.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.196+0000 7f824911d640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82300678f0 0x7f8230069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.196+0000 7f824911d640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82300678f0 0x7f8230069db0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f823c0af000 tx=0x7f823c0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.200+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8238055e30 con 0x7f82440ce720 2024-04-22T01:18:26.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.344+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f824406f190 con 0x7f82300678f0 2024-04-22T01:18:26.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.360+0000 7f8241ffb640 1 -- 172.21.15.70:0/1424012057 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252011 (secure 0 0 0) 0x7f824406f190 con 0x7f82300678f0 2024-04-22T01:18:26.366 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:26.368 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82300678f0 msgr2=0x7f8230069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82300678f0 0x7f8230069db0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f823c0af000 tx=0x7f823c0046d0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 msgr2=0x7f8244155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 0x7f8244155990 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f823800d4a0 tx=0x7f823800d960 comp rx=0 tx=0).stop 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 shutdown_connections 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f82300678f0 0x7f8230069db0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82440ceba0 0x7f8244155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82440ce720 0x7f8244155990 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 --2- 172.21.15.70:0/1424012057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c0039c0 0x7f824414f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 >> 172.21.15.70:0/1424012057 conn(0x7f824406b1f0 msgr2=0x7f8244159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 shutdown_connections 2024-04-22T01:18:26.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.364+0000 7f824b381640 1 -- 172.21.15.70:0/1424012057 wait complete. 2024-04-22T01:18:26.384 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-22T01:18:24.651403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271508,"kb_used_data":9556,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754703212,"statfs":{"total":773094113280,"available":772816089088,"internally_reserved":0,"allocated":9785344,"data_stored":5103975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":268095026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000930"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.370584+0000","last_change":"2024-04-22T01:18:23.363808+0000","last_active":"2024-04-22T01:18:23.370584+0000","last_peered":"2024-04-22T01:18:23.370584+0000","last_clean":"2024-04-22T01:18:23.370584+0000","last_became_active":"2024-04-22T01:18:23.362533+0000","last_became_peered":"2024-04-22T01:18:23.362533+0000","last_unstale":"2024-04-22T01:18:23.370584+0000","last_undegraded":"2024-04-22T01:18:23.370584+0000","last_fullsized":"2024-04-22T01:18:23.370584+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:33.821768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.377146+0000","last_change":"2024-04-22T01:18:23.371783+0000","last_active":"2024-04-22T01:18:23.377146+0000","last_peered":"2024-04-22T01:18:23.377146+0000","last_clean":"2024-04-22T01:18:23.377146+0000","last_became_active":"2024-04-22T01:18:23.371028+0000","last_became_peered":"2024-04-22T01:18:23.371028+0000","last_unstale":"2024-04-22T01:18:23.377146+0000","last_undegraded":"2024-04-22T01:18:23.377146+0000","last_fullsized":"2024-04-22T01:18:23.377146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:31.049833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.382297+0000","last_change":"2024-04-22T01:18:23.381733+0000","last_active":"2024-04-22T01:18:23.382297+0000","last_peered":"2024-04-22T01:18:23.382297+0000","last_clean":"2024-04-22T01:18:23.382297+0000","last_became_active":"2024-04-22T01:18:23.381451+0000","last_became_peered":"2024-04-22T01:18:23.381451+0000","last_unstale":"2024-04-22T01:18:23.382297+0000","last_undegraded":"2024-04-22T01:18:23.382297+0000","last_fullsized":"2024-04-22T01:18:23.382297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:55:20.204987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.391207+0000","last_change":"2024-04-22T01:18:23.390064+0000","last_active":"2024-04-22T01:18:23.391207+0000","last_peered":"2024-04-22T01:18:23.391207+0000","last_clean":"2024-04-22T01:18:23.391207+0000","last_became_active":"2024-04-22T01:18:23.389788+0000","last_became_peered":"2024-04-22T01:18:23.389788+0000","last_unstale":"2024-04-22T01:18:23.391207+0000","last_undegraded":"2024-04-22T01:18:23.391207+0000","last_fullsized":"2024-04-22T01:18:23.391207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:46.301034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389015+0000","last_change":"2024-04-22T01:18:23.377587+0000","last_active":"2024-04-22T01:18:23.389015+0000","last_peered":"2024-04-22T01:18:23.389015+0000","last_clean":"2024-04-22T01:18:23.389015+0000","last_became_active":"2024-04-22T01:18:23.374699+0000","last_became_peered":"2024-04-22T01:18:23.374699+0000","last_unstale":"2024-04-22T01:18:23.389015+0000","last_undegraded":"2024-04-22T01:18:23.389015+0000","last_fullsized":"2024-04-22T01:18:23.389015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:01.618682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375788+0000","last_change":"2024-04-22T01:18:23.371825+0000","last_active":"2024-04-22T01:18:23.375788+0000","last_peered":"2024-04-22T01:18:23.375788+0000","last_clean":"2024-04-22T01:18:23.375788+0000","last_became_active":"2024-04-22T01:18:23.371519+0000","last_became_peered":"2024-04-22T01:18:23.371519+0000","last_unstale":"2024-04-22T01:18:23.375788+0000","last_undegraded":"2024-04-22T01:18:23.375788+0000","last_fullsized":"2024-04-22T01:18:23.375788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:39.886334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.394032+0000","last_change":"2024-04-22T01:18:23.366116+0000","last_active":"2024-04-22T01:18:23.394032+0000","last_peered":"2024-04-22T01:18:23.394032+0000","last_clean":"2024-04-22T01:18:23.394032+0000","last_became_active":"2024-04-22T01:18:23.363236+0000","last_became_peered":"2024-04-22T01:18:23.363236+0000","last_unstale":"2024-04-22T01:18:23.394032+0000","last_undegraded":"2024-04-22T01:18:23.394032+0000","last_fullsized":"2024-04-22T01:18:23.394032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:33.772940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.372742+0000","last_change":"2024-04-22T01:18:23.372039+0000","last_active":"2024-04-22T01:18:23.372742+0000","last_peered":"2024-04-22T01:18:23.372742+0000","last_clean":"2024-04-22T01:18:23.372742+0000","last_became_active":"2024-04-22T01:18:23.371085+0000","last_became_peered":"2024-04-22T01:18:23.371085+0000","last_unstale":"2024-04-22T01:18:23.372742+0000","last_undegraded":"2024-04-22T01:18:23.372742+0000","last_fullsized":"2024-04-22T01:18:23.372742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:06.709623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389185+0000","last_change":"2024-04-22T01:18:23.378538+0000","last_active":"2024-04-22T01:18:23.389185+0000","last_peered":"2024-04-22T01:18:23.389185+0000","last_clean":"2024-04-22T01:18:23.389185+0000","last_became_active":"2024-04-22T01:18:23.377988+0000","last_became_peered":"2024-04-22T01:18:23.377988+0000","last_unstale":"2024-04-22T01:18:23.389185+0000","last_undegraded":"2024-04-22T01:18:23.389185+0000","last_fullsized":"2024-04-22T01:18:23.389185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:50.986562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.373090+0000","last_change":"2024-04-22T01:18:23.369080+0000","last_active":"2024-04-22T01:18:23.373090+0000","last_peered":"2024-04-22T01:18:23.373090+0000","last_clean":"2024-04-22T01:18:23.373090+0000","last_became_active":"2024-04-22T01:18:23.368764+0000","last_became_peered":"2024-04-22T01:18:23.368764+0000","last_unstale":"2024-04-22T01:18:23.373090+0000","last_undegraded":"2024-04-22T01:18:23.373090+0000","last_fullsized":"2024-04-22T01:18:23.373090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:13.713535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380354+0000","last_change":"2024-04-22T01:18:23.358977+0000","last_active":"2024-04-22T01:18:23.380354+0000","last_peered":"2024-04-22T01:18:23.380354+0000","last_clean":"2024-04-22T01:18:23.380354+0000","last_became_active":"2024-04-22T01:18:23.358313+0000","last_became_peered":"2024-04-22T01:18:23.358313+0000","last_unstale":"2024-04-22T01:18:23.380354+0000","last_undegraded":"2024-04-22T01:18:23.380354+0000","last_fullsized":"2024-04-22T01:18:23.380354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:42.230469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.363607+0000","last_change":"2024-04-22T01:18:23.353710+0000","last_active":"2024-04-22T01:18:23.363607+0000","last_peered":"2024-04-22T01:18:23.363607+0000","last_clean":"2024-04-22T01:18:23.363607+0000","last_became_active":"2024-04-22T01:18:23.353435+0000","last_became_peered":"2024-04-22T01:18:23.353435+0000","last_unstale":"2024-04-22T01:18:23.363607+0000","last_undegraded":"2024-04-22T01:18:23.363607+0000","last_fullsized":"2024-04-22T01:18:23.363607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:00.754122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.371865+0000","last_change":"2024-04-22T01:18:23.370394+0000","last_active":"2024-04-22T01:18:23.371865+0000","last_peered":"2024-04-22T01:18:23.371865+0000","last_clean":"2024-04-22T01:18:23.371865+0000","last_became_active":"2024-04-22T01:18:23.370087+0000","last_became_peered":"2024-04-22T01:18:23.370087+0000","last_unstale":"2024-04-22T01:18:23.371865+0000","last_undegraded":"2024-04-22T01:18:23.371865+0000","last_fullsized":"2024-04-22T01:18:23.371865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:47.698529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388717+0000","last_change":"2024-04-22T01:18:23.373090+0000","last_active":"2024-04-22T01:18:23.388717+0000","last_peered":"2024-04-22T01:18:23.388717+0000","last_clean":"2024-04-22T01:18:23.388717+0000","last_became_active":"2024-04-22T01:18:23.371665+0000","last_became_peered":"2024-04-22T01:18:23.371665+0000","last_unstale":"2024-04-22T01:18:23.388717+0000","last_undegraded":"2024-04-22T01:18:23.388717+0000","last_fullsized":"2024-04-22T01:18:23.388717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:33.547644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.367809+0000","last_change":"2024-04-22T01:18:23.366160+0000","last_active":"2024-04-22T01:18:23.367809+0000","last_peered":"2024-04-22T01:18:23.367809+0000","last_clean":"2024-04-22T01:18:23.367809+0000","last_became_active":"2024-04-22T01:18:23.365884+0000","last_became_peered":"2024-04-22T01:18:23.365884+0000","last_unstale":"2024-04-22T01:18:23.367809+0000","last_undegraded":"2024-04-22T01:18:23.367809+0000","last_fullsized":"2024-04-22T01:18:23.367809+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:30:18.800811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.374016+0000","last_change":"2024-04-22T01:18:23.359705+0000","last_active":"2024-04-22T01:18:23.374016+0000","last_peered":"2024-04-22T01:18:23.374016+0000","last_clean":"2024-04-22T01:18:23.374016+0000","last_became_active":"2024-04-22T01:18:23.359199+0000","last_became_peered":"2024-04-22T01:18:23.359199+0000","last_unstale":"2024-04-22T01:18:23.374016+0000","last_undegraded":"2024-04-22T01:18:23.374016+0000","last_fullsized":"2024-04-22T01:18:23.374016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:33.772940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376050+0000","last_change":"2024-04-22T01:18:23.366091+0000","last_active":"2024-04-22T01:18:23.376050+0000","last_peered":"2024-04-22T01:18:23.376050+0000","last_clean":"2024-04-22T01:18:23.376050+0000","last_became_active":"2024-04-22T01:18:23.365839+0000","last_became_peered":"2024-04-22T01:18:23.365839+0000","last_unstale":"2024-04-22T01:18:23.376050+0000","last_undegraded":"2024-04-22T01:18:23.376050+0000","last_fullsized":"2024-04-22T01:18:23.376050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:09.734932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.377202+0000","last_change":"2024-04-22T01:18:23.376783+0000","last_active":"2024-04-22T01:18:23.377202+0000","last_peered":"2024-04-22T01:18:23.377202+0000","last_clean":"2024-04-22T01:18:23.377202+0000","last_became_active":"2024-04-22T01:18:23.376477+0000","last_became_peered":"2024-04-22T01:18:23.376477+0000","last_unstale":"2024-04-22T01:18:23.377202+0000","last_undegraded":"2024-04-22T01:18:23.377202+0000","last_fullsized":"2024-04-22T01:18:23.377202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:45.480558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388495+0000","last_change":"2024-04-22T01:18:23.370434+0000","last_active":"2024-04-22T01:18:23.388495+0000","last_peered":"2024-04-22T01:18:23.388495+0000","last_clean":"2024-04-22T01:18:23.388495+0000","last_became_active":"2024-04-22T01:18:23.369587+0000","last_became_peered":"2024-04-22T01:18:23.369587+0000","last_unstale":"2024-04-22T01:18:23.388495+0000","last_undegraded":"2024-04-22T01:18:23.388495+0000","last_fullsized":"2024-04-22T01:18:23.388495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:19:42.139908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.369464+0000","last_change":"2024-04-22T01:18:23.367153+0000","last_active":"2024-04-22T01:18:23.369464+0000","last_peered":"2024-04-22T01:18:23.369464+0000","last_clean":"2024-04-22T01:18:23.369464+0000","last_became_active":"2024-04-22T01:18:23.366641+0000","last_became_peered":"2024-04-22T01:18:23.366641+0000","last_unstale":"2024-04-22T01:18:23.369464+0000","last_undegraded":"2024-04-22T01:18:23.369464+0000","last_fullsized":"2024-04-22T01:18:23.369464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:55.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388503+0000","last_change":"2024-04-22T01:18:23.370462+0000","last_active":"2024-04-22T01:18:23.388503+0000","last_peered":"2024-04-22T01:18:23.388503+0000","last_clean":"2024-04-22T01:18:23.388503+0000","last_became_active":"2024-04-22T01:18:23.369664+0000","last_became_peered":"2024-04-22T01:18:23.369664+0000","last_unstale":"2024-04-22T01:18:23.388503+0000","last_undegraded":"2024-04-22T01:18:23.388503+0000","last_fullsized":"2024-04-22T01:18:23.388503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:10.172833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388666+0000","last_change":"2024-04-22T01:18:23.372400+0000","last_active":"2024-04-22T01:18:23.388666+0000","last_peered":"2024-04-22T01:18:23.388666+0000","last_clean":"2024-04-22T01:18:23.388666+0000","last_became_active":"2024-04-22T01:18:23.370787+0000","last_became_peered":"2024-04-22T01:18:23.370787+0000","last_unstale":"2024-04-22T01:18:23.388666+0000","last_undegraded":"2024-04-22T01:18:23.388666+0000","last_fullsized":"2024-04-22T01:18:23.388666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:21:48.509140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393132+0000","last_change":"2024-04-22T01:18:23.373690+0000","last_active":"2024-04-22T01:18:23.393132+0000","last_peered":"2024-04-22T01:18:23.393132+0000","last_clean":"2024-04-22T01:18:23.393132+0000","last_became_active":"2024-04-22T01:18:23.372709+0000","last_became_peered":"2024-04-22T01:18:23.372709+0000","last_unstale":"2024-04-22T01:18:23.393132+0000","last_undegraded":"2024-04-22T01:18:23.393132+0000","last_fullsized":"2024-04-22T01:18:23.393132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:55.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.382674+0000","last_change":"2024-04-22T01:18:23.381973+0000","last_active":"2024-04-22T01:18:23.382674+0000","last_peered":"2024-04-22T01:18:23.382674+0000","last_clean":"2024-04-22T01:18:23.382674+0000","last_became_active":"2024-04-22T01:18:23.381683+0000","last_became_peered":"2024-04-22T01:18:23.381683+0000","last_unstale":"2024-04-22T01:18:23.382674+0000","last_undegraded":"2024-04-22T01:18:23.382674+0000","last_fullsized":"2024-04-22T01:18:23.382674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:36.487175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388135+0000","last_change":"2024-04-22T01:18:23.376719+0000","last_active":"2024-04-22T01:18:23.388135+0000","last_peered":"2024-04-22T01:18:23.388135+0000","last_clean":"2024-04-22T01:18:23.388135+0000","last_became_active":"2024-04-22T01:18:23.376017+0000","last_became_peered":"2024-04-22T01:18:23.376017+0000","last_unstale":"2024-04-22T01:18:23.388135+0000","last_undegraded":"2024-04-22T01:18:23.388135+0000","last_fullsized":"2024-04-22T01:18:23.388135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:13.120670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380790+0000","last_change":"2024-04-22T01:18:23.364637+0000","last_active":"2024-04-22T01:18:23.380790+0000","last_peered":"2024-04-22T01:18:23.380790+0000","last_clean":"2024-04-22T01:18:23.380790+0000","last_became_active":"2024-04-22T01:18:23.363899+0000","last_became_peered":"2024-04-22T01:18:23.363899+0000","last_unstale":"2024-04-22T01:18:23.380790+0000","last_undegraded":"2024-04-22T01:18:23.380790+0000","last_fullsized":"2024-04-22T01:18:23.380790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:19.846325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376355+0000","last_change":"2024-04-22T01:18:23.373710+0000","last_active":"2024-04-22T01:18:23.376355+0000","last_peered":"2024-04-22T01:18:23.376355+0000","last_clean":"2024-04-22T01:18:23.376355+0000","last_became_active":"2024-04-22T01:18:23.373404+0000","last_became_peered":"2024-04-22T01:18:23.373404+0000","last_unstale":"2024-04-22T01:18:23.376355+0000","last_undegraded":"2024-04-22T01:18:23.376355+0000","last_fullsized":"2024-04-22T01:18:23.376355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:32.350846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383045+0000","last_change":"2024-04-22T01:18:23.382375+0000","last_active":"2024-04-22T01:18:23.383045+0000","last_peered":"2024-04-22T01:18:23.383045+0000","last_clean":"2024-04-22T01:18:23.383045+0000","last_became_active":"2024-04-22T01:18:23.382104+0000","last_became_peered":"2024-04-22T01:18:23.382104+0000","last_unstale":"2024-04-22T01:18:23.383045+0000","last_undegraded":"2024-04-22T01:18:23.383045+0000","last_fullsized":"2024-04-22T01:18:23.383045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:33.430057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396169+0000","last_change":"2024-04-22T01:18:23.384163+0000","last_active":"2024-04-22T01:18:23.396169+0000","last_peered":"2024-04-22T01:18:23.396169+0000","last_clean":"2024-04-22T01:18:23.396169+0000","last_became_active":"2024-04-22T01:18:23.382028+0000","last_became_peered":"2024-04-22T01:18:23.382028+0000","last_unstale":"2024-04-22T01:18:23.396169+0000","last_undegraded":"2024-04-22T01:18:23.396169+0000","last_fullsized":"2024-04-22T01:18:23.396169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:30.422069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389072+0000","last_change":"2024-04-22T01:18:23.378952+0000","last_active":"2024-04-22T01:18:23.389072+0000","last_peered":"2024-04-22T01:18:23.389072+0000","last_clean":"2024-04-22T01:18:23.389072+0000","last_became_active":"2024-04-22T01:18:23.378210+0000","last_became_peered":"2024-04-22T01:18:23.378210+0000","last_unstale":"2024-04-22T01:18:23.389072+0000","last_undegraded":"2024-04-22T01:18:23.389072+0000","last_fullsized":"2024-04-22T01:18:23.389072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:24.136319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376865+0000","last_change":"2024-04-22T01:18:23.370174+0000","last_active":"2024-04-22T01:18:23.376865+0000","last_peered":"2024-04-22T01:18:23.376865+0000","last_clean":"2024-04-22T01:18:23.376865+0000","last_became_active":"2024-04-22T01:18:23.369296+0000","last_became_peered":"2024-04-22T01:18:23.369296+0000","last_unstale":"2024-04-22T01:18:23.376865+0000","last_undegraded":"2024-04-22T01:18:23.376865+0000","last_fullsized":"2024-04-22T01:18:23.376865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:46.301034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.373995+0000","last_change":"2024-04-22T01:18:23.359469+0000","last_active":"2024-04-22T01:18:23.373995+0000","last_peered":"2024-04-22T01:18:23.373995+0000","last_clean":"2024-04-22T01:18:23.373995+0000","last_became_active":"2024-04-22T01:18:23.358812+0000","last_became_peered":"2024-04-22T01:18:23.358812+0000","last_unstale":"2024-04-22T01:18:23.373995+0000","last_undegraded":"2024-04-22T01:18:23.373995+0000","last_fullsized":"2024-04-22T01:18:23.373995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:59.963726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383370+0000","last_change":"2024-04-22T01:18:23.382664+0000","last_active":"2024-04-22T01:18:23.383370+0000","last_peered":"2024-04-22T01:18:23.383370+0000","last_clean":"2024-04-22T01:18:23.383370+0000","last_became_active":"2024-04-22T01:18:23.381635+0000","last_became_peered":"2024-04-22T01:18:23.381635+0000","last_unstale":"2024-04-22T01:18:23.383370+0000","last_undegraded":"2024-04-22T01:18:23.383370+0000","last_fullsized":"2024-04-22T01:18:23.383370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:10.075040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"88'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393284+0000","last_change":"2024-04-22T01:18:22.336359+0000","last_active":"2024-04-22T01:18:23.393284+0000","last_peered":"2024-04-22T01:18:23.393284+0000","last_clean":"2024-04-22T01:18:23.393284+0000","last_became_active":"2024-04-22T01:18:22.335616+0000","last_became_peered":"2024-04-22T01:18:22.335616+0000","last_unstale":"2024-04-22T01:18:23.393284+0000","last_undegraded":"2024-04-22T01:18:23.393284+0000","last_fullsized":"2024-04-22T01:18:23.393284+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:56.913141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375888+0000","last_change":"2024-04-22T01:18:23.372439+0000","last_active":"2024-04-22T01:18:23.375888+0000","last_peered":"2024-04-22T01:18:23.375888+0000","last_clean":"2024-04-22T01:18:23.375888+0000","last_became_active":"2024-04-22T01:18:23.371203+0000","last_became_peered":"2024-04-22T01:18:23.371203+0000","last_unstale":"2024-04-22T01:18:23.375888+0000","last_undegraded":"2024-04-22T01:18:23.375888+0000","last_fullsized":"2024-04-22T01:18:23.375888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:36.381347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.387226+0000","last_change":"2024-04-22T01:18:22.354120+0000","last_active":"2024-04-22T01:18:23.387226+0000","last_peered":"2024-04-22T01:18:23.387226+0000","last_clean":"2024-04-22T01:18:23.387226+0000","last_became_active":"2024-04-22T01:18:22.353365+0000","last_became_peered":"2024-04-22T01:18:22.353365+0000","last_unstale":"2024-04-22T01:18:23.387226+0000","last_undegraded":"2024-04-22T01:18:23.387226+0000","last_fullsized":"2024-04-22T01:18:23.387226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:45.987866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396430+0000","last_change":"2024-04-22T01:18:23.384300+0000","last_active":"2024-04-22T01:18:23.396430+0000","last_peered":"2024-04-22T01:18:23.396430+0000","last_clean":"2024-04-22T01:18:23.396430+0000","last_became_active":"2024-04-22T01:18:23.383461+0000","last_became_peered":"2024-04-22T01:18:23.383461+0000","last_unstale":"2024-04-22T01:18:23.396430+0000","last_undegraded":"2024-04-22T01:18:23.396430+0000","last_fullsized":"2024-04-22T01:18:23.396430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:40:11.205592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.394147+0000","last_change":"2024-04-22T01:18:22.338794+0000","last_active":"2024-04-22T01:18:23.394147+0000","last_peered":"2024-04-22T01:18:23.394147+0000","last_clean":"2024-04-22T01:18:23.394147+0000","last_became_active":"2024-04-22T01:18:22.338008+0000","last_became_peered":"2024-04-22T01:18:22.338008+0000","last_unstale":"2024-04-22T01:18:23.394147+0000","last_undegraded":"2024-04-22T01:18:23.394147+0000","last_fullsized":"2024-04-22T01:18:23.394147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:39.116917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.391434+0000","last_change":"2024-04-22T01:18:23.390674+0000","last_active":"2024-04-22T01:18:23.391434+0000","last_peered":"2024-04-22T01:18:23.391434+0000","last_clean":"2024-04-22T01:18:23.391434+0000","last_became_active":"2024-04-22T01:18:23.390360+0000","last_became_peered":"2024-04-22T01:18:23.390360+0000","last_unstale":"2024-04-22T01:18:23.391434+0000","last_undegraded":"2024-04-22T01:18:23.391434+0000","last_fullsized":"2024-04-22T01:18:23.391434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:56:12.400145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.387022+0000","last_change":"2024-04-22T01:18:22.345395+0000","last_active":"2024-04-22T01:18:23.387022+0000","last_peered":"2024-04-22T01:18:23.387022+0000","last_clean":"2024-04-22T01:18:23.387022+0000","last_became_active":"2024-04-22T01:18:22.343707+0000","last_became_peered":"2024-04-22T01:18:22.343707+0000","last_unstale":"2024-04-22T01:18:23.387022+0000","last_undegraded":"2024-04-22T01:18:23.387022+0000","last_fullsized":"2024-04-22T01:18:23.387022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:49.973842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.369507+0000","last_change":"2024-04-22T01:18:23.366992+0000","last_active":"2024-04-22T01:18:23.369507+0000","last_peered":"2024-04-22T01:18:23.369507+0000","last_clean":"2024-04-22T01:18:23.369507+0000","last_became_active":"2024-04-22T01:18:23.366585+0000","last_became_peered":"2024-04-22T01:18:23.366585+0000","last_unstale":"2024-04-22T01:18:23.369507+0000","last_undegraded":"2024-04-22T01:18:23.369507+0000","last_fullsized":"2024-04-22T01:18:23.369507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:06.348633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380306+0000","last_change":"2024-04-22T01:18:22.354314+0000","last_active":"2024-04-22T01:18:23.380306+0000","last_peered":"2024-04-22T01:18:23.380306+0000","last_clean":"2024-04-22T01:18:23.380306+0000","last_became_active":"2024-04-22T01:18:22.351553+0000","last_became_peered":"2024-04-22T01:18:22.351553+0000","last_unstale":"2024-04-22T01:18:23.380306+0000","last_undegraded":"2024-04-22T01:18:23.380306+0000","last_fullsized":"2024-04-22T01:18:23.380306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:24.065014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393192+0000","last_change":"2024-04-22T01:18:23.383305+0000","last_active":"2024-04-22T01:18:23.393192+0000","last_peered":"2024-04-22T01:18:23.393192+0000","last_clean":"2024-04-22T01:18:23.393192+0000","last_became_active":"2024-04-22T01:18:23.382786+0000","last_became_peered":"2024-04-22T01:18:23.382786+0000","last_unstale":"2024-04-22T01:18:23.393192+0000","last_undegraded":"2024-04-22T01:18:23.393192+0000","last_fullsized":"2024-04-22T01:18:23.393192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:47.698529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.374592+0000","last_change":"2024-04-22T01:18:22.337177+0000","last_active":"2024-04-22T01:18:23.374592+0000","last_peered":"2024-04-22T01:18:23.374592+0000","last_clean":"2024-04-22T01:18:23.374592+0000","last_became_active":"2024-04-22T01:18:22.336803+0000","last_became_peered":"2024-04-22T01:18:22.336803+0000","last_unstale":"2024-04-22T01:18:23.374592+0000","last_undegraded":"2024-04-22T01:18:23.374592+0000","last_fullsized":"2024-04-22T01:18:23.374592+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:56.913141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375815+0000","last_change":"2024-04-22T01:18:23.371985+0000","last_active":"2024-04-22T01:18:23.375815+0000","last_peered":"2024-04-22T01:18:23.375815+0000","last_clean":"2024-04-22T01:18:23.375815+0000","last_became_active":"2024-04-22T01:18:23.371594+0000","last_became_peered":"2024-04-22T01:18:23.371594+0000","last_unstale":"2024-04-22T01:18:23.375815+0000","last_undegraded":"2024-04-22T01:18:23.375815+0000","last_fullsized":"2024-04-22T01:18:23.375815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:33.515501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.351748+0000","last_change":"2024-04-22T01:18:22.331579+0000","last_active":"2024-04-22T01:18:23.351748+0000","last_peered":"2024-04-22T01:18:23.351748+0000","last_clean":"2024-04-22T01:18:23.351748+0000","last_became_active":"2024-04-22T01:18:22.331147+0000","last_became_peered":"2024-04-22T01:18:22.331147+0000","last_unstale":"2024-04-22T01:18:23.351748+0000","last_undegraded":"2024-04-22T01:18:23.351748+0000","last_fullsized":"2024-04-22T01:18:23.351748+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.107951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388660+0000","last_change":"2024-04-22T01:18:23.388210+0000","last_active":"2024-04-22T01:18:23.388660+0000","last_peered":"2024-04-22T01:18:23.388660+0000","last_clean":"2024-04-22T01:18:23.388660+0000","last_became_active":"2024-04-22T01:18:23.387959+0000","last_became_peered":"2024-04-22T01:18:23.387959+0000","last_unstale":"2024-04-22T01:18:23.388660+0000","last_undegraded":"2024-04-22T01:18:23.388660+0000","last_fullsized":"2024-04-22T01:18:23.388660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:41.371744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.351144+0000","last_change":"2024-04-22T01:18:22.331454+0000","last_active":"2024-04-22T01:18:23.351144+0000","last_peered":"2024-04-22T01:18:23.351144+0000","last_clean":"2024-04-22T01:18:23.351144+0000","last_became_active":"2024-04-22T01:18:22.330721+0000","last_became_peered":"2024-04-22T01:18:22.330721+0000","last_unstale":"2024-04-22T01:18:23.351144+0000","last_undegraded":"2024-04-22T01:18:23.351144+0000","last_fullsized":"2024-04-22T01:18:23.351144+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:35.474455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380417+0000","last_change":"2024-04-22T01:18:23.354365+0000","last_active":"2024-04-22T01:18:23.380417+0000","last_peered":"2024-04-22T01:18:23.380417+0000","last_clean":"2024-04-22T01:18:23.380417+0000","last_became_active":"2024-04-22T01:18:23.353697+0000","last_became_peered":"2024-04-22T01:18:23.353697+0000","last_unstale":"2024-04-22T01:18:23.380417+0000","last_undegraded":"2024-04-22T01:18:23.380417+0000","last_fullsized":"2024-04-22T01:18:23.380417+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:43.503423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.347030+0000","last_change":"2024-04-22T01:18:22.342152+0000","last_active":"2024-04-22T01:18:23.347030+0000","last_peered":"2024-04-22T01:18:23.347030+0000","last_clean":"2024-04-22T01:18:23.347030+0000","last_became_active":"2024-04-22T01:18:22.341409+0000","last_became_peered":"2024-04-22T01:18:22.341409+0000","last_unstale":"2024-04-22T01:18:23.347030+0000","last_undegraded":"2024-04-22T01:18:23.347030+0000","last_fullsized":"2024-04-22T01:18:23.347030+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:44.836785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396421+0000","last_change":"2024-04-22T01:18:23.391833+0000","last_active":"2024-04-22T01:18:23.396421+0000","last_peered":"2024-04-22T01:18:23.396421+0000","last_clean":"2024-04-22T01:18:23.396421+0000","last_became_active":"2024-04-22T01:18:23.390929+0000","last_became_peered":"2024-04-22T01:18:23.390929+0000","last_unstale":"2024-04-22T01:18:23.396421+0000","last_undegraded":"2024-04-22T01:18:23.396421+0000","last_fullsized":"2024-04-22T01:18:23.396421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:09.734932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.348868+0000","last_change":"2024-04-22T01:18:22.334500+0000","last_active":"2024-04-22T01:18:23.348868+0000","last_peered":"2024-04-22T01:18:23.348868+0000","last_clean":"2024-04-22T01:18:23.348868+0000","last_became_active":"2024-04-22T01:18:22.334090+0000","last_became_peered":"2024-04-22T01:18:22.334090+0000","last_unstale":"2024-04-22T01:18:23.348868+0000","last_undegraded":"2024-04-22T01:18:23.348868+0000","last_fullsized":"2024-04-22T01:18:23.348868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:30.037113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.364788+0000","last_change":"2024-04-22T01:18:23.360412+0000","last_active":"2024-04-22T01:18:23.364788+0000","last_peered":"2024-04-22T01:18:23.364788+0000","last_clean":"2024-04-22T01:18:23.364788+0000","last_became_active":"2024-04-22T01:18:23.360065+0000","last_became_peered":"2024-04-22T01:18:23.360065+0000","last_unstale":"2024-04-22T01:18:23.364788+0000","last_undegraded":"2024-04-22T01:18:23.364788+0000","last_fullsized":"2024-04-22T01:18:23.364788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:10.995821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"88'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.381313+0000","last_change":"2024-04-22T01:18:22.356445+0000","last_active":"2024-04-22T01:18:23.381313+0000","last_peered":"2024-04-22T01:18:23.381313+0000","last_clean":"2024-04-22T01:18:23.381313+0000","last_became_active":"2024-04-22T01:18:22.354380+0000","last_became_peered":"2024-04-22T01:18:22.354380+0000","last_unstale":"2024-04-22T01:18:23.381313+0000","last_undegraded":"2024-04-22T01:18:23.381313+0000","last_fullsized":"2024-04-22T01:18:23.381313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:55.777815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.379510+0000","last_change":"2024-04-22T01:18:23.379153+0000","last_active":"2024-04-22T01:18:23.379510+0000","last_peered":"2024-04-22T01:18:23.379510+0000","last_clean":"2024-04-22T01:18:23.379510+0000","last_became_active":"2024-04-22T01:18:23.378861+0000","last_became_peered":"2024-04-22T01:18:23.378861+0000","last_unstale":"2024-04-22T01:18:23.379510+0000","last_undegraded":"2024-04-22T01:18:23.379510+0000","last_fullsized":"2024-04-22T01:18:23.379510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:45.849505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"88'6","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.395720+0000","last_change":"2024-04-22T01:18:22.343944+0000","last_active":"2024-04-22T01:18:23.395720+0000","last_peered":"2024-04-22T01:18:23.395720+0000","last_clean":"2024-04-22T01:18:23.395720+0000","last_became_active":"2024-04-22T01:18:22.343522+0000","last_became_peered":"2024-04-22T01:18:22.343522+0000","last_unstale":"2024-04-22T01:18:23.395720+0000","last_undegraded":"2024-04-22T01:18:23.395720+0000","last_fullsized":"2024-04-22T01:18:23.395720+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:44.893186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.368029+0000","last_change":"2024-04-22T01:18:23.366766+0000","last_active":"2024-04-22T01:18:23.368029+0000","last_peered":"2024-04-22T01:18:23.368029+0000","last_clean":"2024-04-22T01:18:23.368029+0000","last_became_active":"2024-04-22T01:18:23.365916+0000","last_became_peered":"2024-04-22T01:18:23.365916+0000","last_unstale":"2024-04-22T01:18:23.368029+0000","last_undegraded":"2024-04-22T01:18:23.368029+0000","last_fullsized":"2024-04-22T01:18:23.368029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:35.797753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396477+0000","last_change":"2024-04-22T01:18:22.340410+0000","last_active":"2024-04-22T01:18:23.396477+0000","last_peered":"2024-04-22T01:18:23.396477+0000","last_clean":"2024-04-22T01:18:23.396477+0000","last_became_active":"2024-04-22T01:18:22.339810+0000","last_became_peered":"2024-04-22T01:18:22.339810+0000","last_unstale":"2024-04-22T01:18:23.396477+0000","last_undegraded":"2024-04-22T01:18:23.396477+0000","last_fullsized":"2024-04-22T01:18:23.396477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:58.951006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388958+0000","last_change":"2024-04-22T01:18:23.376607+0000","last_active":"2024-04-22T01:18:23.388958+0000","last_peered":"2024-04-22T01:18:23.388958+0000","last_clean":"2024-04-22T01:18:23.388958+0000","last_became_active":"2024-04-22T01:18:23.375893+0000","last_became_peered":"2024-04-22T01:18:23.375893+0000","last_unstale":"2024-04-22T01:18:23.388958+0000","last_undegraded":"2024-04-22T01:18:23.388958+0000","last_fullsized":"2024-04-22T01:18:23.388958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:20.217880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.354738+0000","last_change":"2024-04-22T01:18:22.348718+0000","last_active":"2024-04-22T01:18:23.354738+0000","last_peered":"2024-04-22T01:18:23.354738+0000","last_clean":"2024-04-22T01:18:23.354738+0000","last_became_active":"2024-04-22T01:18:22.348360+0000","last_became_peered":"2024-04-22T01:18:22.348360+0000","last_unstale":"2024-04-22T01:18:23.354738+0000","last_undegraded":"2024-04-22T01:18:23.354738+0000","last_fullsized":"2024-04-22T01:18:23.354738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:03.205988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.372570+0000","last_change":"2024-04-22T01:18:23.368527+0000","last_active":"2024-04-22T01:18:23.372570+0000","last_peered":"2024-04-22T01:18:23.372570+0000","last_clean":"2024-04-22T01:18:23.372570+0000","last_became_active":"2024-04-22T01:18:23.366030+0000","last_became_peered":"2024-04-22T01:18:23.366030+0000","last_unstale":"2024-04-22T01:18:23.372570+0000","last_undegraded":"2024-04-22T01:18:23.372570+0000","last_fullsized":"2024-04-22T01:18:23.372570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:27:50.968614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.360036+0000","last_change":"2024-04-22T01:18:22.350212+0000","last_active":"2024-04-22T01:18:23.360036+0000","last_peered":"2024-04-22T01:18:23.360036+0000","last_clean":"2024-04-22T01:18:23.360036+0000","last_became_active":"2024-04-22T01:18:22.349847+0000","last_became_peered":"2024-04-22T01:18:22.349847+0000","last_unstale":"2024-04-22T01:18:23.360036+0000","last_undegraded":"2024-04-22T01:18:23.360036+0000","last_fullsized":"2024-04-22T01:18:23.360036+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:49.973842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376320+0000","last_change":"2024-04-22T01:18:23.375253+0000","last_active":"2024-04-22T01:18:23.376320+0000","last_peered":"2024-04-22T01:18:23.376320+0000","last_clean":"2024-04-22T01:18:23.376320+0000","last_became_active":"2024-04-22T01:18:23.374992+0000","last_became_peered":"2024-04-22T01:18:23.374992+0000","last_unstale":"2024-04-22T01:18:23.376320+0000","last_undegraded":"2024-04-22T01:18:23.376320+0000","last_fullsized":"2024-04-22T01:18:23.376320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:35.415321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"88'8","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.374871+0000","last_change":"2024-04-22T01:18:22.340743+0000","last_active":"2024-04-22T01:18:23.374871+0000","last_peered":"2024-04-22T01:18:23.374871+0000","last_clean":"2024-04-22T01:18:23.374871+0000","last_became_active":"2024-04-22T01:18:22.337246+0000","last_became_peered":"2024-04-22T01:18:22.337246+0000","last_unstale":"2024-04-22T01:18:23.374871+0000","last_undegraded":"2024-04-22T01:18:23.374871+0000","last_fullsized":"2024-04-22T01:18:23.374871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:00.605962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383060+0000","last_change":"2024-04-22T01:18:23.382402+0000","last_active":"2024-04-22T01:18:23.383060+0000","last_peered":"2024-04-22T01:18:23.383060+0000","last_clean":"2024-04-22T01:18:23.383060+0000","last_became_active":"2024-04-22T01:18:23.381918+0000","last_became_peered":"2024-04-22T01:18:23.381918+0000","last_unstale":"2024-04-22T01:18:23.383060+0000","last_undegraded":"2024-04-22T01:18:23.383060+0000","last_fullsized":"2024-04-22T01:18:23.383060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:29.397829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396917+0000","last_change":"2024-04-22T01:18:22.343048+0000","last_active":"2024-04-22T01:18:23.396917+0000","last_peered":"2024-04-22T01:18:23.396917+0000","last_clean":"2024-04-22T01:18:23.396917+0000","last_became_active":"2024-04-22T01:18:22.342341+0000","last_became_peered":"2024-04-22T01:18:22.342341+0000","last_unstale":"2024-04-22T01:18:23.396917+0000","last_undegraded":"2024-04-22T01:18:23.396917+0000","last_fullsized":"2024-04-22T01:18:23.396917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:30:17.788092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393415+0000","last_change":"2024-04-22T01:18:23.385370+0000","last_active":"2024-04-22T01:18:23.393415+0000","last_peered":"2024-04-22T01:18:23.393415+0000","last_clean":"2024-04-22T01:18:23.393415+0000","last_became_active":"2024-04-22T01:18:23.384673+0000","last_became_peered":"2024-04-22T01:18:23.384673+0000","last_unstale":"2024-04-22T01:18:23.393415+0000","last_undegraded":"2024-04-22T01:18:23.393415+0000","last_fullsized":"2024-04-22T01:18:23.393415+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:06.709623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396325+0000","last_change":"2024-04-22T01:18:22.338973+0000","last_active":"2024-04-22T01:18:23.396325+0000","last_peered":"2024-04-22T01:18:23.396325+0000","last_clean":"2024-04-22T01:18:23.396325+0000","last_became_active":"2024-04-22T01:18:22.338040+0000","last_became_peered":"2024-04-22T01:18:22.338040+0000","last_unstale":"2024-04-22T01:18:23.396325+0000","last_undegraded":"2024-04-22T01:18:23.396325+0000","last_fullsized":"2024-04-22T01:18:23.396325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:44.893186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380736+0000","last_change":"2024-04-22T01:18:23.363194+0000","last_active":"2024-04-22T01:18:23.380736+0000","last_peered":"2024-04-22T01:18:23.380736+0000","last_clean":"2024-04-22T01:18:23.380736+0000","last_became_active":"2024-04-22T01:18:23.362475+0000","last_became_peered":"2024-04-22T01:18:23.362475+0000","last_unstale":"2024-04-22T01:18:23.380736+0000","last_undegraded":"2024-04-22T01:18:23.380736+0000","last_fullsized":"2024-04-22T01:18:23.380736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:31.724530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.381483+0000","last_change":"2024-04-22T01:18:22.353742+0000","last_active":"2024-04-22T01:18:23.381483+0000","last_peered":"2024-04-22T01:18:23.381483+0000","last_clean":"2024-04-22T01:18:23.381483+0000","last_became_active":"2024-04-22T01:18:22.353105+0000","last_became_peered":"2024-04-22T01:18:22.353105+0000","last_unstale":"2024-04-22T01:18:23.381483+0000","last_undegraded":"2024-04-22T01:18:23.381483+0000","last_fullsized":"2024-04-22T01:18:23.381483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:07.676297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375859+0000","last_change":"2024-04-22T01:18:23.372009+0000","last_active":"2024-04-22T01:18:23.375859+0000","last_peered":"2024-04-22T01:18:23.375859+0000","last_clean":"2024-04-22T01:18:23.375859+0000","last_became_active":"2024-04-22T01:18:23.371729+0000","last_became_peered":"2024-04-22T01:18:23.371729+0000","last_unstale":"2024-04-22T01:18:23.375859+0000","last_undegraded":"2024-04-22T01:18:23.375859+0000","last_fullsized":"2024-04-22T01:18:23.375859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:12:45.354994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.379561+0000","last_change":"2024-04-22T01:18:22.331759+0000","last_active":"2024-04-22T01:18:23.379561+0000","last_peered":"2024-04-22T01:18:23.379561+0000","last_clean":"2024-04-22T01:18:23.379561+0000","last_became_active":"2024-04-22T01:18:22.330770+0000","last_became_peered":"2024-04-22T01:18:22.330770+0000","last_unstale":"2024-04-22T01:18:23.379561+0000","last_undegraded":"2024-04-22T01:18:23.379561+0000","last_fullsized":"2024-04-22T01:18:23.379561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:46.685809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.386221+0000","last_change":"2024-04-22T01:18:23.377105+0000","last_active":"2024-04-22T01:18:23.386221+0000","last_peered":"2024-04-22T01:18:23.386221+0000","last_clean":"2024-04-22T01:18:23.386221+0000","last_became_active":"2024-04-22T01:18:23.375338+0000","last_became_peered":"2024-04-22T01:18:23.375338+0000","last_unstale":"2024-04-22T01:18:23.386221+0000","last_undegraded":"2024-04-22T01:18:23.386221+0000","last_fullsized":"2024-04-22T01:18:23.386221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:10.995821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.342316+0000","last_change":"2024-04-22T01:18:22.340985+0000","last_active":"2024-04-22T01:18:23.342316+0000","last_peered":"2024-04-22T01:18:23.342316+0000","last_clean":"2024-04-22T01:18:23.342316+0000","last_became_active":"2024-04-22T01:18:22.340645+0000","last_became_peered":"2024-04-22T01:18:22.340645+0000","last_unstale":"2024-04-22T01:18:23.342316+0000","last_undegraded":"2024-04-22T01:18:23.342316+0000","last_fullsized":"2024-04-22T01:18:23.342316+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:21:57.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.377100+0000","last_change":"2024-04-22T01:18:23.369879+0000","last_active":"2024-04-22T01:18:23.377100+0000","last_peered":"2024-04-22T01:18:23.377100+0000","last_clean":"2024-04-22T01:18:23.377100+0000","last_became_active":"2024-04-22T01:18:23.369119+0000","last_became_peered":"2024-04-22T01:18:23.369119+0000","last_unstale":"2024-04-22T01:18:23.377100+0000","last_undegraded":"2024-04-22T01:18:23.377100+0000","last_fullsized":"2024-04-22T01:18:23.377100+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:41.371744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.346798+0000","last_change":"2024-04-22T01:18:22.331656+0000","last_active":"2024-04-22T01:18:23.346798+0000","last_peered":"2024-04-22T01:18:23.346798+0000","last_clean":"2024-04-22T01:18:23.346798+0000","last_became_active":"2024-04-22T01:18:22.330698+0000","last_became_peered":"2024-04-22T01:18:22.330698+0000","last_unstale":"2024-04-22T01:18:23.346798+0000","last_undegraded":"2024-04-22T01:18:23.346798+0000","last_fullsized":"2024-04-22T01:18:23.346798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:05.696903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375550+0000","last_change":"2024-04-22T01:18:23.363172+0000","last_active":"2024-04-22T01:18:23.375550+0000","last_peered":"2024-04-22T01:18:23.375550+0000","last_clean":"2024-04-22T01:18:23.375550+0000","last_became_active":"2024-04-22T01:18:23.361665+0000","last_became_peered":"2024-04-22T01:18:23.361665+0000","last_unstale":"2024-04-22T01:18:23.375550+0000","last_undegraded":"2024-04-22T01:18:23.375550+0000","last_fullsized":"2024-04-22T01:18:23.375550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:40:11.205592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.341888+0000","last_change":"2024-04-22T01:18:22.337757+0000","last_active":"2024-04-22T01:18:23.341888+0000","last_peered":"2024-04-22T01:18:23.341888+0000","last_clean":"2024-04-22T01:18:23.341888+0000","last_became_active":"2024-04-22T01:18:22.337493+0000","last_became_peered":"2024-04-22T01:18:22.337493+0000","last_unstale":"2024-04-22T01:18:23.341888+0000","last_undegraded":"2024-04-22T01:18:23.341888+0000","last_fullsized":"2024-04-22T01:18:23.341888+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:35:31.947361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393372+0000","last_change":"2024-04-22T01:18:23.385542+0000","last_active":"2024-04-22T01:18:23.393372+0000","last_peered":"2024-04-22T01:18:23.393372+0000","last_clean":"2024-04-22T01:18:23.393372+0000","last_became_active":"2024-04-22T01:18:23.384860+0000","last_became_peered":"2024-04-22T01:18:23.384860+0000","last_unstale":"2024-04-22T01:18:23.393372+0000","last_undegraded":"2024-04-22T01:18:23.393372+0000","last_fullsized":"2024-04-22T01:18:23.393372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:45.849505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396370+0000","last_change":"2024-04-22T01:18:22.339051+0000","last_active":"2024-04-22T01:18:23.396370+0000","last_peered":"2024-04-22T01:18:23.396370+0000","last_clean":"2024-04-22T01:18:23.396370+0000","last_became_active":"2024-04-22T01:18:22.338260+0000","last_became_peered":"2024-04-22T01:18:22.338260+0000","last_unstale":"2024-04-22T01:18:23.396370+0000","last_undegraded":"2024-04-22T01:18:23.396370+0000","last_fullsized":"2024-04-22T01:18:23.396370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:03.205988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383024+0000","last_change":"2024-04-22T01:18:23.381847+0000","last_active":"2024-04-22T01:18:23.383024+0000","last_peered":"2024-04-22T01:18:23.383024+0000","last_clean":"2024-04-22T01:18:23.383024+0000","last_became_active":"2024-04-22T01:18:23.381553+0000","last_became_peered":"2024-04-22T01:18:23.381553+0000","last_unstale":"2024-04-22T01:18:23.383024+0000","last_undegraded":"2024-04-22T01:18:23.383024+0000","last_fullsized":"2024-04-22T01:18:23.383024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:08:47.007435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.348950+0000","last_change":"2024-04-22T01:18:22.352558+0000","last_active":"2024-04-22T01:18:23.348950+0000","last_peered":"2024-04-22T01:18:23.348950+0000","last_clean":"2024-04-22T01:18:23.348950+0000","last_became_active":"2024-04-22T01:18:22.351798+0000","last_became_peered":"2024-04-22T01:18:22.351798+0000","last_unstale":"2024-04-22T01:18:23.348950+0000","last_undegraded":"2024-04-22T01:18:23.348950+0000","last_fullsized":"2024-04-22T01:18:23.348950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:45.895154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375615+0000","last_change":"2024-04-22T01:18:23.364528+0000","last_active":"2024-04-22T01:18:23.375615+0000","last_peered":"2024-04-22T01:18:23.375615+0000","last_clean":"2024-04-22T01:18:23.375615+0000","last_became_active":"2024-04-22T01:18:23.364226+0000","last_became_peered":"2024-04-22T01:18:23.364226+0000","last_unstale":"2024-04-22T01:18:23.375615+0000","last_undegraded":"2024-04-22T01:18:23.375615+0000","last_fullsized":"2024-04-22T01:18:23.375615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:31.685001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.349983+0000","last_change":"2024-04-22T01:18:22.358249+0000","last_active":"2024-04-22T01:18:23.349983+0000","last_peered":"2024-04-22T01:18:23.349983+0000","last_clean":"2024-04-22T01:18:23.349983+0000","last_became_active":"2024-04-22T01:18:22.357823+0000","last_became_peered":"2024-04-22T01:18:22.357823+0000","last_unstale":"2024-04-22T01:18:23.349983+0000","last_undegraded":"2024-04-22T01:18:23.349983+0000","last_fullsized":"2024-04-22T01:18:23.349983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:35:20.057528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.392344+0000","last_change":"2024-04-22T01:18:23.387863+0000","last_active":"2024-04-22T01:18:23.392344+0000","last_peered":"2024-04-22T01:18:23.392344+0000","last_clean":"2024-04-22T01:18:23.392344+0000","last_became_active":"2024-04-22T01:18:23.385568+0000","last_became_peered":"2024-04-22T01:18:23.385568+0000","last_unstale":"2024-04-22T01:18:23.392344+0000","last_undegraded":"2024-04-22T01:18:23.392344+0000","last_fullsized":"2024-04-22T01:18:23.392344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:33.821768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.354442+0000","last_change":"2024-04-22T01:18:22.341262+0000","last_active":"2024-04-22T01:18:23.354442+0000","last_peered":"2024-04-22T01:18:23.354442+0000","last_clean":"2024-04-22T01:18:23.354442+0000","last_became_active":"2024-04-22T01:18:22.335746+0000","last_became_peered":"2024-04-22T01:18:22.335746+0000","last_unstale":"2024-04-22T01:18:23.354442+0000","last_undegraded":"2024-04-22T01:18:23.354442+0000","last_fullsized":"2024-04-22T01:18:23.354442+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:23.123599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.363640+0000","last_change":"2024-04-22T01:18:23.352744+0000","last_active":"2024-04-22T01:18:23.363640+0000","last_peered":"2024-04-22T01:18:23.363640+0000","last_clean":"2024-04-22T01:18:23.363640+0000","last_became_active":"2024-04-22T01:18:23.352506+0000","last_became_peered":"2024-04-22T01:18:23.352506+0000","last_unstale":"2024-04-22T01:18:23.363640+0000","last_undegraded":"2024-04-22T01:18:23.363640+0000","last_fullsized":"2024-04-22T01:18:23.363640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:22.215257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.352476+0000","last_change":"2024-04-22T01:18:22.358743+0000","last_active":"2024-04-22T01:18:23.352476+0000","last_peered":"2024-04-22T01:18:23.352476+0000","last_clean":"2024-04-22T01:18:23.352476+0000","last_became_active":"2024-04-22T01:18:22.358436+0000","last_became_peered":"2024-04-22T01:18:22.358436+0000","last_unstale":"2024-04-22T01:18:23.352476+0000","last_undegraded":"2024-04-22T01:18:23.352476+0000","last_fullsized":"2024-04-22T01:18:23.352476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:39.116917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.370916+0000","last_change":"2024-04-22T01:18:23.370040+0000","last_active":"2024-04-22T01:18:23.370916+0000","last_peered":"2024-04-22T01:18:23.370916+0000","last_clean":"2024-04-22T01:18:23.370916+0000","last_became_active":"2024-04-22T01:18:23.367507+0000","last_became_peered":"2024-04-22T01:18:23.367507+0000","last_unstale":"2024-04-22T01:18:23.370916+0000","last_undegraded":"2024-04-22T01:18:23.370916+0000","last_fullsized":"2024-04-22T01:18:23.370916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:17:37.722726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.347285+0000","last_change":"2024-04-22T01:18:22.327211+0000","last_active":"2024-04-22T01:18:23.347285+0000","last_peered":"2024-04-22T01:18:23.347285+0000","last_clean":"2024-04-22T01:18:23.347285+0000","last_became_active":"2024-04-22T01:18:22.326518+0000","last_became_peered":"2024-04-22T01:18:22.326518+0000","last_unstale":"2024-04-22T01:18:23.347285+0000","last_undegraded":"2024-04-22T01:18:23.347285+0000","last_fullsized":"2024-04-22T01:18:23.347285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:32.417337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389210+0000","last_change":"2024-04-22T01:18:23.373450+0000","last_active":"2024-04-22T01:18:23.389210+0000","last_peered":"2024-04-22T01:18:23.389210+0000","last_clean":"2024-04-22T01:18:23.389210+0000","last_became_active":"2024-04-22T01:18:23.372607+0000","last_became_peered":"2024-04-22T01:18:23.372607+0000","last_unstale":"2024-04-22T01:18:23.389210+0000","last_undegraded":"2024-04-22T01:18:23.389210+0000","last_fullsized":"2024-04-22T01:18:23.389210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:06.348633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.357254+0000","last_change":"2024-04-22T01:18:22.336118+0000","last_active":"2024-04-22T01:18:23.357254+0000","last_peered":"2024-04-22T01:18:23.357254+0000","last_clean":"2024-04-22T01:18:23.357254+0000","last_became_active":"2024-04-22T01:18:22.334729+0000","last_became_peered":"2024-04-22T01:18:22.334729+0000","last_unstale":"2024-04-22T01:18:23.357254+0000","last_undegraded":"2024-04-22T01:18:23.357254+0000","last_fullsized":"2024-04-22T01:18:23.357254+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:34.785033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.373932+0000","last_change":"2024-04-22T01:18:23.359810+0000","last_active":"2024-04-22T01:18:23.373932+0000","last_peered":"2024-04-22T01:18:23.373932+0000","last_clean":"2024-04-22T01:18:23.373932+0000","last_became_active":"2024-04-22T01:18:23.358742+0000","last_became_peered":"2024-04-22T01:18:23.358742+0000","last_unstale":"2024-04-22T01:18:23.373932+0000","last_undegraded":"2024-04-22T01:18:23.373932+0000","last_fullsized":"2024-04-22T01:18:23.373932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:30.422069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.352770+0000","last_change":"2024-04-22T01:18:22.341349+0000","last_active":"2024-04-22T01:18:23.352770+0000","last_peered":"2024-04-22T01:18:23.352770+0000","last_clean":"2024-04-22T01:18:23.352770+0000","last_became_active":"2024-04-22T01:18:22.340497+0000","last_became_peered":"2024-04-22T01:18:22.340497+0000","last_unstale":"2024-04-22T01:18:23.352770+0000","last_undegraded":"2024-04-22T01:18:23.352770+0000","last_fullsized":"2024-04-22T01:18:23.352770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:07:05.665482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396558+0000","last_change":"2024-04-22T01:18:23.390781+0000","last_active":"2024-04-22T01:18:23.396558+0000","last_peered":"2024-04-22T01:18:23.396558+0000","last_clean":"2024-04-22T01:18:23.396558+0000","last_became_active":"2024-04-22T01:18:23.384276+0000","last_became_peered":"2024-04-22T01:18:23.384276+0000","last_unstale":"2024-04-22T01:18:23.396558+0000","last_undegraded":"2024-04-22T01:18:23.396558+0000","last_fullsized":"2024-04-22T01:18:23.396558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:31.685001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.360195+0000","last_change":"2024-04-22T01:18:22.340621+0000","last_active":"2024-04-22T01:18:23.360195+0000","last_peered":"2024-04-22T01:18:23.360195+0000","last_clean":"2024-04-22T01:18:23.360195+0000","last_became_active":"2024-04-22T01:18:22.340068+0000","last_became_peered":"2024-04-22T01:18:22.340068+0000","last_unstale":"2024-04-22T01:18:23.360195+0000","last_undegraded":"2024-04-22T01:18:23.360195+0000","last_fullsized":"2024-04-22T01:18:23.360195+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:23.123599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":290,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.352908+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:18:23.352908+0000","last_peered":"2024-04-22T01:18:23.352908+0000","last_clean":"2024-04-22T01:18:23.352908+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:18:23.352908+0000","last_undegraded":"2024-04-22T01:18:23.352908+0000","last_fullsized":"2024-04-22T01:18:23.352908+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1667072,"data_stored":1079877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36396,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335444,"statfs":{"total":96636764160,"available":96599494656,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335440,"statfs":{"total":96636764160,"available":96599490560,"internally_reserved":0,"allocated":1085440,"data_stored":496676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335448,"statfs":{"total":96636764160,"available":96599498752,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36972,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334868,"statfs":{"total":96636764160,"available":96598904832,"internally_reserved":0,"allocated":1667072,"data_stored":1079877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:18:26.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:18:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.540+0000 7f8ee285f640 1 Processor -- start 2024-04-22T01:18:26.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.540+0000 7f8ee285f640 1 -- start start 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee285f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8edc163500 0x7f8edc0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee285f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8edc0cfc90 0x7f8edc06e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee285f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8edc162a60 con 0x7f8edc159380 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee285f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8edc0d0100 con 0x7f8edc163500 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee285f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8edc0b3f40 con 0x7f8edc0cfc90 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8edc163500 0x7f8edc0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8edc163500 0x7f8edc0cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41794/0 (socket says 172.21.15.70:41794) 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8edbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8edc159380 0x7f8edc0b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44518/0 (socket says 172.21.15.70:44518) 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 -- 172.21.15.70:0/4243748830 learned_addr learned my addr 172.21.15.70:0/4243748830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:26.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee0dfc640 1 --2- 172.21.15.70:0/4243748830 >> v2:172.21.15.98:3300/0 conn(0x7f8edc0cfc90 0x7f8edc06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 -- 172.21.15.70:0/4243748830 >> v2:172.21.15.98:3300/0 conn(0x7f8edc0cfc90 msgr2=0x7f8edc06e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 --2- 172.21.15.70:0/4243748830 >> v2:172.21.15.98:3300/0 conn(0x7f8edc0cfc90 0x7f8edc06e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 -- 172.21.15.70:0/4243748830 >> v1:172.21.15.70:6789/0 conn(0x7f8edc159380 legacy=0x7f8edc0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 -- 172.21.15.70:0/4243748830 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc162d00 con 0x7f8edc163500 2024-04-22T01:18:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8edb7fe640 1 -- 172.21.15.70:0/4243748830 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 500013633 0 0) 0x7f8edc162a60 con 0x7f8edc159380 2024-04-22T01:18:26.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ee0dfc640 1 --2- 172.21.15.70:0/4243748830 >> v2:172.21.15.98:3300/0 conn(0x7f8edc0cfc90 0x7f8edc06e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:26.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8ed37fe640 1 --2- 172.21.15.70:0/4243748830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc163500 0x7f8edc0cf750 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0b3290 tx=0x7f8ec401cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143745e420be364b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8edb7fe640 1 -- 172.21.15.70:0/4243748830 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40ab680 con 0x7f8edc163500 2024-04-22T01:18:26.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8edb7fe640 1 -- 172.21.15.70:0/4243748830 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec40b0070 con 0x7f8edc163500 2024-04-22T01:18:26.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.544+0000 7f8edb7fe640 1 -- 172.21.15.70:0/4243748830 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40aba40 con 0x7f8edc163500 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/4243748830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc163500 msgr2=0x7f8edc0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 --2- 172.21.15.70:0/4243748830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc163500 0x7f8edc0cf750 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0b3290 tx=0x7f8ec401cec0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/4243748830 shutdown_connections 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 --2- 172.21.15.70:0/4243748830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc163500 0x7f8edc0cf750 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 --2- 172.21.15.70:0/4243748830 >> v2:172.21.15.98:3300/0 conn(0x7f8edc0cfc90 0x7f8edc06e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/4243748830 >> 172.21.15.70:0/4243748830 conn(0x7f8edc06b1f0 msgr2=0x7f8edc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/4243748830 shutdown_connections 2024-04-22T01:18:26.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/4243748830 wait complete. 2024-04-22T01:18:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 Processor -- start 2024-04-22T01:18:26.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- start start 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8edc0cfc90 0x7f8edc1520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8edc159380 0x7f8edc158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 0x7f8edc15d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc166c50 con 0x7f8edc157f60 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0b4220 con 0x7f8edc0cfc90 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8edc166dd0 con 0x7f8edc159380 2024-04-22T01:18:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 0x7f8edc15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 0x7f8edc15d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41798/0 (socket says 172.21.15.70:41798) 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 -- 172.21.15.70:0/2017950504 learned_addr learned my addr 172.21.15.70:0/2017950504 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ed37fe640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8edc159380 0x7f8edc158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8edbfff640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8edc0cfc90 0x7f8edc1520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 -- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8edc159380 msgr2=0x7f8edc158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8edc159380 0x7f8edc158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 -- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8edc0cfc90 msgr2=0x7f8edc1520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8edc0cfc90 0x7f8edc1520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 -- 172.21.15.70:0/2017950504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc158700 con 0x7f8edc157f60 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee0dfc640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 0x7f8edc15d540 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400cef0 tx=0x7f8ed400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed401a070 con 0x7f8edc157f60 2024-04-22T01:18:26.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed400d860 con 0x7f8edc157f60 2024-04-22T01:18:26.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc158420 con 0x7f8edc157f60 2024-04-22T01:18:26.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400dc50 con 0x7f8edc157f60 2024-04-22T01:18:26.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc15a770 con 0x7f8edc157f60 2024-04-22T01:18:26.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.548+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc068210 con 0x7f8edc157f60 2024-04-22T01:18:26.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.552+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ed401d070 con 0x7f8edc157f60 2024-04-22T01:18:26.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.552+0000 7f8ed97fa640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ebc0678c0 0x7f8ebc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.552+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7f8ed40866e0 con 0x7f8edc157f60 2024-04-22T01:18:26.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.552+0000 7f8edbfff640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ebc0678c0 0x7f8ebc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.552+0000 7f8edbfff640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ebc0678c0 0x7f8ebc069d80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc000f70 tx=0x7f8ecc007b80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.556+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8ed4057150 con 0x7f8edc157f60 2024-04-22T01:18:26.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.712+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8edc1531e0 con 0x7f8ebc0678c0 2024-04-22T01:18:26.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.728+0000 7f8ed97fa640 1 -- 172.21.15.70:0/2017950504 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252011 (secure 0 0 0) 0x7f8edc1531e0 con 0x7f8ebc0678c0 2024-04-22T01:18:26.735 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:26.737 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ebc0678c0 msgr2=0x7f8ebc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ebc0678c0 0x7f8ebc069d80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc000f70 tx=0x7f8ecc007b80 comp rx=0 tx=0).stop 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 msgr2=0x7f8edc15d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 0x7f8edc15d540 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400cef0 tx=0x7f8ed400a710 comp rx=0 tx=0).stop 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 shutdown_connections 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8ebc0678c0 0x7f8ebc069d80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8edc157f60 0x7f8edc15d540 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8edc159380 0x7f8edc158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 --2- 172.21.15.70:0/2017950504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8edc0cfc90 0x7f8edc1520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 >> 172.21.15.70:0/2017950504 conn(0x7f8edc06b1f0 msgr2=0x7f8edc165580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:26.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 shutdown_connections 2024-04-22T01:18:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.736+0000 7f8ee285f640 1 -- 172.21.15.70:0/2017950504 wait complete. 2024-04-22T01:18:26.755 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-04-22T01:18:26.651934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271508,"kb_used_data":9556,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754703212,"statfs":{"total":773094113280,"available":772816089088,"internally_reserved":0,"allocated":9785344,"data_stored":5103975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":268095026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.344989"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.370584+0000","last_change":"2024-04-22T01:18:23.363808+0000","last_active":"2024-04-22T01:18:23.370584+0000","last_peered":"2024-04-22T01:18:23.370584+0000","last_clean":"2024-04-22T01:18:23.370584+0000","last_became_active":"2024-04-22T01:18:23.362533+0000","last_became_peered":"2024-04-22T01:18:23.362533+0000","last_unstale":"2024-04-22T01:18:23.370584+0000","last_undegraded":"2024-04-22T01:18:23.370584+0000","last_fullsized":"2024-04-22T01:18:23.370584+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:33.821768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.377146+0000","last_change":"2024-04-22T01:18:23.371783+0000","last_active":"2024-04-22T01:18:23.377146+0000","last_peered":"2024-04-22T01:18:23.377146+0000","last_clean":"2024-04-22T01:18:23.377146+0000","last_became_active":"2024-04-22T01:18:23.371028+0000","last_became_peered":"2024-04-22T01:18:23.371028+0000","last_unstale":"2024-04-22T01:18:23.377146+0000","last_undegraded":"2024-04-22T01:18:23.377146+0000","last_fullsized":"2024-04-22T01:18:23.377146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:31.049833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.382297+0000","last_change":"2024-04-22T01:18:23.381733+0000","last_active":"2024-04-22T01:18:23.382297+0000","last_peered":"2024-04-22T01:18:23.382297+0000","last_clean":"2024-04-22T01:18:23.382297+0000","last_became_active":"2024-04-22T01:18:23.381451+0000","last_became_peered":"2024-04-22T01:18:23.381451+0000","last_unstale":"2024-04-22T01:18:23.382297+0000","last_undegraded":"2024-04-22T01:18:23.382297+0000","last_fullsized":"2024-04-22T01:18:23.382297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:55:20.204987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.391207+0000","last_change":"2024-04-22T01:18:23.390064+0000","last_active":"2024-04-22T01:18:23.391207+0000","last_peered":"2024-04-22T01:18:23.391207+0000","last_clean":"2024-04-22T01:18:23.391207+0000","last_became_active":"2024-04-22T01:18:23.389788+0000","last_became_peered":"2024-04-22T01:18:23.389788+0000","last_unstale":"2024-04-22T01:18:23.391207+0000","last_undegraded":"2024-04-22T01:18:23.391207+0000","last_fullsized":"2024-04-22T01:18:23.391207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:46.301034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389015+0000","last_change":"2024-04-22T01:18:23.377587+0000","last_active":"2024-04-22T01:18:23.389015+0000","last_peered":"2024-04-22T01:18:23.389015+0000","last_clean":"2024-04-22T01:18:23.389015+0000","last_became_active":"2024-04-22T01:18:23.374699+0000","last_became_peered":"2024-04-22T01:18:23.374699+0000","last_unstale":"2024-04-22T01:18:23.389015+0000","last_undegraded":"2024-04-22T01:18:23.389015+0000","last_fullsized":"2024-04-22T01:18:23.389015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:01.618682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375788+0000","last_change":"2024-04-22T01:18:23.371825+0000","last_active":"2024-04-22T01:18:23.375788+0000","last_peered":"2024-04-22T01:18:23.375788+0000","last_clean":"2024-04-22T01:18:23.375788+0000","last_became_active":"2024-04-22T01:18:23.371519+0000","last_became_peered":"2024-04-22T01:18:23.371519+0000","last_unstale":"2024-04-22T01:18:23.375788+0000","last_undegraded":"2024-04-22T01:18:23.375788+0000","last_fullsized":"2024-04-22T01:18:23.375788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:39.886334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.394032+0000","last_change":"2024-04-22T01:18:23.366116+0000","last_active":"2024-04-22T01:18:23.394032+0000","last_peered":"2024-04-22T01:18:23.394032+0000","last_clean":"2024-04-22T01:18:23.394032+0000","last_became_active":"2024-04-22T01:18:23.363236+0000","last_became_peered":"2024-04-22T01:18:23.363236+0000","last_unstale":"2024-04-22T01:18:23.394032+0000","last_undegraded":"2024-04-22T01:18:23.394032+0000","last_fullsized":"2024-04-22T01:18:23.394032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:33.772940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.372742+0000","last_change":"2024-04-22T01:18:23.372039+0000","last_active":"2024-04-22T01:18:23.372742+0000","last_peered":"2024-04-22T01:18:23.372742+0000","last_clean":"2024-04-22T01:18:23.372742+0000","last_became_active":"2024-04-22T01:18:23.371085+0000","last_became_peered":"2024-04-22T01:18:23.371085+0000","last_unstale":"2024-04-22T01:18:23.372742+0000","last_undegraded":"2024-04-22T01:18:23.372742+0000","last_fullsized":"2024-04-22T01:18:23.372742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:06.709623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389185+0000","last_change":"2024-04-22T01:18:23.378538+0000","last_active":"2024-04-22T01:18:23.389185+0000","last_peered":"2024-04-22T01:18:23.389185+0000","last_clean":"2024-04-22T01:18:23.389185+0000","last_became_active":"2024-04-22T01:18:23.377988+0000","last_became_peered":"2024-04-22T01:18:23.377988+0000","last_unstale":"2024-04-22T01:18:23.389185+0000","last_undegraded":"2024-04-22T01:18:23.389185+0000","last_fullsized":"2024-04-22T01:18:23.389185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:50.986562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.373090+0000","last_change":"2024-04-22T01:18:23.369080+0000","last_active":"2024-04-22T01:18:23.373090+0000","last_peered":"2024-04-22T01:18:23.373090+0000","last_clean":"2024-04-22T01:18:23.373090+0000","last_became_active":"2024-04-22T01:18:23.368764+0000","last_became_peered":"2024-04-22T01:18:23.368764+0000","last_unstale":"2024-04-22T01:18:23.373090+0000","last_undegraded":"2024-04-22T01:18:23.373090+0000","last_fullsized":"2024-04-22T01:18:23.373090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:13.713535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380354+0000","last_change":"2024-04-22T01:18:23.358977+0000","last_active":"2024-04-22T01:18:23.380354+0000","last_peered":"2024-04-22T01:18:23.380354+0000","last_clean":"2024-04-22T01:18:23.380354+0000","last_became_active":"2024-04-22T01:18:23.358313+0000","last_became_peered":"2024-04-22T01:18:23.358313+0000","last_unstale":"2024-04-22T01:18:23.380354+0000","last_undegraded":"2024-04-22T01:18:23.380354+0000","last_fullsized":"2024-04-22T01:18:23.380354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:42.230469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.363607+0000","last_change":"2024-04-22T01:18:23.353710+0000","last_active":"2024-04-22T01:18:23.363607+0000","last_peered":"2024-04-22T01:18:23.363607+0000","last_clean":"2024-04-22T01:18:23.363607+0000","last_became_active":"2024-04-22T01:18:23.353435+0000","last_became_peered":"2024-04-22T01:18:23.353435+0000","last_unstale":"2024-04-22T01:18:23.363607+0000","last_undegraded":"2024-04-22T01:18:23.363607+0000","last_fullsized":"2024-04-22T01:18:23.363607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:00.754122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.371865+0000","last_change":"2024-04-22T01:18:23.370394+0000","last_active":"2024-04-22T01:18:23.371865+0000","last_peered":"2024-04-22T01:18:23.371865+0000","last_clean":"2024-04-22T01:18:23.371865+0000","last_became_active":"2024-04-22T01:18:23.370087+0000","last_became_peered":"2024-04-22T01:18:23.370087+0000","last_unstale":"2024-04-22T01:18:23.371865+0000","last_undegraded":"2024-04-22T01:18:23.371865+0000","last_fullsized":"2024-04-22T01:18:23.371865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:47.698529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388717+0000","last_change":"2024-04-22T01:18:23.373090+0000","last_active":"2024-04-22T01:18:23.388717+0000","last_peered":"2024-04-22T01:18:23.388717+0000","last_clean":"2024-04-22T01:18:23.388717+0000","last_became_active":"2024-04-22T01:18:23.371665+0000","last_became_peered":"2024-04-22T01:18:23.371665+0000","last_unstale":"2024-04-22T01:18:23.388717+0000","last_undegraded":"2024-04-22T01:18:23.388717+0000","last_fullsized":"2024-04-22T01:18:23.388717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:33.547644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.367809+0000","last_change":"2024-04-22T01:18:23.366160+0000","last_active":"2024-04-22T01:18:23.367809+0000","last_peered":"2024-04-22T01:18:23.367809+0000","last_clean":"2024-04-22T01:18:23.367809+0000","last_became_active":"2024-04-22T01:18:23.365884+0000","last_became_peered":"2024-04-22T01:18:23.365884+0000","last_unstale":"2024-04-22T01:18:23.367809+0000","last_undegraded":"2024-04-22T01:18:23.367809+0000","last_fullsized":"2024-04-22T01:18:23.367809+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:30:18.800811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.374016+0000","last_change":"2024-04-22T01:18:23.359705+0000","last_active":"2024-04-22T01:18:23.374016+0000","last_peered":"2024-04-22T01:18:23.374016+0000","last_clean":"2024-04-22T01:18:23.374016+0000","last_became_active":"2024-04-22T01:18:23.359199+0000","last_became_peered":"2024-04-22T01:18:23.359199+0000","last_unstale":"2024-04-22T01:18:23.374016+0000","last_undegraded":"2024-04-22T01:18:23.374016+0000","last_fullsized":"2024-04-22T01:18:23.374016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:33.772940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376050+0000","last_change":"2024-04-22T01:18:23.366091+0000","last_active":"2024-04-22T01:18:23.376050+0000","last_peered":"2024-04-22T01:18:23.376050+0000","last_clean":"2024-04-22T01:18:23.376050+0000","last_became_active":"2024-04-22T01:18:23.365839+0000","last_became_peered":"2024-04-22T01:18:23.365839+0000","last_unstale":"2024-04-22T01:18:23.376050+0000","last_undegraded":"2024-04-22T01:18:23.376050+0000","last_fullsized":"2024-04-22T01:18:23.376050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:09.734932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.377202+0000","last_change":"2024-04-22T01:18:23.376783+0000","last_active":"2024-04-22T01:18:23.377202+0000","last_peered":"2024-04-22T01:18:23.377202+0000","last_clean":"2024-04-22T01:18:23.377202+0000","last_became_active":"2024-04-22T01:18:23.376477+0000","last_became_peered":"2024-04-22T01:18:23.376477+0000","last_unstale":"2024-04-22T01:18:23.377202+0000","last_undegraded":"2024-04-22T01:18:23.377202+0000","last_fullsized":"2024-04-22T01:18:23.377202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:45.480558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388495+0000","last_change":"2024-04-22T01:18:23.370434+0000","last_active":"2024-04-22T01:18:23.388495+0000","last_peered":"2024-04-22T01:18:23.388495+0000","last_clean":"2024-04-22T01:18:23.388495+0000","last_became_active":"2024-04-22T01:18:23.369587+0000","last_became_peered":"2024-04-22T01:18:23.369587+0000","last_unstale":"2024-04-22T01:18:23.388495+0000","last_undegraded":"2024-04-22T01:18:23.388495+0000","last_fullsized":"2024-04-22T01:18:23.388495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:19:42.139908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.369464+0000","last_change":"2024-04-22T01:18:23.367153+0000","last_active":"2024-04-22T01:18:23.369464+0000","last_peered":"2024-04-22T01:18:23.369464+0000","last_clean":"2024-04-22T01:18:23.369464+0000","last_became_active":"2024-04-22T01:18:23.366641+0000","last_became_peered":"2024-04-22T01:18:23.366641+0000","last_unstale":"2024-04-22T01:18:23.369464+0000","last_undegraded":"2024-04-22T01:18:23.369464+0000","last_fullsized":"2024-04-22T01:18:23.369464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:55.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388503+0000","last_change":"2024-04-22T01:18:23.370462+0000","last_active":"2024-04-22T01:18:23.388503+0000","last_peered":"2024-04-22T01:18:23.388503+0000","last_clean":"2024-04-22T01:18:23.388503+0000","last_became_active":"2024-04-22T01:18:23.369664+0000","last_became_peered":"2024-04-22T01:18:23.369664+0000","last_unstale":"2024-04-22T01:18:23.388503+0000","last_undegraded":"2024-04-22T01:18:23.388503+0000","last_fullsized":"2024-04-22T01:18:23.388503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:10.172833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388666+0000","last_change":"2024-04-22T01:18:23.372400+0000","last_active":"2024-04-22T01:18:23.388666+0000","last_peered":"2024-04-22T01:18:23.388666+0000","last_clean":"2024-04-22T01:18:23.388666+0000","last_became_active":"2024-04-22T01:18:23.370787+0000","last_became_peered":"2024-04-22T01:18:23.370787+0000","last_unstale":"2024-04-22T01:18:23.388666+0000","last_undegraded":"2024-04-22T01:18:23.388666+0000","last_fullsized":"2024-04-22T01:18:23.388666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:21:48.509140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393132+0000","last_change":"2024-04-22T01:18:23.373690+0000","last_active":"2024-04-22T01:18:23.393132+0000","last_peered":"2024-04-22T01:18:23.393132+0000","last_clean":"2024-04-22T01:18:23.393132+0000","last_became_active":"2024-04-22T01:18:23.372709+0000","last_became_peered":"2024-04-22T01:18:23.372709+0000","last_unstale":"2024-04-22T01:18:23.393132+0000","last_undegraded":"2024-04-22T01:18:23.393132+0000","last_fullsized":"2024-04-22T01:18:23.393132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:55.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.382674+0000","last_change":"2024-04-22T01:18:23.381973+0000","last_active":"2024-04-22T01:18:23.382674+0000","last_peered":"2024-04-22T01:18:23.382674+0000","last_clean":"2024-04-22T01:18:23.382674+0000","last_became_active":"2024-04-22T01:18:23.381683+0000","last_became_peered":"2024-04-22T01:18:23.381683+0000","last_unstale":"2024-04-22T01:18:23.382674+0000","last_undegraded":"2024-04-22T01:18:23.382674+0000","last_fullsized":"2024-04-22T01:18:23.382674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:36.487175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388135+0000","last_change":"2024-04-22T01:18:23.376719+0000","last_active":"2024-04-22T01:18:23.388135+0000","last_peered":"2024-04-22T01:18:23.388135+0000","last_clean":"2024-04-22T01:18:23.388135+0000","last_became_active":"2024-04-22T01:18:23.376017+0000","last_became_peered":"2024-04-22T01:18:23.376017+0000","last_unstale":"2024-04-22T01:18:23.388135+0000","last_undegraded":"2024-04-22T01:18:23.388135+0000","last_fullsized":"2024-04-22T01:18:23.388135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:13.120670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380790+0000","last_change":"2024-04-22T01:18:23.364637+0000","last_active":"2024-04-22T01:18:23.380790+0000","last_peered":"2024-04-22T01:18:23.380790+0000","last_clean":"2024-04-22T01:18:23.380790+0000","last_became_active":"2024-04-22T01:18:23.363899+0000","last_became_peered":"2024-04-22T01:18:23.363899+0000","last_unstale":"2024-04-22T01:18:23.380790+0000","last_undegraded":"2024-04-22T01:18:23.380790+0000","last_fullsized":"2024-04-22T01:18:23.380790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:19.846325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376355+0000","last_change":"2024-04-22T01:18:23.373710+0000","last_active":"2024-04-22T01:18:23.376355+0000","last_peered":"2024-04-22T01:18:23.376355+0000","last_clean":"2024-04-22T01:18:23.376355+0000","last_became_active":"2024-04-22T01:18:23.373404+0000","last_became_peered":"2024-04-22T01:18:23.373404+0000","last_unstale":"2024-04-22T01:18:23.376355+0000","last_undegraded":"2024-04-22T01:18:23.376355+0000","last_fullsized":"2024-04-22T01:18:23.376355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:32.350846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383045+0000","last_change":"2024-04-22T01:18:23.382375+0000","last_active":"2024-04-22T01:18:23.383045+0000","last_peered":"2024-04-22T01:18:23.383045+0000","last_clean":"2024-04-22T01:18:23.383045+0000","last_became_active":"2024-04-22T01:18:23.382104+0000","last_became_peered":"2024-04-22T01:18:23.382104+0000","last_unstale":"2024-04-22T01:18:23.383045+0000","last_undegraded":"2024-04-22T01:18:23.383045+0000","last_fullsized":"2024-04-22T01:18:23.383045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:33.430057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396169+0000","last_change":"2024-04-22T01:18:23.384163+0000","last_active":"2024-04-22T01:18:23.396169+0000","last_peered":"2024-04-22T01:18:23.396169+0000","last_clean":"2024-04-22T01:18:23.396169+0000","last_became_active":"2024-04-22T01:18:23.382028+0000","last_became_peered":"2024-04-22T01:18:23.382028+0000","last_unstale":"2024-04-22T01:18:23.396169+0000","last_undegraded":"2024-04-22T01:18:23.396169+0000","last_fullsized":"2024-04-22T01:18:23.396169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:30.422069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389072+0000","last_change":"2024-04-22T01:18:23.378952+0000","last_active":"2024-04-22T01:18:23.389072+0000","last_peered":"2024-04-22T01:18:23.389072+0000","last_clean":"2024-04-22T01:18:23.389072+0000","last_became_active":"2024-04-22T01:18:23.378210+0000","last_became_peered":"2024-04-22T01:18:23.378210+0000","last_unstale":"2024-04-22T01:18:23.389072+0000","last_undegraded":"2024-04-22T01:18:23.389072+0000","last_fullsized":"2024-04-22T01:18:23.389072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:24.136319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376865+0000","last_change":"2024-04-22T01:18:23.370174+0000","last_active":"2024-04-22T01:18:23.376865+0000","last_peered":"2024-04-22T01:18:23.376865+0000","last_clean":"2024-04-22T01:18:23.376865+0000","last_became_active":"2024-04-22T01:18:23.369296+0000","last_became_peered":"2024-04-22T01:18:23.369296+0000","last_unstale":"2024-04-22T01:18:23.376865+0000","last_undegraded":"2024-04-22T01:18:23.376865+0000","last_fullsized":"2024-04-22T01:18:23.376865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:46.301034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.373995+0000","last_change":"2024-04-22T01:18:23.359469+0000","last_active":"2024-04-22T01:18:23.373995+0000","last_peered":"2024-04-22T01:18:23.373995+0000","last_clean":"2024-04-22T01:18:23.373995+0000","last_became_active":"2024-04-22T01:18:23.358812+0000","last_became_peered":"2024-04-22T01:18:23.358812+0000","last_unstale":"2024-04-22T01:18:23.373995+0000","last_undegraded":"2024-04-22T01:18:23.373995+0000","last_fullsized":"2024-04-22T01:18:23.373995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:59.963726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383370+0000","last_change":"2024-04-22T01:18:23.382664+0000","last_active":"2024-04-22T01:18:23.383370+0000","last_peered":"2024-04-22T01:18:23.383370+0000","last_clean":"2024-04-22T01:18:23.383370+0000","last_became_active":"2024-04-22T01:18:23.381635+0000","last_became_peered":"2024-04-22T01:18:23.381635+0000","last_unstale":"2024-04-22T01:18:23.383370+0000","last_undegraded":"2024-04-22T01:18:23.383370+0000","last_fullsized":"2024-04-22T01:18:23.383370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:10.075040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"88'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393284+0000","last_change":"2024-04-22T01:18:22.336359+0000","last_active":"2024-04-22T01:18:23.393284+0000","last_peered":"2024-04-22T01:18:23.393284+0000","last_clean":"2024-04-22T01:18:23.393284+0000","last_became_active":"2024-04-22T01:18:22.335616+0000","last_became_peered":"2024-04-22T01:18:22.335616+0000","last_unstale":"2024-04-22T01:18:23.393284+0000","last_undegraded":"2024-04-22T01:18:23.393284+0000","last_fullsized":"2024-04-22T01:18:23.393284+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:56.913141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375888+0000","last_change":"2024-04-22T01:18:23.372439+0000","last_active":"2024-04-22T01:18:23.375888+0000","last_peered":"2024-04-22T01:18:23.375888+0000","last_clean":"2024-04-22T01:18:23.375888+0000","last_became_active":"2024-04-22T01:18:23.371203+0000","last_became_peered":"2024-04-22T01:18:23.371203+0000","last_unstale":"2024-04-22T01:18:23.375888+0000","last_undegraded":"2024-04-22T01:18:23.375888+0000","last_fullsized":"2024-04-22T01:18:23.375888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:36.381347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.387226+0000","last_change":"2024-04-22T01:18:22.354120+0000","last_active":"2024-04-22T01:18:23.387226+0000","last_peered":"2024-04-22T01:18:23.387226+0000","last_clean":"2024-04-22T01:18:23.387226+0000","last_became_active":"2024-04-22T01:18:22.353365+0000","last_became_peered":"2024-04-22T01:18:22.353365+0000","last_unstale":"2024-04-22T01:18:23.387226+0000","last_undegraded":"2024-04-22T01:18:23.387226+0000","last_fullsized":"2024-04-22T01:18:23.387226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:45.987866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396430+0000","last_change":"2024-04-22T01:18:23.384300+0000","last_active":"2024-04-22T01:18:23.396430+0000","last_peered":"2024-04-22T01:18:23.396430+0000","last_clean":"2024-04-22T01:18:23.396430+0000","last_became_active":"2024-04-22T01:18:23.383461+0000","last_became_peered":"2024-04-22T01:18:23.383461+0000","last_unstale":"2024-04-22T01:18:23.396430+0000","last_undegraded":"2024-04-22T01:18:23.396430+0000","last_fullsized":"2024-04-22T01:18:23.396430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:40:11.205592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.394147+0000","last_change":"2024-04-22T01:18:22.338794+0000","last_active":"2024-04-22T01:18:23.394147+0000","last_peered":"2024-04-22T01:18:23.394147+0000","last_clean":"2024-04-22T01:18:23.394147+0000","last_became_active":"2024-04-22T01:18:22.338008+0000","last_became_peered":"2024-04-22T01:18:22.338008+0000","last_unstale":"2024-04-22T01:18:23.394147+0000","last_undegraded":"2024-04-22T01:18:23.394147+0000","last_fullsized":"2024-04-22T01:18:23.394147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:39.116917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.391434+0000","last_change":"2024-04-22T01:18:23.390674+0000","last_active":"2024-04-22T01:18:23.391434+0000","last_peered":"2024-04-22T01:18:23.391434+0000","last_clean":"2024-04-22T01:18:23.391434+0000","last_became_active":"2024-04-22T01:18:23.390360+0000","last_became_peered":"2024-04-22T01:18:23.390360+0000","last_unstale":"2024-04-22T01:18:23.391434+0000","last_undegraded":"2024-04-22T01:18:23.391434+0000","last_fullsized":"2024-04-22T01:18:23.391434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:56:12.400145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.387022+0000","last_change":"2024-04-22T01:18:22.345395+0000","last_active":"2024-04-22T01:18:23.387022+0000","last_peered":"2024-04-22T01:18:23.387022+0000","last_clean":"2024-04-22T01:18:23.387022+0000","last_became_active":"2024-04-22T01:18:22.343707+0000","last_became_peered":"2024-04-22T01:18:22.343707+0000","last_unstale":"2024-04-22T01:18:23.387022+0000","last_undegraded":"2024-04-22T01:18:23.387022+0000","last_fullsized":"2024-04-22T01:18:23.387022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:49.973842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.369507+0000","last_change":"2024-04-22T01:18:23.366992+0000","last_active":"2024-04-22T01:18:23.369507+0000","last_peered":"2024-04-22T01:18:23.369507+0000","last_clean":"2024-04-22T01:18:23.369507+0000","last_became_active":"2024-04-22T01:18:23.366585+0000","last_became_peered":"2024-04-22T01:18:23.366585+0000","last_unstale":"2024-04-22T01:18:23.369507+0000","last_undegraded":"2024-04-22T01:18:23.369507+0000","last_fullsized":"2024-04-22T01:18:23.369507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:06.348633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380306+0000","last_change":"2024-04-22T01:18:22.354314+0000","last_active":"2024-04-22T01:18:23.380306+0000","last_peered":"2024-04-22T01:18:23.380306+0000","last_clean":"2024-04-22T01:18:23.380306+0000","last_became_active":"2024-04-22T01:18:22.351553+0000","last_became_peered":"2024-04-22T01:18:22.351553+0000","last_unstale":"2024-04-22T01:18:23.380306+0000","last_undegraded":"2024-04-22T01:18:23.380306+0000","last_fullsized":"2024-04-22T01:18:23.380306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:24.065014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393192+0000","last_change":"2024-04-22T01:18:23.383305+0000","last_active":"2024-04-22T01:18:23.393192+0000","last_peered":"2024-04-22T01:18:23.393192+0000","last_clean":"2024-04-22T01:18:23.393192+0000","last_became_active":"2024-04-22T01:18:23.382786+0000","last_became_peered":"2024-04-22T01:18:23.382786+0000","last_unstale":"2024-04-22T01:18:23.393192+0000","last_undegraded":"2024-04-22T01:18:23.393192+0000","last_fullsized":"2024-04-22T01:18:23.393192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:47.698529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.374592+0000","last_change":"2024-04-22T01:18:22.337177+0000","last_active":"2024-04-22T01:18:23.374592+0000","last_peered":"2024-04-22T01:18:23.374592+0000","last_clean":"2024-04-22T01:18:23.374592+0000","last_became_active":"2024-04-22T01:18:22.336803+0000","last_became_peered":"2024-04-22T01:18:22.336803+0000","last_unstale":"2024-04-22T01:18:23.374592+0000","last_undegraded":"2024-04-22T01:18:23.374592+0000","last_fullsized":"2024-04-22T01:18:23.374592+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:29:56.913141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375815+0000","last_change":"2024-04-22T01:18:23.371985+0000","last_active":"2024-04-22T01:18:23.375815+0000","last_peered":"2024-04-22T01:18:23.375815+0000","last_clean":"2024-04-22T01:18:23.375815+0000","last_became_active":"2024-04-22T01:18:23.371594+0000","last_became_peered":"2024-04-22T01:18:23.371594+0000","last_unstale":"2024-04-22T01:18:23.375815+0000","last_undegraded":"2024-04-22T01:18:23.375815+0000","last_fullsized":"2024-04-22T01:18:23.375815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:33.515501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.351748+0000","last_change":"2024-04-22T01:18:22.331579+0000","last_active":"2024-04-22T01:18:23.351748+0000","last_peered":"2024-04-22T01:18:23.351748+0000","last_clean":"2024-04-22T01:18:23.351748+0000","last_became_active":"2024-04-22T01:18:22.331147+0000","last_became_peered":"2024-04-22T01:18:22.331147+0000","last_unstale":"2024-04-22T01:18:23.351748+0000","last_undegraded":"2024-04-22T01:18:23.351748+0000","last_fullsized":"2024-04-22T01:18:23.351748+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.107951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388660+0000","last_change":"2024-04-22T01:18:23.388210+0000","last_active":"2024-04-22T01:18:23.388660+0000","last_peered":"2024-04-22T01:18:23.388660+0000","last_clean":"2024-04-22T01:18:23.388660+0000","last_became_active":"2024-04-22T01:18:23.387959+0000","last_became_peered":"2024-04-22T01:18:23.387959+0000","last_unstale":"2024-04-22T01:18:23.388660+0000","last_undegraded":"2024-04-22T01:18:23.388660+0000","last_fullsized":"2024-04-22T01:18:23.388660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:41.371744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.351144+0000","last_change":"2024-04-22T01:18:22.331454+0000","last_active":"2024-04-22T01:18:23.351144+0000","last_peered":"2024-04-22T01:18:23.351144+0000","last_clean":"2024-04-22T01:18:23.351144+0000","last_became_active":"2024-04-22T01:18:22.330721+0000","last_became_peered":"2024-04-22T01:18:22.330721+0000","last_unstale":"2024-04-22T01:18:23.351144+0000","last_undegraded":"2024-04-22T01:18:23.351144+0000","last_fullsized":"2024-04-22T01:18:23.351144+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:35.474455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380417+0000","last_change":"2024-04-22T01:18:23.354365+0000","last_active":"2024-04-22T01:18:23.380417+0000","last_peered":"2024-04-22T01:18:23.380417+0000","last_clean":"2024-04-22T01:18:23.380417+0000","last_became_active":"2024-04-22T01:18:23.353697+0000","last_became_peered":"2024-04-22T01:18:23.353697+0000","last_unstale":"2024-04-22T01:18:23.380417+0000","last_undegraded":"2024-04-22T01:18:23.380417+0000","last_fullsized":"2024-04-22T01:18:23.380417+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:08:43.503423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.347030+0000","last_change":"2024-04-22T01:18:22.342152+0000","last_active":"2024-04-22T01:18:23.347030+0000","last_peered":"2024-04-22T01:18:23.347030+0000","last_clean":"2024-04-22T01:18:23.347030+0000","last_became_active":"2024-04-22T01:18:22.341409+0000","last_became_peered":"2024-04-22T01:18:22.341409+0000","last_unstale":"2024-04-22T01:18:23.347030+0000","last_undegraded":"2024-04-22T01:18:23.347030+0000","last_fullsized":"2024-04-22T01:18:23.347030+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:44.836785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396421+0000","last_change":"2024-04-22T01:18:23.391833+0000","last_active":"2024-04-22T01:18:23.396421+0000","last_peered":"2024-04-22T01:18:23.396421+0000","last_clean":"2024-04-22T01:18:23.396421+0000","last_became_active":"2024-04-22T01:18:23.390929+0000","last_became_peered":"2024-04-22T01:18:23.390929+0000","last_unstale":"2024-04-22T01:18:23.396421+0000","last_undegraded":"2024-04-22T01:18:23.396421+0000","last_fullsized":"2024-04-22T01:18:23.396421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:09.734932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.348868+0000","last_change":"2024-04-22T01:18:22.334500+0000","last_active":"2024-04-22T01:18:23.348868+0000","last_peered":"2024-04-22T01:18:23.348868+0000","last_clean":"2024-04-22T01:18:23.348868+0000","last_became_active":"2024-04-22T01:18:22.334090+0000","last_became_peered":"2024-04-22T01:18:22.334090+0000","last_unstale":"2024-04-22T01:18:23.348868+0000","last_undegraded":"2024-04-22T01:18:23.348868+0000","last_fullsized":"2024-04-22T01:18:23.348868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:30.037113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.364788+0000","last_change":"2024-04-22T01:18:23.360412+0000","last_active":"2024-04-22T01:18:23.364788+0000","last_peered":"2024-04-22T01:18:23.364788+0000","last_clean":"2024-04-22T01:18:23.364788+0000","last_became_active":"2024-04-22T01:18:23.360065+0000","last_became_peered":"2024-04-22T01:18:23.360065+0000","last_unstale":"2024-04-22T01:18:23.364788+0000","last_undegraded":"2024-04-22T01:18:23.364788+0000","last_fullsized":"2024-04-22T01:18:23.364788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:10.995821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"88'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.381313+0000","last_change":"2024-04-22T01:18:22.356445+0000","last_active":"2024-04-22T01:18:23.381313+0000","last_peered":"2024-04-22T01:18:23.381313+0000","last_clean":"2024-04-22T01:18:23.381313+0000","last_became_active":"2024-04-22T01:18:22.354380+0000","last_became_peered":"2024-04-22T01:18:22.354380+0000","last_unstale":"2024-04-22T01:18:23.381313+0000","last_undegraded":"2024-04-22T01:18:23.381313+0000","last_fullsized":"2024-04-22T01:18:23.381313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:55.777815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.379510+0000","last_change":"2024-04-22T01:18:23.379153+0000","last_active":"2024-04-22T01:18:23.379510+0000","last_peered":"2024-04-22T01:18:23.379510+0000","last_clean":"2024-04-22T01:18:23.379510+0000","last_became_active":"2024-04-22T01:18:23.378861+0000","last_became_peered":"2024-04-22T01:18:23.378861+0000","last_unstale":"2024-04-22T01:18:23.379510+0000","last_undegraded":"2024-04-22T01:18:23.379510+0000","last_fullsized":"2024-04-22T01:18:23.379510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:45.849505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"88'6","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.395720+0000","last_change":"2024-04-22T01:18:22.343944+0000","last_active":"2024-04-22T01:18:23.395720+0000","last_peered":"2024-04-22T01:18:23.395720+0000","last_clean":"2024-04-22T01:18:23.395720+0000","last_became_active":"2024-04-22T01:18:22.343522+0000","last_became_peered":"2024-04-22T01:18:22.343522+0000","last_unstale":"2024-04-22T01:18:23.395720+0000","last_undegraded":"2024-04-22T01:18:23.395720+0000","last_fullsized":"2024-04-22T01:18:23.395720+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:44.893186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.368029+0000","last_change":"2024-04-22T01:18:23.366766+0000","last_active":"2024-04-22T01:18:23.368029+0000","last_peered":"2024-04-22T01:18:23.368029+0000","last_clean":"2024-04-22T01:18:23.368029+0000","last_became_active":"2024-04-22T01:18:23.365916+0000","last_became_peered":"2024-04-22T01:18:23.365916+0000","last_unstale":"2024-04-22T01:18:23.368029+0000","last_undegraded":"2024-04-22T01:18:23.368029+0000","last_fullsized":"2024-04-22T01:18:23.368029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:35.797753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396477+0000","last_change":"2024-04-22T01:18:22.340410+0000","last_active":"2024-04-22T01:18:23.396477+0000","last_peered":"2024-04-22T01:18:23.396477+0000","last_clean":"2024-04-22T01:18:23.396477+0000","last_became_active":"2024-04-22T01:18:22.339810+0000","last_became_peered":"2024-04-22T01:18:22.339810+0000","last_unstale":"2024-04-22T01:18:23.396477+0000","last_undegraded":"2024-04-22T01:18:23.396477+0000","last_fullsized":"2024-04-22T01:18:23.396477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:58.951006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.388958+0000","last_change":"2024-04-22T01:18:23.376607+0000","last_active":"2024-04-22T01:18:23.388958+0000","last_peered":"2024-04-22T01:18:23.388958+0000","last_clean":"2024-04-22T01:18:23.388958+0000","last_became_active":"2024-04-22T01:18:23.375893+0000","last_became_peered":"2024-04-22T01:18:23.375893+0000","last_unstale":"2024-04-22T01:18:23.388958+0000","last_undegraded":"2024-04-22T01:18:23.388958+0000","last_fullsized":"2024-04-22T01:18:23.388958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:20.217880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.354738+0000","last_change":"2024-04-22T01:18:22.348718+0000","last_active":"2024-04-22T01:18:23.354738+0000","last_peered":"2024-04-22T01:18:23.354738+0000","last_clean":"2024-04-22T01:18:23.354738+0000","last_became_active":"2024-04-22T01:18:22.348360+0000","last_became_peered":"2024-04-22T01:18:22.348360+0000","last_unstale":"2024-04-22T01:18:23.354738+0000","last_undegraded":"2024-04-22T01:18:23.354738+0000","last_fullsized":"2024-04-22T01:18:23.354738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:03.205988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.372570+0000","last_change":"2024-04-22T01:18:23.368527+0000","last_active":"2024-04-22T01:18:23.372570+0000","last_peered":"2024-04-22T01:18:23.372570+0000","last_clean":"2024-04-22T01:18:23.372570+0000","last_became_active":"2024-04-22T01:18:23.366030+0000","last_became_peered":"2024-04-22T01:18:23.366030+0000","last_unstale":"2024-04-22T01:18:23.372570+0000","last_undegraded":"2024-04-22T01:18:23.372570+0000","last_fullsized":"2024-04-22T01:18:23.372570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:27:50.968614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.360036+0000","last_change":"2024-04-22T01:18:22.350212+0000","last_active":"2024-04-22T01:18:23.360036+0000","last_peered":"2024-04-22T01:18:23.360036+0000","last_clean":"2024-04-22T01:18:23.360036+0000","last_became_active":"2024-04-22T01:18:22.349847+0000","last_became_peered":"2024-04-22T01:18:22.349847+0000","last_unstale":"2024-04-22T01:18:23.360036+0000","last_undegraded":"2024-04-22T01:18:23.360036+0000","last_fullsized":"2024-04-22T01:18:23.360036+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:49:49.973842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.376320+0000","last_change":"2024-04-22T01:18:23.375253+0000","last_active":"2024-04-22T01:18:23.376320+0000","last_peered":"2024-04-22T01:18:23.376320+0000","last_clean":"2024-04-22T01:18:23.376320+0000","last_became_active":"2024-04-22T01:18:23.374992+0000","last_became_peered":"2024-04-22T01:18:23.374992+0000","last_unstale":"2024-04-22T01:18:23.376320+0000","last_undegraded":"2024-04-22T01:18:23.376320+0000","last_fullsized":"2024-04-22T01:18:23.376320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:11:35.415321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"88'8","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.374871+0000","last_change":"2024-04-22T01:18:22.340743+0000","last_active":"2024-04-22T01:18:23.374871+0000","last_peered":"2024-04-22T01:18:23.374871+0000","last_clean":"2024-04-22T01:18:23.374871+0000","last_became_active":"2024-04-22T01:18:22.337246+0000","last_became_peered":"2024-04-22T01:18:22.337246+0000","last_unstale":"2024-04-22T01:18:23.374871+0000","last_undegraded":"2024-04-22T01:18:23.374871+0000","last_fullsized":"2024-04-22T01:18:23.374871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:00.605962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383060+0000","last_change":"2024-04-22T01:18:23.382402+0000","last_active":"2024-04-22T01:18:23.383060+0000","last_peered":"2024-04-22T01:18:23.383060+0000","last_clean":"2024-04-22T01:18:23.383060+0000","last_became_active":"2024-04-22T01:18:23.381918+0000","last_became_peered":"2024-04-22T01:18:23.381918+0000","last_unstale":"2024-04-22T01:18:23.383060+0000","last_undegraded":"2024-04-22T01:18:23.383060+0000","last_fullsized":"2024-04-22T01:18:23.383060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:29.397829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396917+0000","last_change":"2024-04-22T01:18:22.343048+0000","last_active":"2024-04-22T01:18:23.396917+0000","last_peered":"2024-04-22T01:18:23.396917+0000","last_clean":"2024-04-22T01:18:23.396917+0000","last_became_active":"2024-04-22T01:18:22.342341+0000","last_became_peered":"2024-04-22T01:18:22.342341+0000","last_unstale":"2024-04-22T01:18:23.396917+0000","last_undegraded":"2024-04-22T01:18:23.396917+0000","last_fullsized":"2024-04-22T01:18:23.396917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:30:17.788092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393415+0000","last_change":"2024-04-22T01:18:23.385370+0000","last_active":"2024-04-22T01:18:23.393415+0000","last_peered":"2024-04-22T01:18:23.393415+0000","last_clean":"2024-04-22T01:18:23.393415+0000","last_became_active":"2024-04-22T01:18:23.384673+0000","last_became_peered":"2024-04-22T01:18:23.384673+0000","last_unstale":"2024-04-22T01:18:23.393415+0000","last_undegraded":"2024-04-22T01:18:23.393415+0000","last_fullsized":"2024-04-22T01:18:23.393415+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:06.709623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396325+0000","last_change":"2024-04-22T01:18:22.338973+0000","last_active":"2024-04-22T01:18:23.396325+0000","last_peered":"2024-04-22T01:18:23.396325+0000","last_clean":"2024-04-22T01:18:23.396325+0000","last_became_active":"2024-04-22T01:18:22.338040+0000","last_became_peered":"2024-04-22T01:18:22.338040+0000","last_unstale":"2024-04-22T01:18:23.396325+0000","last_undegraded":"2024-04-22T01:18:23.396325+0000","last_fullsized":"2024-04-22T01:18:23.396325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:44.893186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.380736+0000","last_change":"2024-04-22T01:18:23.363194+0000","last_active":"2024-04-22T01:18:23.380736+0000","last_peered":"2024-04-22T01:18:23.380736+0000","last_clean":"2024-04-22T01:18:23.380736+0000","last_became_active":"2024-04-22T01:18:23.362475+0000","last_became_peered":"2024-04-22T01:18:23.362475+0000","last_unstale":"2024-04-22T01:18:23.380736+0000","last_undegraded":"2024-04-22T01:18:23.380736+0000","last_fullsized":"2024-04-22T01:18:23.380736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:31.724530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.381483+0000","last_change":"2024-04-22T01:18:22.353742+0000","last_active":"2024-04-22T01:18:23.381483+0000","last_peered":"2024-04-22T01:18:23.381483+0000","last_clean":"2024-04-22T01:18:23.381483+0000","last_became_active":"2024-04-22T01:18:22.353105+0000","last_became_peered":"2024-04-22T01:18:22.353105+0000","last_unstale":"2024-04-22T01:18:23.381483+0000","last_undegraded":"2024-04-22T01:18:23.381483+0000","last_fullsized":"2024-04-22T01:18:23.381483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:07.676297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375859+0000","last_change":"2024-04-22T01:18:23.372009+0000","last_active":"2024-04-22T01:18:23.375859+0000","last_peered":"2024-04-22T01:18:23.375859+0000","last_clean":"2024-04-22T01:18:23.375859+0000","last_became_active":"2024-04-22T01:18:23.371729+0000","last_became_peered":"2024-04-22T01:18:23.371729+0000","last_unstale":"2024-04-22T01:18:23.375859+0000","last_undegraded":"2024-04-22T01:18:23.375859+0000","last_fullsized":"2024-04-22T01:18:23.375859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:12:45.354994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.379561+0000","last_change":"2024-04-22T01:18:22.331759+0000","last_active":"2024-04-22T01:18:23.379561+0000","last_peered":"2024-04-22T01:18:23.379561+0000","last_clean":"2024-04-22T01:18:23.379561+0000","last_became_active":"2024-04-22T01:18:22.330770+0000","last_became_peered":"2024-04-22T01:18:22.330770+0000","last_unstale":"2024-04-22T01:18:23.379561+0000","last_undegraded":"2024-04-22T01:18:23.379561+0000","last_fullsized":"2024-04-22T01:18:23.379561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:46.685809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.386221+0000","last_change":"2024-04-22T01:18:23.377105+0000","last_active":"2024-04-22T01:18:23.386221+0000","last_peered":"2024-04-22T01:18:23.386221+0000","last_clean":"2024-04-22T01:18:23.386221+0000","last_became_active":"2024-04-22T01:18:23.375338+0000","last_became_peered":"2024-04-22T01:18:23.375338+0000","last_unstale":"2024-04-22T01:18:23.386221+0000","last_undegraded":"2024-04-22T01:18:23.386221+0000","last_fullsized":"2024-04-22T01:18:23.386221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:10.995821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.342316+0000","last_change":"2024-04-22T01:18:22.340985+0000","last_active":"2024-04-22T01:18:23.342316+0000","last_peered":"2024-04-22T01:18:23.342316+0000","last_clean":"2024-04-22T01:18:23.342316+0000","last_became_active":"2024-04-22T01:18:22.340645+0000","last_became_peered":"2024-04-22T01:18:22.340645+0000","last_unstale":"2024-04-22T01:18:23.342316+0000","last_undegraded":"2024-04-22T01:18:23.342316+0000","last_fullsized":"2024-04-22T01:18:23.342316+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:21:57.635981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.377100+0000","last_change":"2024-04-22T01:18:23.369879+0000","last_active":"2024-04-22T01:18:23.377100+0000","last_peered":"2024-04-22T01:18:23.377100+0000","last_clean":"2024-04-22T01:18:23.377100+0000","last_became_active":"2024-04-22T01:18:23.369119+0000","last_became_peered":"2024-04-22T01:18:23.369119+0000","last_unstale":"2024-04-22T01:18:23.377100+0000","last_undegraded":"2024-04-22T01:18:23.377100+0000","last_fullsized":"2024-04-22T01:18:23.377100+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:41.371744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.346798+0000","last_change":"2024-04-22T01:18:22.331656+0000","last_active":"2024-04-22T01:18:23.346798+0000","last_peered":"2024-04-22T01:18:23.346798+0000","last_clean":"2024-04-22T01:18:23.346798+0000","last_became_active":"2024-04-22T01:18:22.330698+0000","last_became_peered":"2024-04-22T01:18:22.330698+0000","last_unstale":"2024-04-22T01:18:23.346798+0000","last_undegraded":"2024-04-22T01:18:23.346798+0000","last_fullsized":"2024-04-22T01:18:23.346798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:05.696903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375550+0000","last_change":"2024-04-22T01:18:23.363172+0000","last_active":"2024-04-22T01:18:23.375550+0000","last_peered":"2024-04-22T01:18:23.375550+0000","last_clean":"2024-04-22T01:18:23.375550+0000","last_became_active":"2024-04-22T01:18:23.361665+0000","last_became_peered":"2024-04-22T01:18:23.361665+0000","last_unstale":"2024-04-22T01:18:23.375550+0000","last_undegraded":"2024-04-22T01:18:23.375550+0000","last_fullsized":"2024-04-22T01:18:23.375550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:40:11.205592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.341888+0000","last_change":"2024-04-22T01:18:22.337757+0000","last_active":"2024-04-22T01:18:23.341888+0000","last_peered":"2024-04-22T01:18:23.341888+0000","last_clean":"2024-04-22T01:18:23.341888+0000","last_became_active":"2024-04-22T01:18:22.337493+0000","last_became_peered":"2024-04-22T01:18:22.337493+0000","last_unstale":"2024-04-22T01:18:23.341888+0000","last_undegraded":"2024-04-22T01:18:23.341888+0000","last_fullsized":"2024-04-22T01:18:23.341888+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:35:31.947361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.393372+0000","last_change":"2024-04-22T01:18:23.385542+0000","last_active":"2024-04-22T01:18:23.393372+0000","last_peered":"2024-04-22T01:18:23.393372+0000","last_clean":"2024-04-22T01:18:23.393372+0000","last_became_active":"2024-04-22T01:18:23.384860+0000","last_became_peered":"2024-04-22T01:18:23.384860+0000","last_unstale":"2024-04-22T01:18:23.393372+0000","last_undegraded":"2024-04-22T01:18:23.393372+0000","last_fullsized":"2024-04-22T01:18:23.393372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:45.849505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396370+0000","last_change":"2024-04-22T01:18:22.339051+0000","last_active":"2024-04-22T01:18:23.396370+0000","last_peered":"2024-04-22T01:18:23.396370+0000","last_clean":"2024-04-22T01:18:23.396370+0000","last_became_active":"2024-04-22T01:18:22.338260+0000","last_became_peered":"2024-04-22T01:18:22.338260+0000","last_unstale":"2024-04-22T01:18:23.396370+0000","last_undegraded":"2024-04-22T01:18:23.396370+0000","last_fullsized":"2024-04-22T01:18:23.396370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:03.205988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.383024+0000","last_change":"2024-04-22T01:18:23.381847+0000","last_active":"2024-04-22T01:18:23.383024+0000","last_peered":"2024-04-22T01:18:23.383024+0000","last_clean":"2024-04-22T01:18:23.383024+0000","last_became_active":"2024-04-22T01:18:23.381553+0000","last_became_peered":"2024-04-22T01:18:23.381553+0000","last_unstale":"2024-04-22T01:18:23.383024+0000","last_undegraded":"2024-04-22T01:18:23.383024+0000","last_fullsized":"2024-04-22T01:18:23.383024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:08:47.007435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.348950+0000","last_change":"2024-04-22T01:18:22.352558+0000","last_active":"2024-04-22T01:18:23.348950+0000","last_peered":"2024-04-22T01:18:23.348950+0000","last_clean":"2024-04-22T01:18:23.348950+0000","last_became_active":"2024-04-22T01:18:22.351798+0000","last_became_peered":"2024-04-22T01:18:22.351798+0000","last_unstale":"2024-04-22T01:18:23.348950+0000","last_undegraded":"2024-04-22T01:18:23.348950+0000","last_fullsized":"2024-04-22T01:18:23.348950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:45.895154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.375615+0000","last_change":"2024-04-22T01:18:23.364528+0000","last_active":"2024-04-22T01:18:23.375615+0000","last_peered":"2024-04-22T01:18:23.375615+0000","last_clean":"2024-04-22T01:18:23.375615+0000","last_became_active":"2024-04-22T01:18:23.364226+0000","last_became_peered":"2024-04-22T01:18:23.364226+0000","last_unstale":"2024-04-22T01:18:23.375615+0000","last_undegraded":"2024-04-22T01:18:23.375615+0000","last_fullsized":"2024-04-22T01:18:23.375615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:31.685001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.349983+0000","last_change":"2024-04-22T01:18:22.358249+0000","last_active":"2024-04-22T01:18:23.349983+0000","last_peered":"2024-04-22T01:18:23.349983+0000","last_clean":"2024-04-22T01:18:23.349983+0000","last_became_active":"2024-04-22T01:18:22.357823+0000","last_became_peered":"2024-04-22T01:18:22.357823+0000","last_unstale":"2024-04-22T01:18:23.349983+0000","last_undegraded":"2024-04-22T01:18:23.349983+0000","last_fullsized":"2024-04-22T01:18:23.349983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:35:20.057528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.392344+0000","last_change":"2024-04-22T01:18:23.387863+0000","last_active":"2024-04-22T01:18:23.392344+0000","last_peered":"2024-04-22T01:18:23.392344+0000","last_clean":"2024-04-22T01:18:23.392344+0000","last_became_active":"2024-04-22T01:18:23.385568+0000","last_became_peered":"2024-04-22T01:18:23.385568+0000","last_unstale":"2024-04-22T01:18:23.392344+0000","last_undegraded":"2024-04-22T01:18:23.392344+0000","last_fullsized":"2024-04-22T01:18:23.392344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:33.821768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.354442+0000","last_change":"2024-04-22T01:18:22.341262+0000","last_active":"2024-04-22T01:18:23.354442+0000","last_peered":"2024-04-22T01:18:23.354442+0000","last_clean":"2024-04-22T01:18:23.354442+0000","last_became_active":"2024-04-22T01:18:22.335746+0000","last_became_peered":"2024-04-22T01:18:22.335746+0000","last_unstale":"2024-04-22T01:18:23.354442+0000","last_undegraded":"2024-04-22T01:18:23.354442+0000","last_fullsized":"2024-04-22T01:18:23.354442+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:23.123599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.363640+0000","last_change":"2024-04-22T01:18:23.352744+0000","last_active":"2024-04-22T01:18:23.363640+0000","last_peered":"2024-04-22T01:18:23.363640+0000","last_clean":"2024-04-22T01:18:23.363640+0000","last_became_active":"2024-04-22T01:18:23.352506+0000","last_became_peered":"2024-04-22T01:18:23.352506+0000","last_unstale":"2024-04-22T01:18:23.363640+0000","last_undegraded":"2024-04-22T01:18:23.363640+0000","last_fullsized":"2024-04-22T01:18:23.363640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:22.215257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.352476+0000","last_change":"2024-04-22T01:18:22.358743+0000","last_active":"2024-04-22T01:18:23.352476+0000","last_peered":"2024-04-22T01:18:23.352476+0000","last_clean":"2024-04-22T01:18:23.352476+0000","last_became_active":"2024-04-22T01:18:22.358436+0000","last_became_peered":"2024-04-22T01:18:22.358436+0000","last_unstale":"2024-04-22T01:18:23.352476+0000","last_undegraded":"2024-04-22T01:18:23.352476+0000","last_fullsized":"2024-04-22T01:18:23.352476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:39.116917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.370916+0000","last_change":"2024-04-22T01:18:23.370040+0000","last_active":"2024-04-22T01:18:23.370916+0000","last_peered":"2024-04-22T01:18:23.370916+0000","last_clean":"2024-04-22T01:18:23.370916+0000","last_became_active":"2024-04-22T01:18:23.367507+0000","last_became_peered":"2024-04-22T01:18:23.367507+0000","last_unstale":"2024-04-22T01:18:23.370916+0000","last_undegraded":"2024-04-22T01:18:23.370916+0000","last_fullsized":"2024-04-22T01:18:23.370916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:17:37.722726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.347285+0000","last_change":"2024-04-22T01:18:22.327211+0000","last_active":"2024-04-22T01:18:23.347285+0000","last_peered":"2024-04-22T01:18:23.347285+0000","last_clean":"2024-04-22T01:18:23.347285+0000","last_became_active":"2024-04-22T01:18:22.326518+0000","last_became_peered":"2024-04-22T01:18:22.326518+0000","last_unstale":"2024-04-22T01:18:23.347285+0000","last_undegraded":"2024-04-22T01:18:23.347285+0000","last_fullsized":"2024-04-22T01:18:23.347285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:32.417337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.389210+0000","last_change":"2024-04-22T01:18:23.373450+0000","last_active":"2024-04-22T01:18:23.389210+0000","last_peered":"2024-04-22T01:18:23.389210+0000","last_clean":"2024-04-22T01:18:23.389210+0000","last_became_active":"2024-04-22T01:18:23.372607+0000","last_became_peered":"2024-04-22T01:18:23.372607+0000","last_unstale":"2024-04-22T01:18:23.389210+0000","last_undegraded":"2024-04-22T01:18:23.389210+0000","last_fullsized":"2024-04-22T01:18:23.389210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:06.348633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.357254+0000","last_change":"2024-04-22T01:18:22.336118+0000","last_active":"2024-04-22T01:18:23.357254+0000","last_peered":"2024-04-22T01:18:23.357254+0000","last_clean":"2024-04-22T01:18:23.357254+0000","last_became_active":"2024-04-22T01:18:22.334729+0000","last_became_peered":"2024-04-22T01:18:22.334729+0000","last_unstale":"2024-04-22T01:18:23.357254+0000","last_undegraded":"2024-04-22T01:18:23.357254+0000","last_fullsized":"2024-04-22T01:18:23.357254+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:34.785033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.373932+0000","last_change":"2024-04-22T01:18:23.359810+0000","last_active":"2024-04-22T01:18:23.373932+0000","last_peered":"2024-04-22T01:18:23.373932+0000","last_clean":"2024-04-22T01:18:23.373932+0000","last_became_active":"2024-04-22T01:18:23.358742+0000","last_became_peered":"2024-04-22T01:18:23.358742+0000","last_unstale":"2024-04-22T01:18:23.373932+0000","last_undegraded":"2024-04-22T01:18:23.373932+0000","last_fullsized":"2024-04-22T01:18:23.373932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:30.422069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.352770+0000","last_change":"2024-04-22T01:18:22.341349+0000","last_active":"2024-04-22T01:18:23.352770+0000","last_peered":"2024-04-22T01:18:23.352770+0000","last_clean":"2024-04-22T01:18:23.352770+0000","last_became_active":"2024-04-22T01:18:22.340497+0000","last_became_peered":"2024-04-22T01:18:22.340497+0000","last_unstale":"2024-04-22T01:18:23.352770+0000","last_undegraded":"2024-04-22T01:18:23.352770+0000","last_fullsized":"2024-04-22T01:18:23.352770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:07:05.665482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.396558+0000","last_change":"2024-04-22T01:18:23.390781+0000","last_active":"2024-04-22T01:18:23.396558+0000","last_peered":"2024-04-22T01:18:23.396558+0000","last_clean":"2024-04-22T01:18:23.396558+0000","last_became_active":"2024-04-22T01:18:23.384276+0000","last_became_peered":"2024-04-22T01:18:23.384276+0000","last_unstale":"2024-04-22T01:18:23.396558+0000","last_undegraded":"2024-04-22T01:18:23.396558+0000","last_fullsized":"2024-04-22T01:18:23.396558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:22.305799+0000","last_clean_scrub_stamp":"2024-04-22T01:18:22.305799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:31.685001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.360195+0000","last_change":"2024-04-22T01:18:22.340621+0000","last_active":"2024-04-22T01:18:23.360195+0000","last_peered":"2024-04-22T01:18:23.360195+0000","last_clean":"2024-04-22T01:18:23.360195+0000","last_became_active":"2024-04-22T01:18:22.340068+0000","last_became_peered":"2024-04-22T01:18:22.340068+0000","last_unstale":"2024-04-22T01:18:23.360195+0000","last_undegraded":"2024-04-22T01:18:23.360195+0000","last_fullsized":"2024-04-22T01:18:23.360195+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:21.293079+0000","last_clean_scrub_stamp":"2024-04-22T01:18:21.293079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:23.123599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":290,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T01:18:23.352908+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:18:23.352908+0000","last_peered":"2024-04-22T01:18:23.352908+0000","last_clean":"2024-04-22T01:18:23.352908+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:18:23.352908+0000","last_undegraded":"2024-04-22T01:18:23.352908+0000","last_fullsized":"2024-04-22T01:18:23.352908+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1667072,"data_stored":1079877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36396,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335444,"statfs":{"total":96636764160,"available":96599494656,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335440,"statfs":{"total":96636764160,"available":96599490560,"internally_reserved":0,"allocated":1085440,"data_stored":496676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335448,"statfs":{"total":96636764160,"available":96599498752,"internally_reserved":0,"allocated":1073152,"data_stored":489509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36972,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334868,"statfs":{"total":96636764160,"available":96598904832,"internally_reserved":0,"allocated":1667072,"data_stored":1079877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:18:26.759 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:18:26.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-22T01:18:26.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5f254640 1 Processor -- start 2024-04-22T01:18:26.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5f254640 1 -- start start 2024-04-22T01:18:26.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5f254640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58168b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5f254640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff5806d740 con 0x7eff5815e1e0 2024-04-22T01:18:26.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5f254640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff5806c980 con 0x7eff580b30e0 2024-04-22T01:18:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5f254640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7eff5806d2b0 con 0x7eff580b5f70 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5cff0640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7eff5815e1e0 0x7eff580d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44520/0 (socket says 172.21.15.70:44520) 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5cff0640 1 -- 172.21.15.70:0/1052399643 learned_addr learned my addr 172.21.15.70:0/1052399643 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5d7f1640 1 --2- 172.21.15.70:0/1052399643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58168b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff577fe640 1 -- 172.21.15.70:0/1052399643 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1296836114 0 0) 0x7eff5806d740 con 0x7eff5815e1e0 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff577fe640 1 -- 172.21.15.70:0/1052399643 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff40003530 con 0x7eff5815e1e0 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5d7f1640 1 -- 172.21.15.70:0/1052399643 >> v1:172.21.15.98:6789/0 conn(0x7eff580b30e0 legacy=0x7eff580b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5d7f1640 1 -- 172.21.15.70:0/1052399643 >> v1:172.21.15.70:6789/0 conn(0x7eff5815e1e0 legacy=0x7eff580d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff577fe640 1 -- 172.21.15.70:0/1052399643 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3753492303 0 0) 0x7eff40003530 con 0x7eff5815e1e0 2024-04-22T01:18:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.928+0000 7eff5d7f1640 1 -- 172.21.15.70:0/1052399643 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5806cc60 con 0x7eff580b5f70 2024-04-22T01:18:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5d7f1640 1 --2- 172.21.15.70:0/1052399643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58168b90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7eff50009dc0 tx=0x7eff5001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba5cc11155bb8a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff577fe640 1 -- 172.21.15.70:0/1052399643 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48001680 con 0x7eff580b5f70 2024-04-22T01:18:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff577fe640 1 -- 172.21.15.70:0/1052399643 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff500a6500 con 0x7eff580b5f70 2024-04-22T01:18:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff577fe640 1 -- 172.21.15.70:0/1052399643 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff500a68b0 con 0x7eff580b5f70 2024-04-22T01:18:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 -- 172.21.15.70:0/1052399643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 msgr2=0x7eff58168b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 --2- 172.21.15.70:0/1052399643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58168b90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7eff50009dc0 tx=0x7eff5001c8b0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 -- 172.21.15.70:0/1052399643 shutdown_connections 2024-04-22T01:18:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 --2- 172.21.15.70:0/1052399643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58168b90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 -- 172.21.15.70:0/1052399643 >> 172.21.15.70:0/1052399643 conn(0x7eff58073110 msgr2=0x7eff580d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 -- 172.21.15.70:0/1052399643 shutdown_connections 2024-04-22T01:18:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 -- 172.21.15.70:0/1052399643 wait complete. 2024-04-22T01:18:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 Processor -- start 2024-04-22T01:18:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.932+0000 7eff5f254640 1 -- start start 2024-04-22T01:18:26.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 0x7eff580ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5815e1e0 0x7eff580c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7eff5816bc80 con 0x7eff580b30e0 2024-04-22T01:18:26.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff5816bb00 con 0x7eff5815e1e0 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7eff5816be00 con 0x7eff580b5f70 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 0x7eff580ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 0x7eff580ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41802/0 (socket says 172.21.15.70:41802) 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 -- 172.21.15.70:0/1703643905 learned_addr learned my addr 172.21.15.70:0/1703643905 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff57fff640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5d7f1640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5815e1e0 0x7eff580c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 -- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 msgr2=0x7eff58155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 -- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5815e1e0 msgr2=0x7eff580c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5815e1e0 0x7eff580c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 -- 172.21.15.70:0/1703643905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff580c8160 con 0x7eff580b30e0 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5cff0640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 0x7eff580ca7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c760 tx=0x7eff4800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48017070 con 0x7eff580b30e0 2024-04-22T01:18:26.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff48005b00 con 0x7eff580b30e0 2024-04-22T01:18:26.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff581586e0 con 0x7eff580b30e0 2024-04-22T01:18:26.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48007070 con 0x7eff580b30e0 2024-04-22T01:18:26.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff58158ca0 con 0x7eff580b30e0 2024-04-22T01:18:26.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff58068210 con 0x7eff580b30e0 2024-04-22T01:18:26.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff4801a070 con 0x7eff580b30e0 2024-04-22T01:18:26.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff557fa640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7eff440678c0 0x7eff44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:26.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7eff48085380 con 0x7eff580b30e0 2024-04-22T01:18:26.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff57fff640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7eff440678c0 0x7eff44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:26.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.936+0000 7eff57fff640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7eff440678c0 0x7eff44069d80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7eff58159ab0 tx=0x7eff4c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:26.940+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7eff48055d30 con 0x7eff580b30e0 2024-04-22T01:18:27.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.156+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7eff580b6350 con 0x7eff580b30e0 2024-04-22T01:18:27.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.156+0000 7eff557fa640 1 -- 172.21.15.70:0/1703643905 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7eff48055630 con 0x7eff580b30e0 2024-04-22T01:18:27.161 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:18:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:18:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-22T01:18:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-22T01:18:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-22T01:18:27.161 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-22T01:18:27.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7eff440678c0 msgr2=0x7eff44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7eff440678c0 0x7eff44069d80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7eff58159ab0 tx=0x7eff4c009040 comp rx=0 tx=0).stop 2024-04-22T01:18:27.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 msgr2=0x7eff580ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 0x7eff580ca7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c760 tx=0x7eff4800cc20 comp rx=0 tx=0).stop 2024-04-22T01:18:27.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 shutdown_connections 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7eff440678c0 0x7eff44069d80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5815e1e0 0x7eff580c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7eff580b5f70 0x7eff58155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 --2- 172.21.15.70:0/1703643905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff580b30e0 0x7eff580ca7a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 >> 172.21.15.70:0/1703643905 conn(0x7eff58073110 msgr2=0x7eff580b8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 shutdown_connections 2024-04-22T01:18:27.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.160+0000 7eff5f254640 1 -- 172.21.15.70:0/1703643905 wait complete. 2024-04-22T01:18:27.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:27.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 Processor -- start 2024-04-22T01:18:27.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 -- start start 2024-04-22T01:18:27.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f982c1455b0 0x7f982c0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:27.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f982c072230 0x7f982c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:27.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f982c0d0250 con 0x7f982c0cc150 2024-04-22T01:18:27.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f982c0cd7b0 con 0x7f982c072230 2024-04-22T01:18:27.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f9830dcb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f982c0cc710 con 0x7f982c1455b0 2024-04-22T01:18:27.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f982c072230 0x7f982c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f982c072230 0x7f982c06e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41806/0 (socket says 172.21.15.70:41806) 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 -- 172.21.15.70:0/352122583 learned_addr learned my addr 172.21.15.70:0/352122583 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982a59c640 1 --2- 172.21.15.70:0/352122583 >> v2:172.21.15.98:3300/0 conn(0x7f982c1455b0 0x7f982c0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 -- 172.21.15.70:0/352122583 >> v2:172.21.15.98:3300/0 conn(0x7f982c1455b0 msgr2=0x7f982c0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 --2- 172.21.15.70:0/352122583 >> v2:172.21.15.98:3300/0 conn(0x7f982c1455b0 0x7f982c0cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 -- 172.21.15.70:0/352122583 >> v1:172.21.15.98:6789/0 conn(0x7f982c0cc150 legacy=0x7f982c071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.292+0000 7f982ad9d640 1 -- 172.21.15.70:0/352122583 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c0cc9f0 con 0x7f982c072230 2024-04-22T01:18:27.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f982ad9d640 1 --2- 172.21.15.70:0/352122583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c06e200 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f982c0d0700 tx=0x7f982401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd9f75aaa96782c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:27.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f982959a640 1 -- 172.21.15.70:0/352122583 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240a5590 con 0x7f982c072230 2024-04-22T01:18:27.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f982959a640 1 -- 172.21.15.70:0/352122583 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98240a5730 con 0x7f982c072230 2024-04-22T01:18:27.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f982959a640 1 -- 172.21.15.70:0/352122583 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240a5b00 con 0x7f982c072230 2024-04-22T01:18:27.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 -- 172.21.15.70:0/352122583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 msgr2=0x7f982c06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 --2- 172.21.15.70:0/352122583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c06e200 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f982c0d0700 tx=0x7f982401c8d0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 -- 172.21.15.70:0/352122583 shutdown_connections 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 --2- 172.21.15.70:0/352122583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c06e200 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 --2- 172.21.15.70:0/352122583 >> v2:172.21.15.98:3300/0 conn(0x7f982c1455b0 0x7f982c0cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 -- 172.21.15.70:0/352122583 >> 172.21.15.70:0/352122583 conn(0x7f982c06b1f0 msgr2=0x7f982c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 -- 172.21.15.70:0/352122583 shutdown_connections 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 -- 172.21.15.70:0/352122583 wait complete. 2024-04-22T01:18:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.296+0000 7f9830dcb640 1 Processor -- start 2024-04-22T01:18:27.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- start start 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c13d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f982c0cc150 0x7f982c158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f982c1455b0 0x7f982c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f982c06f2e0 con 0x7f982c072230 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f982c06f160 con 0x7f982c0cc150 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f982c168b40 con 0x7f982c1455b0 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c13d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41818/0 (socket says 172.21.15.70:41818) 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 -- 172.21.15.70:0/2105483444 learned_addr learned my addr 172.21.15.70:0/2105483444 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982ad9d640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f982c1455b0 0x7f982c159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 -- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f982c1455b0 msgr2=0x7f982c159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f982c1455b0 0x7f982c159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 -- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f982c0cc150 msgr2=0x7f982c158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f982c0cc150 0x7f982c158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 -- 172.21.15.70:0/2105483444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c15a070 con 0x7f982c072230 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f982a59c640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c13d560 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f981c00d800 tx=0x7f981c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c00b6c0 con 0x7f982c072230 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f981c019070 con 0x7f982c072230 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c00ba80 con 0x7f982c072230 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982c15a2f0 con 0x7f982c072230 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982c13f470 con 0x7f982c072230 2024-04-22T01:18:27.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982c068210 con 0x7f982c072230 2024-04-22T01:18:27.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f981c00bc20 con 0x7f982c072230 2024-04-22T01:18:27.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f981affd640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f980c0678c0 0x7f980c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:27.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7f981c0862f0 con 0x7f982c072230 2024-04-22T01:18:27.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.300+0000 7f9829d9b640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f980c0678c0 0x7f980c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:27.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.304+0000 7f9829d9b640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f980c0678c0 0x7f980c069d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9820000f70 tx=0x7f98200079e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:27.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.304+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f982c15a070 con 0x7f982c072230 2024-04-22T01:18:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.500+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f982c15f510 con 0x7f982c072230 2024-04-22T01:18:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.500+0000 7f981affd640 1 -- 172.21.15.70:0/2105483444 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7f981c056ca0 con 0x7f982c072230 2024-04-22T01:18:27.503 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:27.503 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:22.684405+0000","modified":"2024-04-22T01:18:24.723325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7215":{"gid":7215,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2337533615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2337533615},{"type":"v1","addr":"172.21.15.70:6837","nonce":2337533615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7215,"qdb_cluster":[7215]},"id":6}]} 2024-04-22T01:18:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f980c0678c0 msgr2=0x7f980c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f980c0678c0 0x7f980c069d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9820000f70 tx=0x7f98200079e0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 msgr2=0x7f982c13d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c13d560 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f981c00d800 tx=0x7f981c00dcc0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 shutdown_connections 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f980c0678c0 0x7f980c069d80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f982c1455b0 0x7f982c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f982c0cc150 0x7f982c158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 --2- 172.21.15.70:0/2105483444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f982c072230 0x7f982c13d560 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 >> 172.21.15.70:0/2105483444 conn(0x7f982c06b1f0 msgr2=0x7f982c13c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 shutdown_connections 2024-04-22T01:18:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:27.504+0000 7f9830dcb640 1 -- 172.21.15.70:0/2105483444 wait complete. 2024-04-22T01:18:27.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:18:22.684405+0000', 'modified': '2024-04-22T01:18:24.723325+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7215}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7215': {'gid': 7215, 'name': 'a', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/2337533615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 2337533615}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 2337533615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7215, 'qdb_cluster': [7215]} 2024-04-22T01:18:27.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:18:27.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-22T01:18:28.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 Processor -- start 2024-04-22T01:18:28.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 -- start start 2024-04-22T01:18:28.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdaa81605e0 0x7fdaa806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa80cd7a0 con 0x7fdaa815bb50 2024-04-22T01:18:28.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa80cd580 con 0x7fdaa81605e0 2024-04-22T01:18:28.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaacaa5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa815f8b0 con 0x7fdaa815bf30 2024-04-22T01:18:28.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa6d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdaa81605e0 0x7fdaa806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdaa815bb50 0x7fdaa80cce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:53462/0 (socket says 172.21.15.70:53462) 2024-04-22T01:18:28.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa659c640 1 -- 172.21.15.70:0/705653384 learned_addr learned my addr 172.21.15.70:0/705653384 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:28.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 msgr2=0x7fdaa806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa806ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/705653384 >> v1:172.21.15.98:6789/0 conn(0x7fdaa815bb50 legacy=0x7fdaa80cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/705653384 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa815fb00 con 0x7fdaa81605e0 2024-04-22T01:18:28.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.056+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:18:28.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa81605e0 0x7fdaa806f2b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fda98009d90 tx=0x7fda9801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9161cf1d1434e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaa559a640 1 -- 172.21.15.70:0/705653384 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980b2070 con 0x7fdaa81605e0 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaa559a640 1 -- 172.21.15.70:0/705653384 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda980a5940 con 0x7fdaa81605e0 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaa559a640 1 -- 172.21.15.70:0/705653384 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980a5d10 con 0x7fdaa81605e0 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 -- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa81605e0 msgr2=0x7fdaa806f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa81605e0 0x7fdaa806f2b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fda98009d90 tx=0x7fda9801c970 comp rx=0 tx=0).stop 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 -- 172.21.15.70:0/705653384 shutdown_connections 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa81605e0 0x7fdaa806f2b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/705653384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 -- 172.21.15.70:0/705653384 >> 172.21.15.70:0/705653384 conn(0x7fdaa80cf9e0 msgr2=0x7fdaa80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 -- 172.21.15.70:0/705653384 shutdown_connections 2024-04-22T01:18:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 -- 172.21.15.70:0/705653384 wait complete. 2024-04-22T01:18:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 Processor -- start 2024-04-22T01:18:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.060+0000 7fdaacaa5640 1 -- start start 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa815bb50 0x7fdaa81498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa81446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 0x7fdaa81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa80726d0 con 0x7fdaa81605e0 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8072550 con 0x7fdaa815bb50 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa8072850 con 0x7fdaa815bf30 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 0x7fdaa81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 0x7fdaa81558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41824/0 (socket says 172.21.15.70:41824) 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/1664058471 learned_addr learned my addr 172.21.15.70:0/1664058471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa659c640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa815bb50 0x7fdaa81498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa81446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 msgr2=0x7fdaa81446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa81446d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa815bb50 msgr2=0x7fdaa81498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa815bb50 0x7fdaa81498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/1664058471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa8146170 con 0x7fdaa81605e0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 0x7fdaa81558f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fda980b0040 tx=0x7fda98019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fda96ffd640 1 -- 172.21.15.70:0/1664058471 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980b2050 con 0x7fdaa81605e0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fda96ffd640 1 -- 172.21.15.70:0/1664058471 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda980be070 con 0x7fdaa81605e0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa8146450 con 0x7fdaa81605e0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fda96ffd640 1 -- 172.21.15.70:0/1664058471 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980adcc0 con 0x7fdaa81605e0 2024-04-22T01:18:28.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa8146a10 con 0x7fdaa81605e0 2024-04-22T01:18:28.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fda96ffd640 1 -- 172.21.15.70:0/1664058471 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fda980c3460 con 0x7fdaa81605e0 2024-04-22T01:18:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fda96ffd640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda880678c0 0x7fda88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fda96ffd640 1 -- 172.21.15.70:0/1664058471 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fda98121110 con 0x7fdaa81605e0 2024-04-22T01:18:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.064+0000 7fdaa659c640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda880678c0 0x7fda88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa659c640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda880678c0 0x7fda88069d80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdaa81549b0 tx=0x7fdaa0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 Processor -- start 2024-04-22T01:18:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- start start 2024-04-22T01:18:28.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb2d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deae3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deae3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5613de9315e0 con 0x5613deb2d810 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613de8b8c50 con 0x5613deae3bc0 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5613de92d5a0 con 0x5613deb2cef0 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deae3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deae3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41836/0 (socket says 172.21.15.70:41836) 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/2902348280 learned_addr learned my addr 172.21.15.70:0/2902348280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deae3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa659c640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb2d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 msgr2=0x5613deb2d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb2d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 msgr2=0x5613deae3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deae3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/2902348280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613de8b6d10 con 0x5613deb2d810 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deae3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa659c640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb2d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deae3680 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009b00 tx=0x7fda9c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cc1400473b769c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fda87fff640 1 -- 172.21.15.70:0/2902348280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c019070 con 0x5613deb2d810 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fda87fff640 1 -- 172.21.15.70:0/2902348280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c011c20 con 0x5613deb2d810 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fda87fff640 1 -- 172.21.15.70:0/2902348280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c015460 con 0x5613deb2d810 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2902348280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 msgr2=0x5613deae3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deae3680 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009b00 tx=0x7fda9c009190 comp rx=0 tx=0).stop 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2902348280 shutdown_connections 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deae3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deae3680 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2902348280 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb2d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2902348280 >> 172.21.15.70:0/2902348280 conn(0x5613deadfb70 msgr2=0x5613deae1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2902348280 shutdown_connections 2024-04-22T01:18:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2902348280 wait complete. 2024-04-22T01:18:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 -- start start 2024-04-22T01:18:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 Processor -- start 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.068+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 0x5613deb11f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb12460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2d810 0x5613deb129a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5613de8eb520 con 0x5613deae3bc0 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613de8d7810 con 0x5613deb2d810 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5613deadf9f0 con 0x5613deb2cef0 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 10 client.0 _ll_register_callbacks cb 0x5613deace5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:18:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 0x5613deb11f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 0x5613deb11f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41848/0 (socket says 172.21.15.70:41848) 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 -- 172.21.15.70:0/2174310102 learned_addr learned my addr 172.21.15.70:0/2174310102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb12460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2d810 0x5613deb129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 -- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 msgr2=0x5613deb12460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb12460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 -- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2d810 msgr2=0x5613deb129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2d810 0x5613deb129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 -- 172.21.15.70:0/2174310102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613de8c97b0 con 0x5613deae3bc0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa659c640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 0x5613deb11f20 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00071a0 tx=0x7fdaa0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d468a4caf2ad7f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 1 -- 172.21.15.70:0/2174310102 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0007720 con 0x5613deae3bc0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 1 -- 172.21.15.70:0/2174310102 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fdaa0021c10 con 0x5613deae3bc0 2024-04-22T01:18:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda9c011c20 con 0x5613deae3bc0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 1 -- 172.21.15.70:0/2174310102 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdaa0021e10 con 0x5613deae3bc0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 10 client.7359 fetch_fsmap learned FSMap version 44 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x5613de8b8ea0 con 0x5613deae3bc0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 1 -- 172.21.15.70:0/2174310102 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c011c20 con 0x5613deae3bc0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 1 -- 172.21.15.70:0/2174310102 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 3494+0+0 (secure 0 0 0) 0x7fdaa000ec40 con 0x5613deae3bc0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 10 client.7359 fetch_fsmap finished waiting for FSMap version 44 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 10 client.7359 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 20 client.7359 populate_metadata read hostname 'smithi070' 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb3cf90 0x5613deb3f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 4 client.7359 mds_command: new command op to 7215 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 --> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613de8ccab0 con 0x5613deb3cf90 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb3cf90 0x5613deb3f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb3cf90 0x5613deb3f380 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bea067a32f69707d server_cookie=ac2b5427f099895b in_seq=0 out_seq=0 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 10 client.7359 ms_handle_connect on v2:172.21.15.70:6836/2337533615 2024-04-22T01:18:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 1 -- 172.21.15.70:0/2174310102 <== mds.0 v2:172.21.15.70:6836/2337533615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613de8ccab0 con 0x5613deb3cf90 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 10 client.7359 handle_command_reply: tid=0 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 client.7359 shutdown 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 msgr2=0x5613deb11f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 0x5613deb11f20 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00071a0 tx=0x7fdaa0000bd0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 shutdown_connections 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb3cf90 0x5613deb3f380 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2d810 0x5613deb129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2cef0 0x5613deb12460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fda857fa640 0 client.7359 ms_handle_reset on v2:172.21.15.70:6836/2337533615 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/2174310102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deae3bc0 0x5613deb11f20 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 >> 172.21.15.70:0/2174310102 conn(0x5613deadfb70 msgr2=0x5613deae27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 shutdown_connections 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 1 -- 172.21.15.70:0/2174310102 wait complete. 2024-04-22T01:18:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.072+0000 7fdaafc1e000 20 client.7359 trim_cache size 0 max 16384 2024-04-22T01:18:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 Processor -- start 2024-04-22T01:18:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- start start 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deae3bc0 0x5613deb3d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2cef0 0x5613deb3d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deb3ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81ff5b0 con 0x5613deb2d810 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81f7360 con 0x5613deb2cef0 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa81f66c0 con 0x5613deae3bc0 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deb3ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deb3ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41852/0 (socket says 172.21.15.70:41852) 2024-04-22T01:18:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/3521859502 learned_addr learned my addr 172.21.15.70:0/3521859502 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa659c640 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deae3bc0 0x5613deb3d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2cef0 0x5613deb3d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deae3bc0 msgr2=0x5613deb3d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deae3bc0 0x5613deb3d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2cef0 msgr2=0x5613deb3d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2cef0 0x5613deb3d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 -- 172.21.15.70:0/3521859502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613de8b9e80 con 0x5613deb2d810 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deb3ddf0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fda980f1360 tx=0x7fda980f0fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4623ac1a8796ac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fda877fe640 1 -- 172.21.15.70:0/3521859502 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980f13d0 con 0x5613deb2d810 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fda877fe640 1 -- 172.21.15.70:0/3521859502 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda980b3040 con 0x5613deb2d810 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fda877fe640 1 -- 172.21.15.70:0/3521859502 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980f5780 con 0x5613deb2d810 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- 172.21.15.70:0/3521859502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 msgr2=0x5613deb3ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deb3ddf0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fda980f1360 tx=0x7fda980f0fd0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- 172.21.15.70:0/3521859502 shutdown_connections 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2d810 0x5613deb3ddf0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deb2cef0 0x5613deb3d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/3521859502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deae3bc0 0x5613deb3d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- 172.21.15.70:0/3521859502 >> 172.21.15.70:0/3521859502 conn(0x5613deadfb70 msgr2=0x5613deb3a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- 172.21.15.70:0/3521859502 shutdown_connections 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- 172.21.15.70:0/3521859502 wait complete. 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 -- start start 2024-04-22T01:18:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.092+0000 7fdaafc1e000 1 Processor -- start 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deb9cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 0x5613deb9d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2d810 0x5613deb99510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5613de9e3e80 con 0x5613deb2cef0 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81f6d10 con 0x5613deae3bc0 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5613de8e5c40 con 0x5613deb2d810 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 10 client.0 _ll_register_callbacks cb 0x7fdaa0001310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 0x5613deb9d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 0x5613deb9d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41864/0 (socket says 172.21.15.70:41864) 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/1513905965 learned_addr learned my addr 172.21.15.70:0/1513905965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa659c640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deb9cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa6d9d640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2d810 0x5613deb99510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2d810 msgr2=0x5613deb99510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2d810 0x5613deb99510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 msgr2=0x5613deb9cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deb9cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 -- 172.21.15.70:0/1513905965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613de8fe760 con 0x5613deb2cef0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa5d9b640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 0x5613deb9d490 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009a20 tx=0x7fda9c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3fe9d95d27aceb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 1 -- 172.21.15.70:0/1513905965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c019040 con 0x5613deb2cef0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 1 -- 172.21.15.70:0/1513905965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fda9c004230 con 0x5613deb2cef0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda980b3040 con 0x5613deb2cef0 2024-04-22T01:18:28.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 1 -- 172.21.15.70:0/1513905965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fda9c033050 con 0x5613deb2cef0 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 10 client.7365 fetch_fsmap learned FSMap version 44 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x5613de98a9e0 con 0x5613deb2cef0 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 1 -- 172.21.15.70:0/1513905965 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda980b3040 con 0x5613deb2cef0 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 1 -- 172.21.15.70:0/1513905965 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 3494+0+0 (secure 0 0 0) 0x7fda9c036020 con 0x5613deb2cef0 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 10 client.7365 fetch_fsmap finished waiting for FSMap version 44 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 10 client.7365 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 20 client.7365 populate_metadata read hostname 'smithi070' 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb99a50 0x5613deb99e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 4 client.7365 mds_command: new command op to 7215 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 --> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] -- command(tid 0: {"prefix": "status"}) -- 0x5613de93eb20 con 0x5613deb99a50 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa659c640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb99a50 0x5613deb99e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 10 client.7365 ms_handle_connect on v2:172.21.15.70:6836/2337533615 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaa659c640 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb99a50 0x5613deb99e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a07a03e6d36b71b server_cookie=7ad376dab6f4328a in_seq=0 out_seq=0 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 1 -- 172.21.15.70:0/1513905965 <== mds.0 v2:172.21.15.70:6836/2337533615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5613de93eb20 con 0x5613deb99a50 2024-04-22T01:18:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 10 client.7365 handle_command_reply: tid=0 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 client.7365 shutdown 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 msgr2=0x5613deb9d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 0x5613deb9d490 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009a20 tx=0x7fda9c004740 comp rx=0 tx=0).stop 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 shutdown_connections 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:6836/2337533615,v1:172.21.15.70:6837/2337533615] conn(0x5613deb99a50 0x5613deb99e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5613deb2d810 0x5613deb99510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613deb2cef0 0x5613deb9d490 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fda857fa640 0 client.7365 ms_handle_reset on v2:172.21.15.70:6836/2337533615 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 --2- 172.21.15.70:0/1513905965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613deae3bc0 0x5613deb9cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 >> 172.21.15.70:0/1513905965 conn(0x5613deadfb70 msgr2=0x5613deb2b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 shutdown_connections 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 1 -- 172.21.15.70:0/1513905965 wait complete. 2024-04-22T01:18:28.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaafc1e000 20 client.7365 trim_cache size 0 max 16384 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 7215, 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 4.762876092, 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 44, 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 89, 2024-04-22T01:18:28.106 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:18:28.107 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 9.8163563259999993 2024-04-22T01:18:28.107 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:18:28.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda880678c0 msgr2=0x7fda88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda880678c0 0x7fda88069d80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdaa81549b0 tx=0x7fdaa0009040 comp rx=0 tx=0).stop 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 msgr2=0x7fdaa81558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 0x7fdaa81558f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fda980b0040 tx=0x7fda98019710 comp rx=0 tx=0).stop 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 shutdown_connections 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fda880678c0 0x7fda88069d80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa81605e0 0x7fdaa81558f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdaa815bf30 0x7fdaa81446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 --2- 172.21.15.70:0/1664058471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaa815bb50 0x7fdaa81498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 >> 172.21.15.70:0/1664058471 conn(0x7fdaa80cf9e0 msgr2=0x7fdaa8152ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 shutdown_connections 2024-04-22T01:18:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.096+0000 7fdaacaa5640 1 -- 172.21.15.70:0/1664058471 wait complete. 2024-04-22T01:18:28.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:18:28.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 Processor -- start 2024-04-22T01:18:28.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 -- start start 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe43c15f3d0 0x7fe43c099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c09a200 0x7fe43c09a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe43c0c5630 0x7fe43c0c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe43c0d0ba0 con 0x7fe43c15f3d0 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe43c0cd190 con 0x7fe43c0c5630 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44407e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe43c09aba0 con 0x7fe43c09a200 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe43c15f3d0 0x7fe43c099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe43c15f3d0 0x7fe43c099cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41872/0 (socket says 172.21.15.70:41872) 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 -- 172.21.15.70:0/936346089 learned_addr learned my addr 172.21.15.70:0/936346089 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441619640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c09a200 0x7fe43c09a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe44261b640 1 --2- 172.21.15.70:0/936346089 >> v2:172.21.15.98:3300/0 conn(0x7fe43c0c5630 0x7fe43c0c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 -- 172.21.15.70:0/936346089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c09a200 msgr2=0x7fe43c09a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c09a200 0x7fe43c09a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 -- 172.21.15.70:0/936346089 >> v2:172.21.15.98:3300/0 conn(0x7fe43c0c5630 msgr2=0x7fe43c0c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 --2- 172.21.15.70:0/936346089 >> v2:172.21.15.98:3300/0 conn(0x7fe43c0c5630 0x7fe43c0c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441e1a640 1 -- 172.21.15.70:0/936346089 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43c15f7b0 con 0x7fe43c15f3d0 2024-04-22T01:18:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.216+0000 7fe441619640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c09a200 0x7fe43c09a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441e1a640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c15f3d0 0x7fe43c099cc0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0d1990 tx=0x7fe43801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1605e9ede46ac10a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe440e18640 1 -- 172.21.15.70:0/936346089 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4380b1070 con 0x7fe43c15f3d0 2024-04-22T01:18:28.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe440e18640 1 -- 172.21.15.70:0/936346089 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4380ac630 con 0x7fe43c15f3d0 2024-04-22T01:18:28.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe440e18640 1 -- 172.21.15.70:0/936346089 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4380a8050 con 0x7fe43c15f3d0 2024-04-22T01:18:28.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/936346089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c15f3d0 msgr2=0x7fe43c099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c15f3d0 0x7fe43c099cc0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0d1990 tx=0x7fe43801c520 comp rx=0 tx=0).stop 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/936346089 shutdown_connections 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- 172.21.15.70:0/936346089 >> v2:172.21.15.98:3300/0 conn(0x7fe43c0c5630 0x7fe43c0c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c09a200 0x7fe43c09a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- 172.21.15.70:0/936346089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c15f3d0 0x7fe43c099cc0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/936346089 >> 172.21.15.70:0/936346089 conn(0x7fe43c0b35d0 msgr2=0x7fe43c0b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/936346089 shutdown_connections 2024-04-22T01:18:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/936346089 wait complete. 2024-04-22T01:18:28.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 Processor -- start 2024-04-22T01:18:28.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- start start 2024-04-22T01:18:28.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43c09a200 0x7fe43c14c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 0x7fe43c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c15f3d0 0x7fe43c1549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe43c0b2b50 con 0x7fe43c0c5630 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe43c0cdc80 con 0x7fe43c09a200 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe43c0cf6b0 con 0x7fe43c15f3d0 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 0x7fe43c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 0x7fe43c1544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41884/0 (socket says 172.21.15.70:41884) 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 -- 172.21.15.70:0/949263088 learned_addr learned my addr 172.21.15.70:0/949263088 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44261b640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c15f3d0 0x7fe43c1549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441e1a640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43c09a200 0x7fe43c14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 -- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c15f3d0 msgr2=0x7fe43c1549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c15f3d0 0x7fe43c1549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 -- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43c09a200 msgr2=0x7fe43c14c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43c09a200 0x7fe43c14c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 -- 172.21.15.70:0/949263088 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43c14c770 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe441619640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 0x7fe43c1544b0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00e810 tx=0x7fe42c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c017070 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe42c00c8d0 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe43c0b6360 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c00f040 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.220+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43c0b6920 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.224+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe43c068920 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.224+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe42c01a070 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.224+0000 7fe4327fc640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe4200678c0 0x7fe420069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.224+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fe42c085a60 con 0x7fe43c0c5630 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.224+0000 7fe441e1a640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe4200678c0 0x7fe420069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:28.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.224+0000 7fe441e1a640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe4200678c0 0x7fe420069d80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ae000 tx=0x7fe438008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:28.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.228+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe42c056410 con 0x7fe43c0c5630 2024-04-22T01:18:28.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe43c074920 con 0x7fe43c0c5630 2024-04-22T01:18:28.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe4327fc640 1 -- 172.21.15.70:0/949263088 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7fe42c055d10 con 0x7fe43c0c5630 2024-04-22T01:18:28.417 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe4200678c0 msgr2=0x7fe420069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe4200678c0 0x7fe420069d80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ae000 tx=0x7fe438008040 comp rx=0 tx=0).stop 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 msgr2=0x7fe43c1544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 0x7fe43c1544b0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00e810 tx=0x7fe42c00ecd0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 shutdown_connections 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe4200678c0 0x7fe420069d80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe43c15f3d0 0x7fe43c1549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe43c0c5630 0x7fe43c1544b0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 --2- 172.21.15.70:0/949263088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43c09a200 0x7fe43c14c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:28.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 >> 172.21.15.70:0/949263088 conn(0x7fe43c0b35d0 msgr2=0x7fe43c0c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:28.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 shutdown_connections 2024-04-22T01:18:28.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28.416+0000 7fe44407e640 1 -- 172.21.15.70:0/949263088 wait complete. 2024-04-22T01:18:28.435 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:18:28.435 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:18:28.435 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:18:28.435 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:18:28.436 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:28.436 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:18:28.436 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:18:28.436 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:18:28.436 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:18:28.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:28.436 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:28.442 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42313sec preferred_lft 42313sec 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:18:28.443 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:18:28.444 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:28.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:28.445 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:18:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:18:28.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:28.498 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:28.550 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:28.550 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:18:28.550 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:18:28.550 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:18:28.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:18:28.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:18:28.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:18:28.550 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:18:28.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:18:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:28.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:28.669 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:18:28.718 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:18:28.718 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:18:28.718 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:18:28.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:28.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:18:28.773 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:18:28.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:28.774 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:28.821 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:18:28.821 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:28.822 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:28.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:18:28.926 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:28.976 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:18:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:18:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:28.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:28.980 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:18:29.025 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:18:29.025 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:18:29.070 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:18:29.084 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:29.133 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:18:29.133 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:18:29.134 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:18:29.135 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:29.136 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:29.136 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:29.136 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:29.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:29.137 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:18:29.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:18:29.148+0000 7f4705e9a580 -1 init, newargv = 0x560f10b305c0 newargc=15 2024-04-22T01:18:29.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[45163]: starting ceph client 2024-04-22T01:18:29.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[45163]: starting fuse 2024-04-22T01:18:29.297 INFO:tasks.ceph.mds.a.smithi070.stderr:2024-04-22T01:18:29.296+0000 7fe0e8cf2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:18:30.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:18:30.273 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:18:30.322 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:18:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:30.325 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:30.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:30.325 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:18:30.373 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:18:30.373 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:18:30.374 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:18:30.375 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:18:30.498 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.45163.asok 2024-04-22T01:18:30.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:30.509 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45163.asok 2024-04-22T01:18:30.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:30.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45163.asok status 2024-04-22T01:18:30.667 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:18:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:18:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:18:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "45163", 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 7380, 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 7380 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:18:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3338668060 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3338668060 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.7380 192.168.0.1:0/3338668060", 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3338668060", 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 44, 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 90, 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:18:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:18:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:18:30.670 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:18:30.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:30.682 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:18:30.682 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:30.688 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:18:30.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:30.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:30.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:30.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:18:30.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.900+0000 7fb008845640 1 Processor -- start 2024-04-22T01:18:30.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.900+0000 7fb008845640 1 -- start start 2024-04-22T01:18:30.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb008845640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb00415bb40 0x7fb004072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb008845640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb00406ed40 0x7fb0040b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb008845640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00415bf20 con 0x7fb0040b3b00 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb008845640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb0040b3f70 con 0x7fb00406ed40 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb008845640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb00406e8b0 con 0x7fb00415bb40 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb00406ed40 0x7fb0040b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb002d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb0040b3b00 0x7fb004161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44536/0 (socket says 172.21.15.70:44536) 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb002d9d640 1 -- 172.21.15.70:0/3825742297 learned_addr learned my addr 172.21.15.70:0/3825742297 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 --2- 172.21.15.70:0/3825742297 >> v2:172.21.15.70:3300/0 conn(0x7fb00406ed40 0x7fb0040b35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41900/0 (socket says 172.21.15.70:41900) 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb00259c640 1 --2- 172.21.15.70:0/3825742297 >> v2:172.21.15.98:3300/0 conn(0x7fb00415bb40 0x7fb004072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 -- 172.21.15.70:0/3825742297 >> v2:172.21.15.98:3300/0 conn(0x7fb00415bb40 msgr2=0x7fb004072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 --2- 172.21.15.70:0/3825742297 >> v2:172.21.15.98:3300/0 conn(0x7fb00415bb40 0x7fb004072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 -- 172.21.15.70:0/3825742297 >> v1:172.21.15.70:6789/0 conn(0x7fb0040b3b00 legacy=0x7fb004161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 -- 172.21.15.70:0/3825742297 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0040aec30 con 0x7fb00406ed40 2024-04-22T01:18:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb00159a640 1 -- 172.21.15.70:0/3825742297 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154316709 0 0) 0x7fb00415bf20 con 0x7fb0040b3b00 2024-04-22T01:18:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb001d9b640 1 --2- 172.21.15.70:0/3825742297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb00406ed40 0x7fb0040b35c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fb00406fce0 tx=0x7faff801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1741afb356100c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb00159a640 1 -- 172.21.15.70:0/3825742297 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80ab650 con 0x7fb00406ed40 2024-04-22T01:18:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb00159a640 1 -- 172.21.15.70:0/3825742297 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff80b0070 con 0x7fb00406ed40 2024-04-22T01:18:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.904+0000 7fb00159a640 1 -- 172.21.15.70:0/3825742297 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80aba10 con 0x7fb00406ed40 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3825742297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb00406ed40 msgr2=0x7fb0040b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 --2- 172.21.15.70:0/3825742297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb00406ed40 0x7fb0040b35c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fb00406fce0 tx=0x7faff801c450 comp rx=0 tx=0).stop 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3825742297 shutdown_connections 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 --2- 172.21.15.70:0/3825742297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb00406ed40 0x7fb0040b35c0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 --2- 172.21.15.70:0/3825742297 >> v2:172.21.15.98:3300/0 conn(0x7fb00415bb40 0x7fb004072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3825742297 >> 172.21.15.70:0/3825742297 conn(0x7fb00406b1f0 msgr2=0x7fb00406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3825742297 shutdown_connections 2024-04-22T01:18:30.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3825742297 wait complete. 2024-04-22T01:18:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 Processor -- start 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- start start 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb00406ed40 0x7fb004148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 0x7fb0040b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb00415bb40 0x7fb00413f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0040ae9d0 con 0x7fb0040b3b00 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0040ae850 con 0x7fb00406ed40 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb00406f5c0 con 0x7fb00415bb40 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 0x7fb0040b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 0x7fb0040b1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41914/0 (socket says 172.21.15.70:41914) 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 -- 172.21.15.70:0/3797995098 learned_addr learned my addr 172.21.15.70:0/3797995098 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb00259c640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb00406ed40 0x7fb004148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb002d9d640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb00415bb40 0x7fb00413f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 -- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb00415bb40 msgr2=0x7fb00413f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb00415bb40 0x7fb00413f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 -- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb00406ed40 msgr2=0x7fb004148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb00406ed40 0x7fb004148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 -- 172.21.15.70:0/3797995098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb004141800 con 0x7fb0040b3b00 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb00259c640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb00406ed40 0x7fb004148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb001d9b640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 0x7fb0040b1b90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7faff8019060 tx=0x7faff80191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80bf610 con 0x7fb0040b3b00 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff80bf7d0 con 0x7fb0040b3b00 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00412e0c0 con 0x7fb0040b3b00 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80bfbc0 con 0x7fb0040b3b00 2024-04-22T01:18:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00412e680 con 0x7fb0040b3b00 2024-04-22T01:18:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.908+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb004068210 con 0x7fb0040b3b00 2024-04-22T01:18:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.912+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faff80bd020 con 0x7fb0040b3b00 2024-04-22T01:18:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.912+0000 7faff2ffd640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafe40678c0 0x7fafe4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.912+0000 7fb00259c640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafe40678c0 0x7fafe4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.912+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7faff8120890 con 0x7fb0040b3b00 2024-04-22T01:18:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.912+0000 7fb00259c640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafe40678c0 0x7fafe4069d80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faff4006fd0 tx=0x7faff4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:30.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:30.916+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7faff8016070 con 0x7fb0040b3b00 2024-04-22T01:18:31.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb00406e040 con 0x7fb0040b3b00 2024-04-22T01:18:31.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7faff2ffd640 1 -- 172.21.15.70:0/3797995098 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7faff80f11d0 con 0x7fb0040b3b00 2024-04-22T01:18:31.115 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafe40678c0 msgr2=0x7fafe4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafe40678c0 0x7fafe4069d80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faff4006fd0 tx=0x7faff4008040 comp rx=0 tx=0).stop 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 msgr2=0x7fb0040b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 0x7fb0040b1b90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7faff8019060 tx=0x7faff80191f0 comp rx=0 tx=0).stop 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 shutdown_connections 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fafe40678c0 0x7fafe4069d80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb00415bb40 0x7fb00413f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0040b3b00 0x7fb0040b1b90 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 --2- 172.21.15.70:0/3797995098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb00406ed40 0x7fb004148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:31.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.112+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 >> 172.21.15.70:0/3797995098 conn(0x7fb00406b1f0 msgr2=0x7fb004072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:31.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.116+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 shutdown_connections 2024-04-22T01:18:31.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:31.116+0000 7fb008845640 1 -- 172.21.15.70:0/3797995098 wait complete. 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:18:31.132 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:18:31.133 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:18:31.133 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:18:31.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.133 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42315sec preferred_lft 42315sec 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:18:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:18:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:31.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.144 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:18:31.201 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:18:31.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.203 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:18:31.255 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:31.256 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:18:31.256 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:18:31.257 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:18:31.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:18:31.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:18:31.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:18:31.257 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:18:31.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:18:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:18:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:18:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:31.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.381 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:18:31.386 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:18:31.386 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:18:31.387 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:18:31.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:18:31.442 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:18:31.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:31.443 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:31.494 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:18:31.494 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:31.494 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:31.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:18:31.599 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:18:31.647 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:18:31.648 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:31.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.650 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:18:31.698 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:18:31.698 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:18:31.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:18:31.757 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:18:31.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:18:31.807 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:31.808 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:31.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:31.809 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:18:31.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:18:31.819+0000 7fda95038580 -1 init, newargv = 0x5651ffab7b00 newargc=15 2024-04-22T01:18:31.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34649]: starting ceph client 2024-04-22T01:18:31.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34649]: starting fuse 2024-04-22T01:18:32.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:18:32.942 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:32.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:32.994 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:18:33.042 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:18:33.042 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:18:33.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:18:33.042 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:18:33.042 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:18:33.043 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:18:33.044 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:18:33.164 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.34649.asok 2024-04-22T01:18:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:18:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:33.176 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34649.asok 2024-04-22T01:18:33.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34649.asok status 2024-04-22T01:18:33.327 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:18:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:18:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:18:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:18:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "34649", 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5477, 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:18:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5477 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2926744118 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2926744118 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5477 192.168.0.1:0/2926744118", 2024-04-22T01:18:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/2926744118", 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 44, 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 91, 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:18:33.330 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:18:33.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.341 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:18:33.341 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:33.347 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:18:33.347 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:33.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:33.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.401 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-22T01:18:33.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.418 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-22T01:18:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:18:33.473 INFO:teuthology.orchestra.run.smithi070.stderr:./subdir: ceph.quota.max_files: No such attribute 2024-04-22T01:18:33.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.474 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-22T01:18:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:./subdir: ceph.quota.max_files: No such attribute 2024-04-22T01:18:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:18:33.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.507 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 10 ./subdir) 2024-04-22T01:18:33.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.527 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-22T01:18:33.581 INFO:teuthology.orchestra.run.smithi070.stdout:10 2024-04-22T01:18:33.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:33.581 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-22T01:18:33.594 INFO:teuthology.orchestra.run.smithi098.stdout:10 2024-04-22T01:18:33.594 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:18:33.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:33.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:33.699 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:18:33.699 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:33.699 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:33.699 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:33.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:33.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.880+0000 7f5557fbd640 1 Processor -- start 2024-04-22T01:18:33.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.880+0000 7f5557fbd640 1 -- start start 2024-04-22T01:18:33.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5557fbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55500cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:33.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5557fbd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55500cc150 0x7f5550071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5557fbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55500d0250 con 0x7f5550072060 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5557fbd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f55500724d0 con 0x7f55500cc150 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5557fbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f55500cd7b0 con 0x7f55501455b0 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555558640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55500cc150 0x7f5550071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f555655a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5550072060 0x7f555015f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44542/0 (socket says 172.21.15.70:44542) 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f555655a640 1 -- 172.21.15.70:0/1523356262 learned_addr learned my addr 172.21.15.70:0/1523356262 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555d59640 1 --2- 172.21.15.70:0/1523356262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55500cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555558640 1 -- 172.21.15.70:0/1523356262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 msgr2=0x7f55500cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555558640 1 --2- 172.21.15.70:0/1523356262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55500cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555558640 1 -- 172.21.15.70:0/1523356262 >> v1:172.21.15.70:6789/0 conn(0x7f5550072060 legacy=0x7f555015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555558640 1 -- 172.21.15.70:0/1523356262 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55500cc710 con 0x7f55500cc150 2024-04-22T01:18:33.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5554d57640 1 -- 172.21.15.70:0/1523356262 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388062538 0 0) 0x7f55500d0250 con 0x7f5550072060 2024-04-22T01:18:33.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5555558640 1 --2- 172.21.15.70:0/1523356262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f5550071b20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f554c0097c0 tx=0x7f554c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c036f760b07d390c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:33.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5554d57640 1 -- 172.21.15.70:0/1523356262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c0b0070 con 0x7f55500cc150 2024-04-22T01:18:33.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5554d57640 1 -- 172.21.15.70:0/1523356262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f554c0ab600 con 0x7f55500cc150 2024-04-22T01:18:33.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.884+0000 7f5554d57640 1 -- 172.21.15.70:0/1523356262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c0ab9d0 con 0x7f55500cc150 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/1523356262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 msgr2=0x7f5550071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 --2- 172.21.15.70:0/1523356262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f5550071b20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f554c0097c0 tx=0x7f554c01c380 comp rx=0 tx=0).stop 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/1523356262 shutdown_connections 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 --2- 172.21.15.70:0/1523356262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f5550071b20 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 --2- 172.21.15.70:0/1523356262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55500cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/1523356262 >> 172.21.15.70:0/1523356262 conn(0x7f555006b1f0 msgr2=0x7f555006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/1523356262 shutdown_connections 2024-04-22T01:18:33.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/1523356262 wait complete. 2024-04-22T01:18:33.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 Processor -- start 2024-04-22T01:18:33.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- start start 2024-04-22T01:18:33.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550072060 0x7f5550164c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:33.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f55501424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:33.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55501593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5550168b40 con 0x7f55500cc150 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f555006f210 con 0x7f5550072060 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5550168cc0 con 0x7f55501455b0 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f55501424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f55501424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41926/0 (socket says 172.21.15.70:41926) 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 -- 172.21.15.70:0/3409896831 learned_addr learned my addr 172.21.15.70:0/3409896831 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 -- 172.21.15.70:0/3409896831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 msgr2=0x7f55501593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55501593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 -- 172.21.15.70:0/3409896831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550072060 msgr2=0x7f5550164c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550072060 0x7f5550164c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 -- 172.21.15.70:0/3409896831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5550165300 con 0x7f55500cc150 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5555558640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f55501424b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f554c0ae040 tx=0x7f554c019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c0b8030 con 0x7f55500cc150 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f554c0ab400 con 0x7f55500cc150 2024-04-22T01:18:33.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c0b0040 con 0x7f55500cc150 2024-04-22T01:18:33.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55501655e0 con 0x7f55500cc150 2024-04-22T01:18:33.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.888+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5550165ba0 con 0x7f55500cc150 2024-04-22T01:18:33.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.892+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5550068210 con 0x7f55500cc150 2024-04-22T01:18:33.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.892+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f554c0bb070 con 0x7f55500cc150 2024-04-22T01:18:33.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.892+0000 7f55467fc640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55340678c0 0x7f5534069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:33.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.892+0000 7f5555d59640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55340678c0 0x7f5534069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:33.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.892+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f554c11f660 con 0x7f55500cc150 2024-04-22T01:18:33.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.892+0000 7f5555d59640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55340678c0 0x7f5534069d80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f55400098f0 tx=0x7f5540008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:33.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:33.896+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f554c0c2070 con 0x7f55500cc150 2024-04-22T01:18:34.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.060+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f555015f510 con 0x7f55500cc150 2024-04-22T01:18:34.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f55467fc640 1 -- 172.21.15.70:0/3409896831 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+17042 (secure 0 0 0) 0x7f554c016090 con 0x7f55500cc150 2024-04-22T01:18:34.066 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:34.066 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:29.400245+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55340678c0 msgr2=0x7f5534069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55340678c0 0x7f5534069d80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f55400098f0 tx=0x7f5540008040 comp rx=0 tx=0).stop 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 msgr2=0x7f55501424b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f55501424b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f554c0ae040 tx=0x7f554c019120 comp rx=0 tx=0).stop 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 shutdown_connections 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f55340678c0 0x7f5534069d80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55501455b0 0x7f55501593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55500cc150 0x7f55501424b0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 --2- 172.21.15.70:0/3409896831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550072060 0x7f5550164c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 >> 172.21.15.70:0/3409896831 conn(0x7f555006b1f0 msgr2=0x7f555013f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:34.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 shutdown_connections 2024-04-22T01:18:34.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.064+0000 7f5557fbd640 1 -- 172.21.15.70:0/3409896831 wait complete. 2024-04-22T01:18:34.084 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:18:20.962202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:18:21.681268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:34.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:34.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 Processor -- start 2024-04-22T01:18:34.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 -- start start 2024-04-22T01:18:34.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7438072280 0x7f74380ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 0x7f74380b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743815c290 con 0x7f74380b4450 2024-04-22T01:18:34.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f743815c570 con 0x7f7438072280 2024-04-22T01:18:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743fbe4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f743815c850 con 0x7f74380cd050 2024-04-22T01:18:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743d980640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f74380b4450 0x7f7438071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44550/0 (socket says 172.21.15.70:44550) 2024-04-22T01:18:34.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743d980640 1 -- 172.21.15.70:0/34470766 learned_addr learned my addr 172.21.15.70:0/34470766 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:34.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743e181640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 0x7f74380b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743d17f640 1 --2- 172.21.15.70:0/34470766 >> v2:172.21.15.98:3300/0 conn(0x7f7438072280 0x7f74380ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971355276 0 0) 0x7f743815c290 con 0x7f74380b4450 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7424003530 con 0x7f74380b4450 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 810020795 0 0) 0x7f7424003530 con 0x7f74380b4450 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 msgr2=0x7f74380b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 0x7f74380b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 >> v2:172.21.15.98:3300/0 conn(0x7f7438072280 msgr2=0x7f74380ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 --2- 172.21.15.70:0/34470766 >> v2:172.21.15.98:3300/0 conn(0x7f7438072280 0x7f74380ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743815cb30 con 0x7f74380b4450 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f7434001680 con 0x7f74380b4450 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 >> v1:172.21.15.70:6789/0 conn(0x7f74380b4450 legacy=0x7f7438071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74240039c0 0x7f7424003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 0x7f7424004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424004d50 0x7f74240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.204+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f743815c570 con 0x7f74240039c0 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f743815c850 con 0x7f7424004d50 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f742400b9d0 con 0x7f74380cd050 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74240039c0 0x7f7424003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743e181640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424004d50 0x7f74240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d17f640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 0x7f7424004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 -- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 msgr2=0x7f7424004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74380cd050 0x7f7424004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 -- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424004d50 msgr2=0x7f74240051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424004d50 0x7f74240051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 -- 172.21.15.70:0/34470766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7424003530 con 0x7f74240039c0 2024-04-22T01:18:34.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743d980640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74240039c0 0x7f7424003de0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f743400c240 tx=0x7f743401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d0949af8595a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434007d40 con 0x7f74240039c0 2024-04-22T01:18:34.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74340a9020 con 0x7f74240039c0 2024-04-22T01:18:34.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743c97e640 1 -- 172.21.15.70:0/34470766 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434005b00 con 0x7f74240039c0 2024-04-22T01:18:34.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743fbe4640 1 -- 172.21.15.70:0/34470766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74240039c0 msgr2=0x7f7424003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743fbe4640 1 --2- 172.21.15.70:0/34470766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74240039c0 0x7f7424003de0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f743400c240 tx=0x7f743401cca0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743e181640 1 -- 172.21.15.70:0/34470766 reap_dead start 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743fbe4640 1 -- 172.21.15.70:0/34470766 shutdown_connections 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743fbe4640 1 -- 172.21.15.70:0/34470766 >> 172.21.15.70:0/34470766 conn(0x7f74380b1d40 msgr2=0x7f74380cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743fbe4640 1 -- 172.21.15.70:0/34470766 shutdown_connections 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.208+0000 7f743fbe4640 1 -- 172.21.15.70:0/34470766 wait complete. 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 Processor -- start 2024-04-22T01:18:34.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- start start 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74240039c0 0x7f74381454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743815b550 0x7f74380a8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 0x7f74380a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74380cd980 con 0x7f743815b9b0 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74380cd800 con 0x7f743815b550 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f74380b3250 con 0x7f74240039c0 2024-04-22T01:18:34.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 0x7f74380a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 0x7f74380a9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41940/0 (socket says 172.21.15.70:41940) 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 -- 172.21.15.70:0/3589014449 learned_addr learned my addr 172.21.15.70:0/3589014449 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743d980640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74240039c0 0x7f74381454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743d17f640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743815b550 0x7f74380a8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 -- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74240039c0 msgr2=0x7f74381454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74240039c0 0x7f74381454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 -- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743815b550 msgr2=0x7f74380a8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743815b550 0x7f74380a8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 -- 172.21.15.70:0/3589014449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743815a4b0 con 0x7f743815b9b0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743e181640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 0x7f74380a9310 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f743815f6b0 tx=0x7f74300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7430011e40 con 0x7f743815b9b0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743001b070 con 0x7f743815b9b0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743815a790 con 0x7f743815b9b0 2024-04-22T01:18:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74300039a0 con 0x7f743815b9b0 2024-04-22T01:18:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743815ad50 con 0x7f743815b9b0 2024-04-22T01:18:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.212+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7438068210 con 0x7f743815b9b0 2024-04-22T01:18:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.216+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7430003b40 con 0x7f743815b9b0 2024-04-22T01:18:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.216+0000 7f742e7fc640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f741c0678c0 0x7f741c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.216+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f7430086620 con 0x7f743815b9b0 2024-04-22T01:18:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.216+0000 7f743d980640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f741c0678c0 0x7f741c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.216+0000 7f743d980640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f741c0678c0 0x7f741c069d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f74340aa000 tx=0x7f743815c180 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.220+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7430056f60 con 0x7f743815b9b0 2024-04-22T01:18:34.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74380b3fd0 con 0x7f743815b9b0 2024-04-22T01:18:34.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f742e7fc640 1 -- 172.21.15.70:0/3589014449 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f7430056860 con 0x7f743815b9b0 2024-04-22T01:18:34.382 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f741c0678c0 msgr2=0x7f741c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f743fbe4640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f741c0678c0 0x7f741c069d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f74340aa000 tx=0x7f743815c180 comp rx=0 tx=0).stop 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 msgr2=0x7f74380a9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f743fbe4640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 0x7f74380a9310 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f743815f6b0 tx=0x7f74300115a0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.380+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 shutdown_connections 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f741c0678c0 0x7f741c069d80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f743815b9b0 0x7f74380a9310 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743815b550 0x7f74380a8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 --2- 172.21.15.70:0/3589014449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74240039c0 0x7f74381454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 >> 172.21.15.70:0/3589014449 conn(0x7f74380b1d40 msgr2=0x7f74380b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 shutdown_connections 2024-04-22T01:18:34.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.384+0000 7f743fbe4640 1 -- 172.21.15.70:0/3589014449 wait complete. 2024-04-22T01:18:34.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:18:34.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 Processor -- start 2024-04-22T01:18:34.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 -- start start 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e015a320 0x7fa8e006f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8e0071b70 0x7fa8e0071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8e00b3d60 0x7fa8e00b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e0072510 con 0x7fa8e0071b70 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e006da70 con 0x7fa8e00b3d60 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e838f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e006dd50 con 0x7fa8e015a320 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8e0071b70 0x7fa8e0071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8e0071b70 0x7fa8e0071fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41958/0 (socket says 172.21.15.70:41958) 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 -- 172.21.15.70:0/2765653512 learned_addr learned my addr 172.21.15.70:0/2765653512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e612b640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e015a320 0x7fa8e006f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e692c640 1 --2- 172.21.15.70:0/2765653512 >> v2:172.21.15.98:3300/0 conn(0x7fa8e00b3d60 0x7fa8e00b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 -- 172.21.15.70:0/2765653512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e015a320 msgr2=0x7fa8e006f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e015a320 0x7fa8e006f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 -- 172.21.15.70:0/2765653512 >> v2:172.21.15.98:3300/0 conn(0x7fa8e00b3d60 msgr2=0x7fa8e00b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 --2- 172.21.15.70:0/2765653512 >> v2:172.21.15.98:3300/0 conn(0x7fa8e00b3d60 0x7fa8e00b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e592a640 1 -- 172.21.15.70:0/2765653512 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e006e030 con 0x7fa8e0071b70 2024-04-22T01:18:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.520+0000 7fa8e612b640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e015a320 0x7fa8e006f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:34.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e592a640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e0071b70 0x7fa8e0071fd0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00b2500 tx=0x7fa8dc01d600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34fbdcca3304e53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e5129640 1 -- 172.21.15.70:0/2765653512 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0aa5f0 con 0x7fa8e0071b70 2024-04-22T01:18:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e5129640 1 -- 172.21.15.70:0/2765653512 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8dc0aa790 con 0x7fa8e0071b70 2024-04-22T01:18:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e5129640 1 -- 172.21.15.70:0/2765653512 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0aab60 con 0x7fa8e0071b70 2024-04-22T01:18:34.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- 172.21.15.70:0/2765653512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e0071b70 msgr2=0x7fa8e0071fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e0071b70 0x7fa8e0071fd0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00b2500 tx=0x7fa8dc01d600 comp rx=0 tx=0).stop 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- 172.21.15.70:0/2765653512 shutdown_connections 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- 172.21.15.70:0/2765653512 >> v2:172.21.15.98:3300/0 conn(0x7fa8e00b3d60 0x7fa8e00b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e0071b70 0x7fa8e0071fd0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- 172.21.15.70:0/2765653512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e015a320 0x7fa8e006f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- 172.21.15.70:0/2765653512 >> 172.21.15.70:0/2765653512 conn(0x7fa8e00b1d40 msgr2=0x7fa8e00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- 172.21.15.70:0/2765653512 shutdown_connections 2024-04-22T01:18:34.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- 172.21.15.70:0/2765653512 wait complete. 2024-04-22T01:18:34.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 Processor -- start 2024-04-22T01:18:34.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- start start 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e0071b70 0x7fa8e014e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e00b3d60 0x7fa8e00afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 0x7fa8e00b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e00cf420 con 0x7fa8e015a320 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e015a700 con 0x7fa8e0071b70 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e838f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e00cf5a0 con 0x7fa8e00b3d60 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 0x7fa8e00b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 0x7fa8e00b0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41968/0 (socket says 172.21.15.70:41968) 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 -- 172.21.15.70:0/1080988626 learned_addr learned my addr 172.21.15.70:0/1080988626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:34.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e612b640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e0071b70 0x7fa8e014e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e592a640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e00b3d60 0x7fa8e00afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 -- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e00b3d60 msgr2=0x7fa8e00afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e00b3d60 0x7fa8e00afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 -- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e0071b70 msgr2=0x7fa8e014e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e0071b70 0x7fa8e014e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 -- 172.21.15.70:0/1080988626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e015aad0 con 0x7fa8e015a320 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8e692c640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 0x7fa8e00b0280 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800cd70 tx=0x7fa8d800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.524+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8003ba0 con 0x7fa8e015a320 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d8003d40 con 0x7fa8e015a320 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e015adb0 con 0x7fa8e015a320 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8004240 con 0x7fa8e015a320 2024-04-22T01:18:34.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e015b370 con 0x7fa8e015a320 2024-04-22T01:18:34.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e0068210 con 0x7fa8e015a320 2024-04-22T01:18:34.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8d80043e0 con 0x7fa8e015a320 2024-04-22T01:18:34.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8d2ffd640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8c40678c0 0x7fa8c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7fa8d8086a00 con 0x7fa8e015a320 2024-04-22T01:18:34.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8e612b640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8c40678c0 0x7fa8c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.528+0000 7fa8e612b640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8c40678c0 0x7fa8c4069d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4006fd0 tx=0x7fa8d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.532+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa8d8057340 con 0x7fa8e015a320 2024-04-22T01:18:34.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.692+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8e01596b0 con 0x7fa8e015a320 2024-04-22T01:18:34.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.692+0000 7fa8d2ffd640 1 -- 172.21.15.70:0/1080988626 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fa8d8056c40 con 0x7fa8e015a320 2024-04-22T01:18:34.696 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:18:34.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8c40678c0 msgr2=0x7fa8c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8c40678c0 0x7fa8c4069d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4006fd0 tx=0x7fa8d4008040 comp rx=0 tx=0).stop 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 msgr2=0x7fa8e00b0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 0x7fa8e00b0280 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800cd70 tx=0x7fa8d800a660 comp rx=0 tx=0).stop 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 shutdown_connections 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8c40678c0 0x7fa8c4069d80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e015a320 0x7fa8e00b0280 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8e00b3d60 0x7fa8e00afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 --2- 172.21.15.70:0/1080988626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e0071b70 0x7fa8e014e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 >> 172.21.15.70:0/1080988626 conn(0x7fa8e00b1d40 msgr2=0x7fa8e006f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 shutdown_connections 2024-04-22T01:18:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.696+0000 7fa8e838f640 1 -- 172.21.15.70:0/1080988626 wait complete. 2024-04-22T01:18:34.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:18:34.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.832+0000 7f6ce5000640 1 Processor -- start 2024-04-22T01:18:34.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6ce5000640 1 -- start start 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6ce5000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ce00b98a0 0x7f6ce00d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6ce5000640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ce00b9ce0 con 0x7f6ce00cea10 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6ce5000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ce00b75e0 con 0x7f6ce015de20 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6ce5000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6ce00b78c0 con 0x7f6ce00b98a0 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cded9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6ce00cea10 0x7f6ce00cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44562/0 (socket says 172.21.15.70:44562) 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cded9d640 1 -- 172.21.15.70:0/3315352920 learned_addr learned my addr 172.21.15.70:0/3315352920 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cd7fff640 1 --2- 172.21.15.70:0/3315352920 >> v2:172.21.15.98:3300/0 conn(0x7f6ce00b98a0 0x7f6ce00d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759418226 0 0) 0x7f6ce00b9ce0 con 0x7f6ce00cea10 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ccc003530 con 0x7f6ce00cea10 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3512732116 0 0) 0x7f6ccc003530 con 0x7f6ce00cea10 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 >> v2:172.21.15.98:3300/0 conn(0x7f6ce00b98a0 msgr2=0x7f6ce00d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 --2- 172.21.15.70:0/3315352920 >> v2:172.21.15.98:3300/0 conn(0x7f6ce00b98a0 0x7f6ce00d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 >> v1:172.21.15.98:6789/0 conn(0x7f6ce015de20 legacy=0x7f6ce00b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce00b7ba0 con 0x7f6ce00cea10 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f6cd8001a00 con 0x7f6ce00cea10 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 >> v1:172.21.15.70:6789/0 conn(0x7f6ce00cea10 legacy=0x7f6ce00cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc0039c0 0x7f6ccc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ccc004810 0x7f6ccc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0051d0 0x7f6ccc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce00b78c0 con 0x7f6ccc0039c0 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc00c080 con 0x7f6ccc004810 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc00c280 con 0x7f6ccc0051d0 2024-04-22T01:18:34.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cde59c640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc0039c0 0x7f6ccc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cd7fff640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ccc004810 0x7f6ccc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cde59c640 1 -- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0051d0 msgr2=0x7f6ccc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cde59c640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0051d0 0x7f6ccc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cde59c640 1 -- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ccc004810 msgr2=0x7f6ccc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cde59c640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ccc004810 0x7f6ccc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cded9d640 1 -- 172.21.15.70:0/3315352920 reap_dead start 2024-04-22T01:18:34.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.836+0000 7f6cde59c640 1 -- 172.21.15.70:0/3315352920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc003530 con 0x7f6ccc0039c0 2024-04-22T01:18:34.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cde59c640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc0039c0 0x7f6ccc003de0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0001780 tx=0x7f6cd001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0eee9bbe6d4c34 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00013a0 con 0x7f6ccc0039c0 2024-04-22T01:18:34.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd00a9070 con 0x7f6ccc0039c0 2024-04-22T01:18:34.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cddd9b640 1 -- 172.21.15.70:0/3315352920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0003600 con 0x7f6ccc0039c0 2024-04-22T01:18:34.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- 172.21.15.70:0/3315352920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc0039c0 msgr2=0x7f6ccc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc0039c0 0x7f6ccc003de0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0001780 tx=0x7f6cd001c420 comp rx=0 tx=0).stop 2024-04-22T01:18:34.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- 172.21.15.70:0/3315352920 shutdown_connections 2024-04-22T01:18:34.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 --2- 172.21.15.70:0/3315352920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc0039c0 0x7f6ccc003de0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- 172.21.15.70:0/3315352920 >> 172.21.15.70:0/3315352920 conn(0x7f6ce0072d20 msgr2=0x7f6ce0073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:34.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- 172.21.15.70:0/3315352920 shutdown_connections 2024-04-22T01:18:34.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- 172.21.15.70:0/3315352920 wait complete. 2024-04-22T01:18:34.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 Processor -- start 2024-04-22T01:18:34.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- start start 2024-04-22T01:18:34.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0039c0 0x7f6ce00d2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 0x7f6ce00cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce01531f0 0x7f6ce015bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce015e200 con 0x7f6ce0152d70 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce00b8160 con 0x7f6ce01531f0 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6ce5000640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce00cee60 con 0x7f6ccc0039c0 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 0x7f6ce00cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 0x7f6ce00cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41980/0 (socket says 172.21.15.70:41980) 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 -- 172.21.15.70:0/2137144973 learned_addr learned my addr 172.21.15.70:0/2137144973 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cde59c640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0039c0 0x7f6ce00d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cded9d640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce01531f0 0x7f6ce015bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 -- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0039c0 msgr2=0x7f6ce00d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0039c0 0x7f6ce00d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 -- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce01531f0 msgr2=0x7f6ce015bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce01531f0 0x7f6ce015bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 -- 172.21.15.70:0/2137144973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce01f6ef0 con 0x7f6ce0152d70 2024-04-22T01:18:34.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.840+0000 7f6cd7fff640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 0x7f6ce00cc550 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8011500 tx=0x7f6cc80119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8015870 con 0x7f6ce0152d70 2024-04-22T01:18:34.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc8015a10 con 0x7f6ce0152d70 2024-04-22T01:18:34.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce01f71d0 con 0x7f6ce0152d70 2024-04-22T01:18:34.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8015de0 con 0x7f6ce0152d70 2024-04-22T01:18:34.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce01f7790 con 0x7f6ce0152d70 2024-04-22T01:18:34.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce0068210 con 0x7f6ce0152d70 2024-04-22T01:18:34.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6cc801d400 con 0x7f6ce0152d70 2024-04-22T01:18:34.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cd6ffd640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cc00678c0 0x7f6cc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:34.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f6cc80866b0 con 0x7f6ce0152d70 2024-04-22T01:18:34.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cde59c640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cc00678c0 0x7f6cc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:34.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.844+0000 7f6cde59c640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cc00678c0 0x7f6cc0069d80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0001750 tx=0x7f6cd00011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:34.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:34.848+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6cc8019030 con 0x7f6ce0152d70 2024-04-22T01:18:35.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.004+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6ce006b9b0 con 0x7f6ce0152d70 2024-04-22T01:18:35.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.004+0000 7f6cd6ffd640 1 -- 172.21.15.70:0/2137144973 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f6cc8057000 con 0x7f6ce0152d70 2024-04-22T01:18:35.007 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cc00678c0 msgr2=0x7f6cc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cc00678c0 0x7f6cc0069d80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0001750 tx=0x7f6cd00011a0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 msgr2=0x7f6ce00cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 0x7f6ce00cc550 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8011500 tx=0x7f6cc80119c0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 shutdown_connections 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6cc00678c0 0x7f6cc0069d80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce01531f0 0x7f6ce015bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ce0152d70 0x7f6ce00cc550 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 --2- 172.21.15.70:0/2137144973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ccc0039c0 0x7f6ce00d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 >> 172.21.15.70:0/2137144973 conn(0x7f6ce0072d20 msgr2=0x7f6ce0073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:35.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 shutdown_connections 2024-04-22T01:18:35.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.008+0000 7f6ce5000640 1 -- 172.21.15.70:0/2137144973 wait complete. 2024-04-22T01:18:35.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:18:35.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.132+0000 7fde3b6e5640 1 Processor -- start 2024-04-22T01:18:35.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.132+0000 7fde3b6e5640 1 -- start start 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde3b6e5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fde340cc6a0 0x7fde3406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde3b6e5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3406b390 con 0x7fde340b5060 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde3b6e5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3406b670 con 0x7fde3406c690 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde3b6e5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fde3406cb20 con 0x7fde340cc6a0 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde39481640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fde340b5060 0x7fde340ad6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44568/0 (socket says 172.21.15.70:44568) 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde38c80640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fde340cc6a0 0x7fde3406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde39481640 1 -- 172.21.15.70:0/2685721706 learned_addr learned my addr 172.21.15.70:0/2685721706 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde2bfff640 1 -- 172.21.15.70:0/2685721706 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721108870 0 0) 0x7fde3406b390 con 0x7fde340b5060 2024-04-22T01:18:35.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde2bfff640 1 -- 172.21.15.70:0/2685721706 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde20005740 con 0x7fde340b5060 2024-04-22T01:18:35.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde38c80640 1 -- 172.21.15.70:0/2685721706 >> v1:172.21.15.98:6789/0 conn(0x7fde3406c690 legacy=0x7fde34163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde38c80640 1 -- 172.21.15.70:0/2685721706 >> v1:172.21.15.70:6789/0 conn(0x7fde340b5060 legacy=0x7fde340ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde38c80640 1 -- 172.21.15.70:0/2685721706 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3406b950 con 0x7fde340cc6a0 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde38c80640 1 --2- 172.21.15.70:0/2685721706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3406c150 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fde30009d90 tx=0x7fde3001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb70297473d171c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde2bfff640 1 -- 172.21.15.70:0/2685721706 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde300b2070 con 0x7fde340cc6a0 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde2bfff640 1 -- 172.21.15.70:0/2685721706 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde300a57b0 con 0x7fde340cc6a0 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.136+0000 7fde2bfff640 1 -- 172.21.15.70:0/2685721706 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde300a5b80 con 0x7fde340cc6a0 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/2685721706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 msgr2=0x7fde3406c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/2685721706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3406c150 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fde30009d90 tx=0x7fde3001c970 comp rx=0 tx=0).stop 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/2685721706 shutdown_connections 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/2685721706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3406c150 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/2685721706 >> 172.21.15.70:0/2685721706 conn(0x7fde34099a60 msgr2=0x7fde340b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/2685721706 shutdown_connections 2024-04-22T01:18:35.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/2685721706 wait complete. 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 Processor -- start 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- start start 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde3406c690 0x7fde34144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde340b5060 0x7fde3414d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fde34167810 con 0x7fde340cc6a0 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde3406bcc0 con 0x7fde340b5060 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fde34167990 con 0x7fde3406c690 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3414d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41992/0 (socket says 172.21.15.70:41992) 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 -- 172.21.15.70:0/1929812891 learned_addr learned my addr 172.21.15.70:0/1929812891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:35.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde38c80640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde340b5060 0x7fde3414d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39481640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde3406c690 0x7fde34144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 -- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde3406c690 msgr2=0x7fde34144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde3406c690 0x7fde34144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 -- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde340b5060 msgr2=0x7fde3414d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde340b5060 0x7fde3414d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 -- 172.21.15.70:0/1929812891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde341597e0 con 0x7fde340cc6a0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde39c82640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3414d7b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c340 tx=0x7fde2400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24005220 con 0x7fde340cc6a0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde240053c0 con 0x7fde340cc6a0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24007270 con 0x7fde340cc6a0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde34159ac0 con 0x7fde340cc6a0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3415ada0 con 0x7fde340cc6a0 2024-04-22T01:18:35.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.140+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3406ccb0 con 0x7fde340cc6a0 2024-04-22T01:18:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.144+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fde24019020 con 0x7fde340cc6a0 2024-04-22T01:18:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.144+0000 7fde29ffb640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fde180678c0 0x7fde18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.144+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7fde240851c0 con 0x7fde340cc6a0 2024-04-22T01:18:35.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.144+0000 7fde39481640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fde180678c0 0x7fde18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.144+0000 7fde39481640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fde180678c0 0x7fde18069d80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0011f0 tx=0x7fde2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.148+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fde24055490 con 0x7fde340cc6a0 2024-04-22T01:18:35.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.308+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fde3404f510 con 0x7fde340cc6a0 2024-04-22T01:18:35.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.308+0000 7fde29ffb640 1 -- 172.21.15.70:0/1929812891 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v91) ==== 80+0+0 (secure 0 0 0) 0x7fde24059790 con 0x7fde340cc6a0 2024-04-22T01:18:35.311 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fde180678c0 msgr2=0x7fde18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fde180678c0 0x7fde18069d80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0011f0 tx=0x7fde2c009040 comp rx=0 tx=0).stop 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 msgr2=0x7fde3414d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3414d7b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c340 tx=0x7fde2400c800 comp rx=0 tx=0).stop 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 shutdown_connections 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fde180678c0 0x7fde18069d80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde340cc6a0 0x7fde3414d7b0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde340b5060 0x7fde3414d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 --2- 172.21.15.70:0/1929812891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde3406c690 0x7fde34144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 >> 172.21.15.70:0/1929812891 conn(0x7fde34099a60 msgr2=0x7fde340d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:35.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 shutdown_connections 2024-04-22T01:18:35.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.312+0000 7fde3b6e5640 1 -- 172.21.15.70:0/1929812891 wait complete. 2024-04-22T01:18:35.328 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:18:35.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:35.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:35.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[45163]: fuse finished with error 0 and tester_r 0 2024-04-22T01:18:35.340 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:18:35.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:35.340 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:18:35.512 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:35.512 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:35.512 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:35.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:35.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 Processor -- start 2024-04-22T01:18:35.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 -- start start 2024-04-22T01:18:35.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff7900cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff7900cc150 0x7ff790071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7900d0250 con 0x7ff790072060 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff7900724d0 con 0x7ff7900cc150 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff797614640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7900cd7b0 con 0x7ff7901455b0 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff790072060 0x7ff79015f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44576/0 (socket says 172.21.15.70:44576) 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 -- 172.21.15.70:0/2029509549 learned_addr learned my addr 172.21.15.70:0/2029509549 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff794baf640 1 --2- 172.21.15.70:0/2029509549 >> v2:172.21.15.98:3300/0 conn(0x7ff7900cc150 0x7ff790071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff7953b0640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff7900cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687539040 0 0) 0x7ff7900d0250 con 0x7ff790072060 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff77c003530 con 0x7ff790072060 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2483239961 0 0) 0x7ff77c003530 con 0x7ff790072060 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 msgr2=0x7ff7900cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff7900cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 >> v2:172.21.15.98:3300/0 conn(0x7ff7900cc150 msgr2=0x7ff790071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 --2- 172.21.15.70:0/2029509549 >> v2:172.21.15.98:3300/0 conn(0x7ff7900cc150 0x7ff790071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7900cc710 con 0x7ff790072060 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7ff788001a00 con 0x7ff790072060 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 >> v1:172.21.15.70:6789/0 conn(0x7ff790072060 legacy=0x7ff79015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff77c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c003f40 0x7ff77c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff77c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7900724d0 con 0x7ff77c004d50 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7900cd7b0 con 0x7ff77c003f40 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff77c00bab0 con 0x7ff7901455b0 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff77c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff7953b0640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff77c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff794baf640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c003f40 0x7ff77c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 -- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 msgr2=0x7ff77c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff77c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 -- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c003f40 msgr2=0x7ff77c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c003f40 0x7ff77c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff795bb1640 1 -- 172.21.15.70:0/2029509549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c003530 con 0x7ff77c004d50 2024-04-22T01:18:35.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.676+0000 7ff7953b0640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7901455b0 0x7ff77c0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff795bb1640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff77c0051d0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff78800a870 tx=0x7ff78801d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcc86a51a0bc3c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78800a4c0 con 0x7ff77c004d50 2024-04-22T01:18:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7880ad630 con 0x7ff77c004d50 2024-04-22T01:18:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff787fff640 1 -- 172.21.15.70:0/2029509549 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7880ada20 con 0x7ff77c004d50 2024-04-22T01:18:35.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- 172.21.15.70:0/2029509549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 msgr2=0x7ff77c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 --2- 172.21.15.70:0/2029509549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff77c0051d0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff78800a870 tx=0x7ff78801d3a0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff795bb1640 1 -- 172.21.15.70:0/2029509549 reap_dead start 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- 172.21.15.70:0/2029509549 shutdown_connections 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- 172.21.15.70:0/2029509549 >> 172.21.15.70:0/2029509549 conn(0x7ff79006b1f0 msgr2=0x7ff79006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- 172.21.15.70:0/2029509549 shutdown_connections 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- 172.21.15.70:0/2029509549 wait complete. 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 Processor -- start 2024-04-22T01:18:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- start start 2024-04-22T01:18:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff790165870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7901653d0 0x7ff79015afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff79015b4e0 0x7ff7901f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff790166b30 con 0x7ff77c004d50 2024-04-22T01:18:35.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff79006f210 con 0x7ff7901653d0 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.680+0000 7ff797614640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff790166cb0 con 0x7ff79015b4e0 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff790165870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff790165870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42008/0 (socket says 172.21.15.70:42008) 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 -- 172.21.15.70:0/2344122422 learned_addr learned my addr 172.21.15.70:0/2344122422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff794baf640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7901653d0 0x7ff79015afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff795bb1640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff79015b4e0 0x7ff7901f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 -- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff79015b4e0 msgr2=0x7ff7901f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff79015b4e0 0x7ff7901f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 -- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7901653d0 msgr2=0x7ff79015afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7901653d0 0x7ff79015afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 -- 172.21.15.70:0/2344122422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff790165fe0 con 0x7ff77c004d50 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff7953b0640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff790165870 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009fd0 tx=0x7ff78c011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c004220 con 0x7ff77c004d50 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff78c0043c0 con 0x7ff77c004d50 2024-04-22T01:18:35.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff790164610 con 0x7ff77c004d50 2024-04-22T01:18:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c00f9e0 con 0x7ff77c004d50 2024-04-22T01:18:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff790164bd0 con 0x7ff77c004d50 2024-04-22T01:18:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff79015b8f0 con 0x7ff77c004d50 2024-04-22T01:18:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff78c00fb80 con 0x7ff77c004d50 2024-04-22T01:18:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff785ffb640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7740678f0 0x7ff774069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:35.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7ff78c086570 con 0x7ff77c004d50 2024-04-22T01:18:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff794baf640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7740678f0 0x7ff774069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.684+0000 7ff794baf640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7740678f0 0x7ff774069db0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff790141120 tx=0x7ff780008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:35.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.688+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff78c019030 con 0x7ff77c004d50 2024-04-22T01:18:35.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.852+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff790068210 con 0x7ff77c004d50 2024-04-22T01:18:35.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.852+0000 7ff785ffb640 1 -- 172.21.15.70:0/2344122422 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+17042 (secure 0 0 0) 0x7ff78c056eb0 con 0x7ff77c004d50 2024-04-22T01:18:35.856 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:35.856 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:29.400245+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7740678f0 msgr2=0x7ff774069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7740678f0 0x7ff774069db0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff790141120 tx=0x7ff780008040 comp rx=0 tx=0).stop 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 msgr2=0x7ff790165870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff790165870 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009fd0 tx=0x7ff78c011b30 comp rx=0 tx=0).stop 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 shutdown_connections 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7740678f0 0x7ff774069db0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff79015b4e0 0x7ff7901f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7901653d0 0x7ff79015afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 --2- 172.21.15.70:0/2344122422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff77c004d50 0x7ff790165870 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:35.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 >> 172.21.15.70:0/2344122422 conn(0x7ff79006b1f0 msgr2=0x7ff7900b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:35.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 shutdown_connections 2024-04-22T01:18:35.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:35.856+0000 7ff797614640 1 -- 172.21.15.70:0/2344122422 wait complete. 2024-04-22T01:18:35.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:18:20.962202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:18:21.681268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:35.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:36.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 Processor -- start 2024-04-22T01:18:36.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 -- start start 2024-04-22T01:18:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f929815de20 0x7f92980b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92980cea10 0x7f92980cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92980b75e0 con 0x7f92980b6db0 2024-04-22T01:18:36.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f92980b78c0 con 0x7f929815de20 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929f476640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f92980b7ba0 con 0x7f92980cea10 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f929815de20 0x7f92980b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f929815de20 0x7f92980b98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42018/0 (socket says 172.21.15.70:42018) 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 -- 172.21.15.70:0/3654577170 learned_addr learned my addr 172.21.15.70:0/3654577170 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929da13640 1 --2- 172.21.15.70:0/3654577170 >> v2:172.21.15.98:3300/0 conn(0x7f92980cea10 0x7f92980cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 -- 172.21.15.70:0/3654577170 >> v2:172.21.15.98:3300/0 conn(0x7f92980cea10 msgr2=0x7f92980cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 --2- 172.21.15.70:0/3654577170 >> v2:172.21.15.98:3300/0 conn(0x7f92980cea10 0x7f92980cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 -- 172.21.15.70:0/3654577170 >> v1:172.21.15.98:6789/0 conn(0x7f92980b6db0 legacy=0x7f92980d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.000+0000 7f929d212640 1 -- 172.21.15.70:0/3654577170 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92980b7e80 con 0x7f929815de20 2024-04-22T01:18:36.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929d212640 1 --2- 172.21.15.70:0/3654577170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929815de20 0x7f92980b98a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f92980b6d00 tx=0x7f929401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1950bb1fa117d2df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f928ffff640 1 -- 172.21.15.70:0/3654577170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940aa5f0 con 0x7f929815de20 2024-04-22T01:18:36.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f928ffff640 1 -- 172.21.15.70:0/3654577170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92940af070 con 0x7f929815de20 2024-04-22T01:18:36.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f928ffff640 1 -- 172.21.15.70:0/3654577170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940aa9b0 con 0x7f929815de20 2024-04-22T01:18:36.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 -- 172.21.15.70:0/3654577170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929815de20 msgr2=0x7f92980b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 --2- 172.21.15.70:0/3654577170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929815de20 0x7f92980b98a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f92980b6d00 tx=0x7f929401c4b0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 -- 172.21.15.70:0/3654577170 shutdown_connections 2024-04-22T01:18:36.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 --2- 172.21.15.70:0/3654577170 >> v2:172.21.15.98:3300/0 conn(0x7f92980cea10 0x7f92980cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 --2- 172.21.15.70:0/3654577170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929815de20 0x7f92980b98a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 -- 172.21.15.70:0/3654577170 >> 172.21.15.70:0/3654577170 conn(0x7f9298072d20 msgr2=0x7f9298073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 -- 172.21.15.70:0/3654577170 shutdown_connections 2024-04-22T01:18:36.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 -- 172.21.15.70:0/3654577170 wait complete. 2024-04-22T01:18:36.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 Processor -- start 2024-04-22T01:18:36.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.004+0000 7f929f476640 1 -- start start 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92980b6db0 0x7f92980d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 0x7f929815c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f929815de20 0x7f929814b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f92980b7370 con 0x7f92980cea10 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92980b71f0 con 0x7f92980b6db0 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f92980b8e90 con 0x7f929815de20 2024-04-22T01:18:36.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 0x7f929815c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 0x7f929815c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42022/0 (socket says 172.21.15.70:42022) 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 -- 172.21.15.70:0/4251201893 learned_addr learned my addr 172.21.15.70:0/4251201893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929d212640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92980b6db0 0x7f92980d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929da13640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f929815de20 0x7f929814b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 -- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f929815de20 msgr2=0x7f929814b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f929815de20 0x7f929814b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 -- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92980b6db0 msgr2=0x7f92980d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92980b6db0 0x7f92980d1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 -- 172.21.15.70:0/4251201893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929815d330 con 0x7f92980cea10 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929d212640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92980b6db0 0x7f92980d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929ca11640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 0x7f929815c190 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f928800c480 tx=0x7f928800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288012830 con 0x7f92980cea10 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9288017070 con 0x7f92980cea10 2024-04-22T01:18:36.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288012c30 con 0x7f92980cea10 2024-04-22T01:18:36.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929815ca90 con 0x7f92980cea10 2024-04-22T01:18:36.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929815d050 con 0x7f92980cea10 2024-04-22T01:18:36.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9298068210 con 0x7f92980cea10 2024-04-22T01:18:36.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f928801c460 con 0x7f92980cea10 2024-04-22T01:18:36.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f928dffb640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f927c0678c0 0x7f927c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f9288085be0 con 0x7f92980cea10 2024-04-22T01:18:36.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.008+0000 7f929d212640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f927c0678c0 0x7f927c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.012+0000 7f929d212640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f927c0678c0 0x7f927c069d80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f92940ae000 tx=0x7f92940a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.016+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f92880565b0 con 0x7f92980cea10 2024-04-22T01:18:36.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.176+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f92980b4030 con 0x7f92980cea10 2024-04-22T01:18:36.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.176+0000 7f928dffb640 1 -- 172.21.15.70:0/4251201893 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f9288055eb0 con 0x7f92980cea10 2024-04-22T01:18:36.179 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:36.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f927c0678c0 msgr2=0x7f927c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f927c0678c0 0x7f927c069d80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f92940ae000 tx=0x7f92940a7000 comp rx=0 tx=0).stop 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 msgr2=0x7f929815c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 0x7f929815c190 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f928800c480 tx=0x7f928800c940 comp rx=0 tx=0).stop 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 shutdown_connections 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f927c0678c0 0x7f927c069d80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f929815de20 0x7f929814b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92980cea10 0x7f929815c190 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 --2- 172.21.15.70:0/4251201893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92980b6db0 0x7f92980d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 >> 172.21.15.70:0/4251201893 conn(0x7f9298072d20 msgr2=0x7f92980cb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 shutdown_connections 2024-04-22T01:18:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.180+0000 7f929f476640 1 -- 172.21.15.70:0/4251201893 wait complete. 2024-04-22T01:18:36.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:18:36.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eb199c640 1 Processor -- start 2024-04-22T01:18:36.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eb199c640 1 -- start start 2024-04-22T01:18:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eb199c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 0x7f2eac071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eb199c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac06da70 con 0x7f2eac0cd410 2024-04-22T01:18:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eb199c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac06dd50 con 0x7f2eac15a320 2024-04-22T01:18:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eb199c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac06f860 con 0x7f2eac071b70 2024-04-22T01:18:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2eac0cd410 0x7f2eac0cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34966/0 (socket says 172.21.15.70:34966) 2024-04-22T01:18:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 -- 172.21.15.70:0/4001339716 learned_addr learned my addr 172.21.15.70:0/4001339716 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:36.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230062783 0 0) 0x7f2eac06da70 con 0x7f2eac0cd410 2024-04-22T01:18:36.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e94003450 con 0x7f2eac0cd410 2024-04-22T01:18:36.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 0x7f2eac071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 109113196 0 0) 0x7f2e94003450 con 0x7f2eac0cd410 2024-04-22T01:18:36.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 msgr2=0x7f2eac071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 0x7f2eac071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 >> v1:172.21.15.98:6789/0 conn(0x7f2eac15a320 legacy=0x7f2eac06f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac06e030 con 0x7f2eac0cd410 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f2ea4001a00 con 0x7f2eac0cd410 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 >> v1:172.21.15.70:6789/0 conn(0x7f2eac0cd410 legacy=0x7f2eac0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 0x7f2e94003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e94004060 0x7f2e9400a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9400b000 0x7f2e9400d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac06f860 con 0x7f2e9400b000 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9400dcd0 con 0x7f2e94004060 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9400ded0 con 0x7f2eac071b70 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9400b000 0x7f2e9400d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eaaffd640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 0x7f2e94003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e94004060 0x7f2e9400a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 -- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 msgr2=0x7f2e94003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac071b70 0x7f2e94003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 -- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e94004060 msgr2=0x7f2e9400a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e94004060 0x7f2e9400a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eab7fe640 1 -- 172.21.15.70:0/4001339716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e94003450 con 0x7f2e9400b000 2024-04-22T01:18:36.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.320+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e94004060 0x7f2e9400a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:36.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eab7fe640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9400b000 0x7f2e9400d490 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400a850 tx=0x7f2ea401d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290160898eccf19b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400a4a0 con 0x7f2e9400b000 2024-04-22T01:18:36.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea40ad650 con 0x7f2e9400b000 2024-04-22T01:18:36.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2ea9ffb640 1 -- 172.21.15.70:0/4001339716 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40ada80 con 0x7f2e9400b000 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- 172.21.15.70:0/4001339716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9400b000 msgr2=0x7f2e9400d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 --2- 172.21.15.70:0/4001339716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e9400b000 0x7f2e9400d490 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400a850 tx=0x7f2ea401d3a0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eab7fe640 1 -- 172.21.15.70:0/4001339716 reap_dead start 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- 172.21.15.70:0/4001339716 shutdown_connections 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- 172.21.15.70:0/4001339716 >> 172.21.15.70:0/4001339716 conn(0x7f2eac0b1d40 msgr2=0x7f2eac0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- 172.21.15.70:0/4001339716 shutdown_connections 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- 172.21.15.70:0/4001339716 wait complete. 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 Processor -- start 2024-04-22T01:18:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- start start 2024-04-22T01:18:36.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 0x7f2eac13e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eac149610 0x7f2eac14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac14cb60 0x7f2eac14cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac15a700 con 0x7f2eac13df80 2024-04-22T01:18:36.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac06e5f0 con 0x7f2eac149610 2024-04-22T01:18:36.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.324+0000 7f2eb199c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac0cf420 con 0x7f2eac14cb60 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 0x7f2eac13e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 0x7f2eac13e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57658/0 (socket says 172.21.15.70:57658) 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 -- 172.21.15.70:0/494517379 learned_addr learned my addr 172.21.15.70:0/494517379 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eac149610 0x7f2eac14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 -- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac14cb60 msgr2=0x7f2eac14cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac14cb60 0x7f2eac14cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 -- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eac149610 msgr2=0x7f2eac14c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eac149610 0x7f2eac14c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 -- 172.21.15.70:0/494517379 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac14b8a0 con 0x7f2eac13df80 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eac149610 0x7f2eac14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaaffd640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 0x7f2eac13e500 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00e530 tx=0x7f2e9c00e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0052e0 con 0x7f2eac13df80 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9c018070 con 0x7f2eac13df80 2024-04-22T01:18:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac15d4f0 con 0x7f2eac13df80 2024-04-22T01:18:36.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c003820 con 0x7f2eac13df80 2024-04-22T01:18:36.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac14be20 con 0x7f2eac13df80 2024-04-22T01:18:36.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac068210 con 0x7f2eac13df80 2024-04-22T01:18:36.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e9c0039c0 con 0x7f2eac13df80 2024-04-22T01:18:36.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2e93fff640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e980678f0 0x7f2e98069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f2e9c085700 con 0x7f2eac13df80 2024-04-22T01:18:36.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e980678f0 0x7f2e98069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.328+0000 7f2eaa7fc640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e980678f0 0x7f2e98069db0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0b0ef0 tx=0x7f2ea0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.332+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2e9c056040 con 0x7f2eac13df80 2024-04-22T01:18:36.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.488+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2eac157ae0 con 0x7f2eac13df80 2024-04-22T01:18:36.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.488+0000 7f2e93fff640 1 -- 172.21.15.70:0/494517379 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f2e9c005480 con 0x7f2eac13df80 2024-04-22T01:18:36.491 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e980678f0 msgr2=0x7f2e98069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e980678f0 0x7f2e98069db0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0b0ef0 tx=0x7f2ea0008040 comp rx=0 tx=0).stop 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 msgr2=0x7f2eac13e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 0x7f2eac13e500 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00e530 tx=0x7f2e9c00e9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 shutdown_connections 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e980678f0 0x7f2e98069db0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2eac14cb60 0x7f2eac14cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eac149610 0x7f2eac14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 --2- 172.21.15.70:0/494517379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2eac13df80 0x7f2eac13e500 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 >> 172.21.15.70:0/494517379 conn(0x7f2eac0b1d40 msgr2=0x7f2eac1510c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 shutdown_connections 2024-04-22T01:18:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.492+0000 7f2eb199c640 1 -- 172.21.15.70:0/494517379 wait complete. 2024-04-22T01:18:36.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:18:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1187d13640 1 Processor -- start 2024-04-22T01:18:36.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1187d13640 1 -- start start 2024-04-22T01:18:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1187d13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 0x7f11800b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1187d13640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11800b75e0 con 0x7f11800b6db0 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1187d13640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11800b78c0 con 0x7f11800cea10 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1187d13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f11800b7ba0 con 0x7f118015de20 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11852ae640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f11800b6db0 0x7f11800d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34970/0 (socket says 172.21.15.70:34970) 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11852ae640 1 -- 172.21.15.70:0/3982449084 learned_addr learned my addr 172.21.15.70:0/3982449084 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1185aaf640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 0x7f11800b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1722344568 0 0) 0x7f11800b75e0 con 0x7f11800b6db0 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f116c003530 con 0x7f11800b6db0 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4115632625 0 0) 0x7f116c003530 con 0x7f11800b6db0 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 msgr2=0x7f11800b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 0x7f11800b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 >> v1:172.21.15.98:6789/0 conn(0x7f11800cea10 legacy=0x7f11800ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11800b7e80 con 0x7f11800b6db0 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1170001680 con 0x7f11800b6db0 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 >> v1:172.21.15.70:6789/0 conn(0x7f11800b6db0 legacy=0x7f11800d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f116c0039f0 0x7f116c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 0x7f116c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f116c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11800b7ba0 con 0x7f116c004d50 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f116c00bb10 con 0x7f116c0039f0 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f116c00bd10 con 0x7f118015de20 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11862b0640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f116c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f1185aaf640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f116c0039f0 0x7f116c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11852ae640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 0x7f116c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11862b0640 1 -- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 msgr2=0x7f116c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11862b0640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f118015de20 0x7f116c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11862b0640 1 -- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f116c0039f0 msgr2=0x7f116c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11862b0640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f116c0039f0 0x7f116c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.616+0000 7f11862b0640 1 -- 172.21.15.70:0/3982449084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116c003530 con 0x7f116c004d50 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f11862b0640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f116c0051d0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f1178001750 tx=0x7f117801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d79305f48d4e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11780013a0 con 0x7f116c004d50 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11780052f0 con 0x7f116c004d50 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1184aad640 1 -- 172.21.15.70:0/3982449084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1178003840 con 0x7f116c004d50 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- 172.21.15.70:0/3982449084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 msgr2=0x7f116c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 --2- 172.21.15.70:0/3982449084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f116c0051d0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f1178001750 tx=0x7f117801c3e0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f11862b0640 1 -- 172.21.15.70:0/3982449084 reap_dead start 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- 172.21.15.70:0/3982449084 shutdown_connections 2024-04-22T01:18:36.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- 172.21.15.70:0/3982449084 >> 172.21.15.70:0/3982449084 conn(0x7f1180072d20 msgr2=0x7f1180073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- 172.21.15.70:0/3982449084 shutdown_connections 2024-04-22T01:18:36.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- 172.21.15.70:0/3982449084 wait complete. 2024-04-22T01:18:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 Processor -- start 2024-04-22T01:18:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- start start 2024-04-22T01:18:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f11801f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11801f30d0 0x7f11800cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11800c99b0 0x7f1180154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11800b71f0 con 0x7f116c004d50 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f118015e200 con 0x7f11801f30d0 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.620+0000 7f1187d13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f11800b7370 con 0x7f11800c99b0 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f11801f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f11801f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57688/0 (socket says 172.21.15.70:57688) 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 -- 172.21.15.70:0/469684014 learned_addr learned my addr 172.21.15.70:0/469684014 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11852ae640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11801f30d0 0x7f11800cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11862b0640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11800c99b0 0x7f1180154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 -- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11800c99b0 msgr2=0x7f1180154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11800c99b0 0x7f1180154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 -- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11801f30d0 msgr2=0x7f11800cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11801f30d0 0x7f11800cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 -- 172.21.15.70:0/469684014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11800c8c10 con 0x7f116c004d50 2024-04-22T01:18:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11852ae640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11801f30d0 0x7f11800cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1185aaf640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f11801f41f0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f117c007f60 tx=0x7f117c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c01a020 con 0x7f116c004d50 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f117c016470 con 0x7f116c004d50 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11801f3550 con 0x7f116c004d50 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c016860 con 0x7f116c004d50 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11800c91b0 con 0x7f116c004d50 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1180068210 con 0x7f116c004d50 2024-04-22T01:18:36.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f117c01d020 con 0x7f116c004d50 2024-04-22T01:18:36.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11767fc640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f11640678f0 0x7f1164069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f117c086f30 con 0x7f116c004d50 2024-04-22T01:18:36.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11852ae640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f11640678f0 0x7f1164069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.624+0000 7f11852ae640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f11640678f0 0x7f1164069db0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1170001000 tx=0x7f11700014d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.628+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f117c024030 con 0x7f116c004d50 2024-04-22T01:18:36.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1180074a00 con 0x7f116c004d50 2024-04-22T01:18:36.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f11767fc640 1 -- 172.21.15.70:0/469684014 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f117c057220 con 0x7f116c004d50 2024-04-22T01:18:36.794 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:18:36.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f11640678f0 msgr2=0x7f1164069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f11640678f0 0x7f1164069db0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1170001000 tx=0x7f11700014d0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 msgr2=0x7f11801f41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f11801f41f0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f117c007f60 tx=0x7f117c007f90 comp rx=0 tx=0).stop 2024-04-22T01:18:36.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 shutdown_connections 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f11640678f0 0x7f1164069db0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11800c99b0 0x7f1180154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11801f30d0 0x7f11800cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 --2- 172.21.15.70:0/469684014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116c004d50 0x7f11801f41f0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 >> 172.21.15.70:0/469684014 conn(0x7f1180072d20 msgr2=0x7f118015c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 shutdown_connections 2024-04-22T01:18:36.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.792+0000 7f1187d13640 1 -- 172.21.15.70:0/469684014 wait complete. 2024-04-22T01:18:36.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:18:36.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 Processor -- start 2024-04-22T01:18:36.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 -- start start 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f69e015ba20 0x7f69e015e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69e00b0300 0x7f69e00b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69e00b07b0 con 0x7f69e00cfd00 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f69e015be00 con 0x7f69e015ba20 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69e5c07640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f69e015ea00 con 0x7f69e00b0300 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69df7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f69e015ba20 0x7f69e015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69dffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f69e00cfd00 0x7f69e00d0120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34982/0 (socket says 172.21.15.70:34982) 2024-04-22T01:18:36.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69dffff640 1 -- 172.21.15.70:0/3922979108 learned_addr learned my addr 172.21.15.70:0/3922979108 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:36.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69deffd640 1 --2- 172.21.15.70:0/3922979108 >> v2:172.21.15.98:3300/0 conn(0x7f69e00b0300 0x7f69e00b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69df7fe640 1 -- 172.21.15.70:0/3922979108 >> v2:172.21.15.98:3300/0 conn(0x7f69e00b0300 msgr2=0x7f69e00b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:36.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69df7fe640 1 --2- 172.21.15.70:0/3922979108 >> v2:172.21.15.98:3300/0 conn(0x7f69e00b0300 0x7f69e00b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69df7fe640 1 -- 172.21.15.70:0/3922979108 >> v1:172.21.15.70:6789/0 conn(0x7f69e00cfd00 legacy=0x7f69e00d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69de7fc640 1 -- 172.21.15.70:0/3922979108 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28607072 0 0) 0x7f69e00b07b0 con 0x7f69e00cfd00 2024-04-22T01:18:36.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.932+0000 7f69df7fe640 1 -- 172.21.15.70:0/3922979108 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e015ece0 con 0x7f69e015ba20 2024-04-22T01:18:36.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69df7fe640 1 --2- 172.21.15.70:0/3922979108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015ba20 0x7f69e015e420 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f69e0072f90 tx=0x7f69d001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c525e8f963301db0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69de7fc640 1 -- 172.21.15.70:0/3922979108 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00b0070 con 0x7f69e015ba20 2024-04-22T01:18:36.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69de7fc640 1 -- 172.21.15.70:0/3922979108 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d00ab600 con 0x7f69e015ba20 2024-04-22T01:18:36.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69de7fc640 1 -- 172.21.15.70:0/3922979108 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00ab9d0 con 0x7f69e015ba20 2024-04-22T01:18:36.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 -- 172.21.15.70:0/3922979108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015ba20 msgr2=0x7f69e015e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 --2- 172.21.15.70:0/3922979108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015ba20 0x7f69e015e420 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f69e0072f90 tx=0x7f69d001c480 comp rx=0 tx=0).stop 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 -- 172.21.15.70:0/3922979108 shutdown_connections 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 --2- 172.21.15.70:0/3922979108 >> v2:172.21.15.98:3300/0 conn(0x7f69e00b0300 0x7f69e00b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 --2- 172.21.15.70:0/3922979108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015ba20 0x7f69e015e420 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 -- 172.21.15.70:0/3922979108 >> 172.21.15.70:0/3922979108 conn(0x7f69e0099a60 msgr2=0x7f69e0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 -- 172.21.15.70:0/3922979108 shutdown_connections 2024-04-22T01:18:36.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.936+0000 7f69e5c07640 1 -- 172.21.15.70:0/3922979108 wait complete. 2024-04-22T01:18:36.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 Processor -- start 2024-04-22T01:18:36.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- start start 2024-04-22T01:18:36.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00b0300 0x7f69e015f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 0x7f69e01f9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015ba20 0x7f69e00b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69e00ce0a0 con 0x7f69e00cfd00 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69e00cdf20 con 0x7f69e00b0300 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f69e00ce220 con 0x7f69e015ba20 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 0x7f69e01f9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 0x7f69e01f9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57702/0 (socket says 172.21.15.70:57702) 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 -- 172.21.15.70:0/662865569 learned_addr learned my addr 172.21.15.70:0/662865569 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 -- 172.21.15.70:0/662865569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015ba20 msgr2=0x7f69e00b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015ba20 0x7f69e00b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 -- 172.21.15.70:0/662865569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00b0300 msgr2=0x7f69e015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00b0300 0x7f69e015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 -- 172.21.15.70:0/662865569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e01519d0 con 0x7f69e00cfd00 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69deffd640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 0x7f69e01f9960 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f69d400d620 tx=0x7f69d400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d400b620 con 0x7f69e00cfd00 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d4019070 con 0x7f69e00cfd00 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e0151c70 con 0x7f69e00cfd00 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d400b9e0 con 0x7f69e00cfd00 2024-04-22T01:18:36.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e0150bc0 con 0x7f69e00cfd00 2024-04-22T01:18:36.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e004f510 con 0x7f69e00cfd00 2024-04-22T01:18:36.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69d400bb80 con 0x7f69e00cfd00 2024-04-22T01:18:36.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69c7fff640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69cc0678c0 0x7f69cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:36.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69df7fe640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69cc0678c0 0x7f69cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:36.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.940+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f69d4086240 con 0x7f69e00cfd00 2024-04-22T01:18:36.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.944+0000 7f69df7fe640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69cc0678c0 0x7f69cc069d80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f69d00ad000 tx=0x7f69d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:36.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:36.948+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f69d4056c10 con 0x7f69e00cfd00 2024-04-22T01:18:37.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.104+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f69e0068c40 con 0x7f69e00cfd00 2024-04-22T01:18:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69c7fff640 1 -- 172.21.15.70:0/662865569 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v91) ==== 80+0+0 (secure 0 0 0) 0x7f69d4056510 con 0x7f69e00cfd00 2024-04-22T01:18:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:18:37.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69cc0678c0 msgr2=0x7f69cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69cc0678c0 0x7f69cc069d80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f69d00ad000 tx=0x7f69d0008040 comp rx=0 tx=0).stop 2024-04-22T01:18:37.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 msgr2=0x7f69e01f9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 0x7f69e01f9960 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f69d400d620 tx=0x7f69d400dae0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 shutdown_connections 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69cc0678c0 0x7f69cc069d80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015ba20 0x7f69e00b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00cfd00 0x7f69e01f9960 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 --2- 172.21.15.70:0/662865569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00b0300 0x7f69e015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 >> 172.21.15.70:0/662865569 conn(0x7f69e0099a60 msgr2=0x7f69e00b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 shutdown_connections 2024-04-22T01:18:37.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.108+0000 7f69e5c07640 1 -- 172.21.15.70:0/662865569 wait complete. 2024-04-22T01:18:37.126 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:18:37.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:37.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:18:37.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34649]: fuse finished with error 0 and tester_r 0 2024-04-22T01:18:37.142 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:18:37.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:37.142 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:18:37.214 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:37.214 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:37.214 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:37.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:18:37.220 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:18:37.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:37.220 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:18:37.270 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-22T01:18:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:18:37.273 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:18:37.273 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:37.273 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:37.279 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:18:37.279 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:18:37.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:37.279 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:18:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-22T01:18:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:18:37.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 Processor -- start 2024-04-22T01:18:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 -- start start 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f093415dfb0 0x7f09340b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340cea80 0x7f09340ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f09340b5730 0x7f09340b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f09340cf420 con 0x7f09340b5730 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f093415e390 con 0x7f093415dfb0 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093c40e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0934161e00 con 0x7f09340cea80 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f09340b5730 0x7f09340b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f09340b5730 0x7f09340b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57708/0 (socket says 172.21.15.70:57708) 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 -- 172.21.15.70:0/1078648540 learned_addr learned my addr 172.21.15.70:0/1078648540 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a1aa640 1 --2- 172.21.15.70:0/1078648540 >> v2:172.21.15.98:3300/0 conn(0x7f093415dfb0 0x7f09340b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f09399a9640 1 --2- 172.21.15.70:0/1078648540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340cea80 0x7f09340ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 -- 172.21.15.70:0/1078648540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340cea80 msgr2=0x7f09340ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:37.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 --2- 172.21.15.70:0/1078648540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340cea80 0x7f09340ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 -- 172.21.15.70:0/1078648540 >> v2:172.21.15.98:3300/0 conn(0x7f093415dfb0 msgr2=0x7f09340b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:37.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 --2- 172.21.15.70:0/1078648540 >> v2:172.21.15.98:3300/0 conn(0x7f093415dfb0 0x7f09340b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a9ab640 1 -- 172.21.15.70:0/1078648540 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09341620e0 con 0x7f09340b5730 2024-04-22T01:18:37.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.448+0000 7f093a1aa640 1 --2- 172.21.15.70:0/1078648540 >> v2:172.21.15.98:3300/0 conn(0x7f093415dfb0 0x7f09340b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093a9ab640 1 --2- 172.21.15.70:0/1078648540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340b5730 0x7f09340b5b90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f09340d4d80 tx=0x7f092c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60ec5c2cf7b0d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f09391a8640 1 -- 172.21.15.70:0/1078648540 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c0a5780 con 0x7f09340b5730 2024-04-22T01:18:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f09391a8640 1 -- 172.21.15.70:0/1078648540 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f092c0a5920 con 0x7f09340b5730 2024-04-22T01:18:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f09391a8640 1 -- 172.21.15.70:0/1078648540 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c0a5cf0 con 0x7f09340b5730 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 -- 172.21.15.70:0/1078648540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340b5730 msgr2=0x7f09340b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 --2- 172.21.15.70:0/1078648540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340b5730 0x7f09340b5b90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f09340d4d80 tx=0x7f092c01c9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 -- 172.21.15.70:0/1078648540 shutdown_connections 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 --2- 172.21.15.70:0/1078648540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340b5730 0x7f09340b5b90 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 --2- 172.21.15.70:0/1078648540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340cea80 0x7f09340ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 --2- 172.21.15.70:0/1078648540 >> v2:172.21.15.98:3300/0 conn(0x7f093415dfb0 0x7f09340b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 -- 172.21.15.70:0/1078648540 >> 172.21.15.70:0/1078648540 conn(0x7f093406c9b0 msgr2=0x7f093406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 -- 172.21.15.70:0/1078648540 shutdown_connections 2024-04-22T01:18:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 -- 172.21.15.70:0/1078648540 wait complete. 2024-04-22T01:18:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 Processor -- start 2024-04-22T01:18:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.452+0000 7f093c40e640 1 -- start start 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340b5730 0x7f09340c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 0x7f0934155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093415dfb0 0x7f093415ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0934161a90 con 0x7f09340cea80 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09340b9b10 con 0x7f093415dfb0 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f09340d2900 con 0x7f09340b5730 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 0x7f0934155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 0x7f0934155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57712/0 (socket says 172.21.15.70:57712) 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 -- 172.21.15.70:0/1865080155 learned_addr learned my addr 172.21.15.70:0/1865080155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093a9ab640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093415dfb0 0x7f093415ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 -- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340b5730 msgr2=0x7f09340c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340b5730 0x7f09340c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 -- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093415dfb0 msgr2=0x7f093415ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093415dfb0 0x7f093415ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 -- 172.21.15.70:0/1865080155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0934155ef0 con 0x7f09340cea80 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f09399a9640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 0x7f0934155430 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f093000d800 tx=0x7f093000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930018020 con 0x7f09340cea80 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09300096c0 con 0x7f09340cea80 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09341561d0 con 0x7f09340cea80 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930009ab0 con 0x7f09340cea80 2024-04-22T01:18:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09341542a0 con 0x7f09340cea80 2024-04-22T01:18:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f093415b920 con 0x7f09340cea80 2024-04-22T01:18:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f093001b020 con 0x7f09340cea80 2024-04-22T01:18:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f092affd640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09180678c0 0x7f0918069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f09300863c0 con 0x7f09340cea80 2024-04-22T01:18:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093a1aa640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09180678c0 0x7f0918069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.456+0000 7f093a1aa640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09180678c0 0x7f0918069d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0924006f10 tx=0x7f0924008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:37.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.460+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0930022020 con 0x7f09340cea80 2024-04-22T01:18:37.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.640+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0934068210 con 0x7f09340cea80 2024-04-22T01:18:37.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.640+0000 7f092affd640 1 -- 172.21.15.70:0/1865080155 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7f0930056d90 con 0x7f09340cea80 2024-04-22T01:18:37.643 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:37.643 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:22.684405+0000","modified":"2024-04-22T01:18:24.723325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7215":{"gid":7215,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2337533615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2337533615},{"type":"v1","addr":"172.21.15.70:6837","nonce":2337533615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7215,"qdb_cluster":[7215]},"id":6}]} 2024-04-22T01:18:37.644 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-04-22T01:18:37.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09180678c0 msgr2=0x7f0918069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09180678c0 0x7f0918069d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0924006f10 tx=0x7f0924008040 comp rx=0 tx=0).stop 2024-04-22T01:18:37.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 msgr2=0x7f0934155430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 0x7f0934155430 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f093000d800 tx=0x7f093000dcc0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 shutdown_connections 2024-04-22T01:18:37.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09180678c0 0x7f0918069d80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093415dfb0 0x7f093415ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09340cea80 0x7f0934155430 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 --2- 172.21.15.70:0/1865080155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09340b5730 0x7f09340c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 >> 172.21.15.70:0/1865080155 conn(0x7f093406c9b0 msgr2=0x7f09340d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:37.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 shutdown_connections 2024-04-22T01:18:37.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.644+0000 7f093c40e640 1 -- 172.21.15.70:0/1865080155 wait complete. 2024-04-22T01:18:37.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:37.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 Processor -- start 2024-04-22T01:18:37.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 -- start start 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efc600d0400 0x7efc600b1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc600b18c0 0x7efc600b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc6006cd10 con 0x7efc60161510 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efc601618f0 con 0x7efc600d0400 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc66070640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efc6006d320 con 0x7efc600b18c0 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efc600d0400 0x7efc600b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efc60161510 0x7efc600cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34988/0 (socket says 172.21.15.70:34988) 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5f7fe640 1 -- 172.21.15.70:0/2800720047 learned_addr learned my addr 172.21.15.70:0/2800720047 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5effd640 1 -- 172.21.15.70:0/2800720047 >> v2:172.21.15.98:3300/0 conn(0x7efc600b18c0 msgr2=0x7efc600b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5effd640 1 --2- 172.21.15.70:0/2800720047 >> v2:172.21.15.98:3300/0 conn(0x7efc600b18c0 0x7efc600b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5effd640 1 -- 172.21.15.70:0/2800720047 >> v1:172.21.15.70:6789/0 conn(0x7efc60161510 legacy=0x7efc600cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5effd640 1 -- 172.21.15.70:0/2800720047 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc600d0cc0 con 0x7efc600d0400 2024-04-22T01:18:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.768+0000 7efc5e7fc640 1 -- 172.21.15.70:0/2800720047 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179263183 0 0) 0x7efc6006cd10 con 0x7efc60161510 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5effd640 1 --2- 172.21.15.70:0/2800720047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600d0400 0x7efc600b1380 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7efc60073600 tx=0x7efc5001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fd1b16d60a6715 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5e7fc640 1 -- 172.21.15.70:0/2800720047 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc500b0070 con 0x7efc600d0400 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5e7fc640 1 -- 172.21.15.70:0/2800720047 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc500ab600 con 0x7efc600d0400 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5e7fc640 1 -- 172.21.15.70:0/2800720047 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc500ab9d0 con 0x7efc600d0400 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- 172.21.15.70:0/2800720047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600d0400 msgr2=0x7efc600b1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 --2- 172.21.15.70:0/2800720047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600d0400 0x7efc600b1380 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7efc60073600 tx=0x7efc5001c3f0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- 172.21.15.70:0/2800720047 shutdown_connections 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 --2- 172.21.15.70:0/2800720047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600d0400 0x7efc600b1380 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 --2- 172.21.15.70:0/2800720047 >> v2:172.21.15.98:3300/0 conn(0x7efc600b18c0 0x7efc600b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- 172.21.15.70:0/2800720047 >> 172.21.15.70:0/2800720047 conn(0x7efc600b35a0 msgr2=0x7efc600b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- 172.21.15.70:0/2800720047 shutdown_connections 2024-04-22T01:18:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- 172.21.15.70:0/2800720047 wait complete. 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 Processor -- start 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- start start 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 0x7efc600709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc600d0400 0x7efc600c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc60161510 0x7efc600c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc600b2b70 con 0x7efc600b18c0 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc6006d7b0 con 0x7efc60161510 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc66070640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efc600b2cf0 con 0x7efc600d0400 2024-04-22T01:18:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 0x7efc600709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 0x7efc600709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57726/0 (socket says 172.21.15.70:57726) 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 -- 172.21.15.70:0/482555988 learned_addr learned my addr 172.21.15.70:0/482555988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5effd640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc600d0400 0x7efc600c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5ffff640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc60161510 0x7efc600c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 -- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc600d0400 msgr2=0x7efc600c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc600d0400 0x7efc600c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 -- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc60161510 msgr2=0x7efc600c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc60161510 0x7efc600c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 -- 172.21.15.70:0/482555988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc600c7a60 con 0x7efc600b18c0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5effd640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc600d0400 0x7efc600c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.772+0000 7efc5f7fe640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 0x7efc600709c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7efc540011f0 tx=0x7efc5400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54018070 con 0x7efc600b18c0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc540074e0 con 0x7efc600b18c0 2024-04-22T01:18:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc60152be0 con 0x7efc600b18c0 2024-04-22T01:18:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54004a10 con 0x7efc600b18c0 2024-04-22T01:18:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc601531a0 con 0x7efc600b18c0 2024-04-22T01:18:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc6006e5e0 con 0x7efc600b18c0 2024-04-22T01:18:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc5401b070 con 0x7efc600b18c0 2024-04-22T01:18:37.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc47fff640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc4c0678c0 0x7efc4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:37.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7efc54085170 con 0x7efc600b18c0 2024-04-22T01:18:37.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc5effd640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc4c0678c0 0x7efc4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:37.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.776+0000 7efc5effd640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc4c0678c0 0x7efc4c069d80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efc500ad000 tx=0x7efc50008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:37.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.780+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7efc54055ab0 con 0x7efc600b18c0 2024-04-22T01:18:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.928+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc60068210 con 0x7efc600b18c0 2024-04-22T01:18:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc47fff640 1 -- 172.21.15.70:0/482555988 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+17042 (secure 0 0 0) 0x7efc540553b0 con 0x7efc600b18c0 2024-04-22T01:18:37.933 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:37.933 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:29.400245+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc4c0678c0 msgr2=0x7efc4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc4c0678c0 0x7efc4c069d80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efc500ad000 tx=0x7efc50008040 comp rx=0 tx=0).stop 2024-04-22T01:18:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 msgr2=0x7efc600709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 0x7efc600709c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7efc540011f0 tx=0x7efc5400c9b0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 shutdown_connections 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7efc4c0678c0 0x7efc4c069d80 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc60161510 0x7efc600c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc600d0400 0x7efc600c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 --2- 172.21.15.70:0/482555988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc600b18c0 0x7efc600709c0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 >> 172.21.15.70:0/482555988 conn(0x7efc600b35a0 msgr2=0x7efc600744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 shutdown_connections 2024-04-22T01:18:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:37.932+0000 7efc66070640 1 -- 172.21.15.70:0/482555988 wait complete. 2024-04-22T01:18:37.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:18:20.962202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:18:21.681268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:37.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.060+0000 7fbad9530640 1 Processor -- start 2024-04-22T01:18:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.060+0000 7fbad9530640 1 -- start start 2024-04-22T01:18:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad9530640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 0x7fbad415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad9530640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbad40ce920 0x7fbad4154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad9530640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad415f000 con 0x7fbad40ceda0 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad9530640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbad40b3800 con 0x7fbad40ce920 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad9530640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbad40b3ae0 con 0x7fbad415e6e0 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbad40ceda0 0x7fbad406d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35004/0 (socket says 172.21.15.70:35004) 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 -- 172.21.15.70:0/529276692 learned_addr learned my addr 172.21.15.70:0/529276692 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad2ffd640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 0x7fbad415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad27fc640 1 --2- 172.21.15.70:0/529276692 >> v2:172.21.15.98:3300/0 conn(0x7fbad40ce920 0x7fbad4154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563047334 0 0) 0x7fbad415f000 con 0x7fbad40ceda0 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbabc003530 con 0x7fbad40ceda0 2024-04-22T01:18:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3884500222 0 0) 0x7fbabc003530 con 0x7fbad40ceda0 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 msgr2=0x7fbad415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 0x7fbad415eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 >> v2:172.21.15.98:3300/0 conn(0x7fbad40ce920 msgr2=0x7fbad4154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 --2- 172.21.15.70:0/529276692 >> v2:172.21.15.98:3300/0 conn(0x7fbad40ce920 0x7fbad4154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad40cde40 con 0x7fbad40ceda0 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fbacc001680 con 0x7fbad40ceda0 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 >> v1:172.21.15.70:6789/0 conn(0x7fbad40ceda0 legacy=0x7fbad406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 0x7fbabc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc003f40 0x7fbabc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbabc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbad40b3800 con 0x7fbabc004d50 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbad40b3ae0 con 0x7fbabc003f40 2024-04-22T01:18:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc00b9d0 con 0x7fbad415e6e0 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbabc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad2ffd640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 0x7fbabc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad27fc640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc003f40 0x7fbabc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 -- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 msgr2=0x7fbabc0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415e6e0 0x7fbabc0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 -- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc003f40 msgr2=0x7fbabc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc003f40 0x7fbabc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 -- 172.21.15.70:0/529276692 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc003530 con 0x7fbabc004d50 2024-04-22T01:18:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad37fe640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbabc0051d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00c240 tx=0x7fbacc01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08372ba09b7be23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc007d40 con 0x7fbabc004d50 2024-04-22T01:18:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbacc0a9020 con 0x7fbabc004d50 2024-04-22T01:18:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.064+0000 7fbad1ffb640 1 -- 172.21.15.70:0/529276692 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc0a5ba0 con 0x7fbabc004d50 2024-04-22T01:18:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/529276692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 msgr2=0x7fbabc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 --2- 172.21.15.70:0/529276692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbabc0051d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00c240 tx=0x7fbacc01cca0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad37fe640 1 -- 172.21.15.70:0/529276692 reap_dead start 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/529276692 shutdown_connections 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/529276692 >> 172.21.15.70:0/529276692 conn(0x7fbad40b1d40 msgr2=0x7fbad40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/529276692 shutdown_connections 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/529276692 wait complete. 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 Processor -- start 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- start start 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbad41f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415dcd0 0x7fbad4152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad40b3f00 0x7fbad40b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbad406f0e0 con 0x7fbabc004d50 2024-04-22T01:18:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbad40ce400 con 0x7fbad40b3f00 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbad406f260 con 0x7fbad415dcd0 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbad41f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbad41f2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57736/0 (socket says 172.21.15.70:57736) 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 -- 172.21.15.70:0/262348205 learned_addr learned my addr 172.21.15.70:0/262348205 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad27fc640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415dcd0 0x7fbad4152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad37fe640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad40b3f00 0x7fbad40b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 -- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415dcd0 msgr2=0x7fbad4152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415dcd0 0x7fbad4152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 -- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad40b3f00 msgr2=0x7fbad40b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad40b3f00 0x7fbad40b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 -- 172.21.15.70:0/262348205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad41f3250 con 0x7fbabc004d50 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad37fe640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad40b3f00 0x7fbad40b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad2ffd640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbad41f2bc0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fbac4007f60 tx=0x7fbac4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac401a020 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac4016470 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad41f3530 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4016860 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad415e130 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.068+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad4068210 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.072+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbac401d020 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.072+0000 7fbabb7fe640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbac00678f0 0x7fbac0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.072+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7fbac4086f00 con 0x7fbabc004d50 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.072+0000 7fbad27fc640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbac00678f0 0x7fbac0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.072+0000 7fbad27fc640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbac00678f0 0x7fbac0069db0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbad40d0150 tx=0x7fbac8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.076+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbac4024030 con 0x7fbabc004d50 2024-04-22T01:18:38.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.236+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbad40d08e0 con 0x7fbabc004d50 2024-04-22T01:18:38.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.236+0000 7fbabb7fe640 1 -- 172.21.15.70:0/262348205 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fbac40578d0 con 0x7fbabc004d50 2024-04-22T01:18:38.238 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.236+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbac00678f0 msgr2=0x7fbac0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.236+0000 7fbad9530640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbac00678f0 0x7fbac0069db0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbad40d0150 tx=0x7fbac8008040 comp rx=0 tx=0).stop 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.236+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 msgr2=0x7fbad41f2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.236+0000 7fbad9530640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbad41f2bc0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fbac4007f60 tx=0x7fbac4007f90 comp rx=0 tx=0).stop 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 shutdown_connections 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbac00678f0 0x7fbac0069db0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbad40b3f00 0x7fbad40b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbad415dcd0 0x7fbad4152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 --2- 172.21.15.70:0/262348205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbabc004d50 0x7fbad41f2bc0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 >> 172.21.15.70:0/262348205 conn(0x7fbad40b1d40 msgr2=0x7fbad4156320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 shutdown_connections 2024-04-22T01:18:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.240+0000 7fbad9530640 1 -- 172.21.15.70:0/262348205 wait complete. 2024-04-22T01:18:38.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:18:38.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 Processor -- start 2024-04-22T01:18:38.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 -- start start 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a415de20 0x7f33a40b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33a40cea10 0x7f33a40ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a40cf3e0 con 0x7f33a40b6db0 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f33a40b75e0 con 0x7f33a40cea10 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33aa320640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33a40b78c0 con 0x7f33a415de20 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33a40cea10 0x7f33a40ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33a40cea10 0x7f33a40ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57744/0 (socket says 172.21.15.70:57744) 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 -- 172.21.15.70:0/3470398446 learned_addr learned my addr 172.21.15.70:0/3470398446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a3fff640 1 --2- 172.21.15.70:0/3470398446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a415de20 0x7f33a40b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 -- 172.21.15.70:0/3470398446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a415de20 msgr2=0x7f33a40b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 --2- 172.21.15.70:0/3470398446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a415de20 0x7f33a40b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 -- 172.21.15.70:0/3470398446 >> v1:172.21.15.98:6789/0 conn(0x7f33a40b6db0 legacy=0x7f33a40d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.380+0000 7f33a88bd640 1 -- 172.21.15.70:0/3470398446 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a40b7ba0 con 0x7f33a40cea10 2024-04-22T01:18:38.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 --2- 172.21.15.70:0/3470398446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a40cea10 0x7f33a40ceea0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f339c009e80 tx=0x7f339c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6786a417d2e527ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a2ffd640 1 -- 172.21.15.70:0/3470398446 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c0b4070 con 0x7f33a40cea10 2024-04-22T01:18:38.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a2ffd640 1 -- 172.21.15.70:0/3470398446 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f339c0af400 con 0x7f33a40cea10 2024-04-22T01:18:38.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a2ffd640 1 -- 172.21.15.70:0/3470398446 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c0af7d0 con 0x7f33a40cea10 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- 172.21.15.70:0/3470398446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a40cea10 msgr2=0x7f33a40ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 --2- 172.21.15.70:0/3470398446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a40cea10 0x7f33a40ceea0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f339c009e80 tx=0x7f339c01f1c0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- 172.21.15.70:0/3470398446 shutdown_connections 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 --2- 172.21.15.70:0/3470398446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a40cea10 0x7f33a40ceea0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 --2- 172.21.15.70:0/3470398446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a415de20 0x7f33a40b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- 172.21.15.70:0/3470398446 >> 172.21.15.70:0/3470398446 conn(0x7f33a4072d20 msgr2=0x7f33a4073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- 172.21.15.70:0/3470398446 shutdown_connections 2024-04-22T01:18:38.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- 172.21.15.70:0/3470398446 wait complete. 2024-04-22T01:18:38.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 Processor -- start 2024-04-22T01:18:38.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- start start 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a40b6db0 0x7f33a414b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a40cea10 0x7f33a41f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 0x7f33a415af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33a415e200 con 0x7f33a415de20 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33a40b8160 con 0x7f33a40cea10 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33aa320640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33a40b71f0 con 0x7f33a40b6db0 2024-04-22T01:18:38.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 0x7f33a415af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 0x7f33a415af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57750/0 (socket says 172.21.15.70:57750) 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 -- 172.21.15.70:0/3437891086 learned_addr learned my addr 172.21.15.70:0/3437891086 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a3fff640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a40b6db0 0x7f33a414b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 -- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a40b6db0 msgr2=0x7f33a414b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a40b6db0 0x7f33a414b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 -- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a40cea10 msgr2=0x7f33a41f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a37fe640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a40cea10 0x7f33a41f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a40cea10 0x7f33a41f3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 -- 172.21.15.70:0/3437891086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a414be90 con 0x7f33a415de20 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.384+0000 7f33a88bd640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 0x7f33a415af20 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f339c01d870 tx=0x7f339c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c0be020 con 0x7f33a415de20 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f339c0afe30 con 0x7f33a415de20 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a41f7270 con 0x7f33a415de20 2024-04-22T01:18:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c0b4040 con 0x7f33a415de20 2024-04-22T01:18:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a41f7830 con 0x7f33a415de20 2024-04-22T01:18:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a4068210 con 0x7f33a415de20 2024-04-22T01:18:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f339c0c1020 con 0x7f33a415de20 2024-04-22T01:18:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a0ff9640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33900678c0 0x7f3390069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f339c1243d0 con 0x7f33a415de20 2024-04-22T01:18:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a3fff640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33900678c0 0x7f3390069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.388+0000 7f33a3fff640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33900678c0 0x7f3390069d80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f339400b930 tx=0x7f3394008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.392+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f339c0c8030 con 0x7f33a415de20 2024-04-22T01:18:38.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33a40b9130 con 0x7f33a415de20 2024-04-22T01:18:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33a0ff9640 1 -- 172.21.15.70:0/3437891086 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f33a414be90 con 0x7f33a415de20 2024-04-22T01:18:38.541 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:18:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33900678c0 msgr2=0x7f3390069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33900678c0 0x7f3390069d80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f339400b930 tx=0x7f3394008040 comp rx=0 tx=0).stop 2024-04-22T01:18:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 msgr2=0x7f33a415af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 0x7f33a415af20 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f339c01d870 tx=0x7f339c01bf60 comp rx=0 tx=0).stop 2024-04-22T01:18:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 shutdown_connections 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33900678c0 0x7f3390069d80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33a415de20 0x7f33a415af20 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33a40cea10 0x7f33a41f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 --2- 172.21.15.70:0/3437891086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33a40b6db0 0x7f33a414b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 >> 172.21.15.70:0/3437891086 conn(0x7f33a4072d20 msgr2=0x7f33a40b84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 shutdown_connections 2024-04-22T01:18:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.540+0000 7f33aa320640 1 -- 172.21.15.70:0/3437891086 wait complete. 2024-04-22T01:18:38.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:18:38.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1dfca02640 1 Processor -- start 2024-04-22T01:18:38.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1dfca02640 1 -- start start 2024-04-22T01:18:38.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1dfca02640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1df80b98a0 0x7f1df80d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1dfca02640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df80b9ce0 con 0x7f1df815de20 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1dfca02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df80b75e0 con 0x7f1df80cea10 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1dfca02640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1df80b78c0 con 0x7f1df80b98a0 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1df815de20 0x7f1df80b6db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35018/0 (socket says 172.21.15.70:35018) 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df659c640 1 -- 172.21.15.70:0/2477168932 learned_addr learned my addr 172.21.15.70:0/2477168932 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df5d9b640 1 --2- 172.21.15.70:0/2477168932 >> v2:172.21.15.70:3300/0 conn(0x7f1df80b98a0 0x7f1df80d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df5d9b640 1 -- 172.21.15.70:0/2477168932 >> v1:172.21.15.98:6789/0 conn(0x7f1df80cea10 legacy=0x7f1df80cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df5d9b640 1 -- 172.21.15.70:0/2477168932 >> v1:172.21.15.70:6789/0 conn(0x7f1df815de20 legacy=0x7f1df80b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df5d9b640 1 -- 172.21.15.70:0/2477168932 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df80b7ba0 con 0x7f1df80b98a0 2024-04-22T01:18:38.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.676+0000 7f1df559a640 1 -- 172.21.15.70:0/2477168932 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373685684 0 0) 0x7f1df80b9ce0 con 0x7f1df815de20 2024-04-22T01:18:38.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df5d9b640 1 --2- 172.21.15.70:0/2477168932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df80d1d80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0097c0 tx=0x7f1dec01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6f621015612c2b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df559a640 1 -- 172.21.15.70:0/2477168932 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0b5070 con 0x7f1df80b98a0 2024-04-22T01:18:38.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df559a640 1 -- 172.21.15.70:0/2477168932 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0a8950 con 0x7f1df80b98a0 2024-04-22T01:18:38.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df559a640 1 -- 172.21.15.70:0/2477168932 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0a8d20 con 0x7f1df80b98a0 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- 172.21.15.70:0/2477168932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 msgr2=0x7f1df80d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 --2- 172.21.15.70:0/2477168932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df80d1d80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0097c0 tx=0x7f1dec01ec50 comp rx=0 tx=0).stop 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- 172.21.15.70:0/2477168932 shutdown_connections 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 --2- 172.21.15.70:0/2477168932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df80d1d80 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- 172.21.15.70:0/2477168932 >> 172.21.15.70:0/2477168932 conn(0x7f1df8072d20 msgr2=0x7f1df8073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- 172.21.15.70:0/2477168932 shutdown_connections 2024-04-22T01:18:38.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- 172.21.15.70:0/2477168932 wait complete. 2024-04-22T01:18:38.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 Processor -- start 2024-04-22T01:18:38.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- start start 2024-04-22T01:18:38.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df8158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cea10 0x7f1df80c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df815de20 0x7f1df8149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1df815e200 con 0x7f1df80b98a0 2024-04-22T01:18:38.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1df80b8160 con 0x7f1df815de20 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1dfca02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1df80d22c0 con 0x7f1df80cea10 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df8158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df8158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57766/0 (socket says 172.21.15.70:57766) 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 -- 172.21.15.70:0/3615777942 learned_addr learned my addr 172.21.15.70:0/3615777942 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df5d9b640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cea10 0x7f1df80c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df6d9d640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df815de20 0x7f1df8149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 -- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cea10 msgr2=0x7f1df80c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cea10 0x7f1df80c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 -- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df815de20 msgr2=0x7f1df8149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df815de20 0x7f1df8149fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 -- 172.21.15.70:0/3615777942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df80cb800 con 0x7f1df80b98a0 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df5d9b640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cea10 0x7f1df80c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.680+0000 7f1df659c640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df8158900 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f1de80011f0 tx=0x7f1de800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de8018070 con 0x7f1df80b98a0 2024-04-22T01:18:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1de80074e0 con 0x7f1df80b98a0 2024-04-22T01:18:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df80c7fe0 con 0x7f1df80b98a0 2024-04-22T01:18:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de8004a10 con 0x7f1df80b98a0 2024-04-22T01:18:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df80c85a0 con 0x7f1df80b98a0 2024-04-22T01:18:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df8068210 con 0x7f1df80b98a0 2024-04-22T01:18:38.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1de801b070 con 0x7f1df80b98a0 2024-04-22T01:18:38.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1de6ffd640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1dd80678c0 0x7f1dd8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f1de80851b0 con 0x7f1df80b98a0 2024-04-22T01:18:38.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1df5d9b640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1dd80678c0 0x7f1dd8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.684+0000 7f1df5d9b640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1dd80678c0 0x7f1dd8069d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0b2000 tx=0x7f1dec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.688+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1de8055af0 con 0x7f1df80b98a0 2024-04-22T01:18:38.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.852+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1df80ce400 con 0x7f1df80b98a0 2024-04-22T01:18:38.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.852+0000 7f1de6ffd640 1 -- 172.21.15.70:0/3615777942 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f1de80553f0 con 0x7f1df80b98a0 2024-04-22T01:18:38.856 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:18:38.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1dd80678c0 msgr2=0x7f1dd8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1dd80678c0 0x7f1dd8069d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0b2000 tx=0x7f1dec008040 comp rx=0 tx=0).stop 2024-04-22T01:18:38.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 msgr2=0x7f1df8158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df8158900 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f1de80011f0 tx=0x7f1de800c9b0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 shutdown_connections 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1dd80678c0 0x7f1dd8069d80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1df815de20 0x7f1df8149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1df80cea10 0x7f1df80c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 --2- 172.21.15.70:0/3615777942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1df80b98a0 0x7f1df8158900 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 >> 172.21.15.70:0/3615777942 conn(0x7f1df8072d20 msgr2=0x7f1df815d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 shutdown_connections 2024-04-22T01:18:38.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.856+0000 7f1dfca02640 1 -- 172.21.15.70:0/3615777942 wait complete. 2024-04-22T01:18:38.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:38.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.988+0000 7f09303a8640 1 Processor -- start 2024-04-22T01:18:38.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.988+0000 7f09303a8640 1 -- start start 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f09303a8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f09280aee40 0x7f09280b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f09303a8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f092815bb50 0x7f09280b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f09303a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f092806dbd0 con 0x7f092815aa30 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f09303a8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f092806de70 con 0x7f092815bb50 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f09303a8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f092806e150 con 0x7f09280aee40 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f092815bb50 0x7f09280b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f092815bb50 0x7f09280b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57772/0 (socket says 172.21.15.70:57772) 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 -- 172.21.15.70:0/3901267286 learned_addr learned my addr 172.21.15.70:0/3901267286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092d943640 1 --2- 172.21.15.70:0/3901267286 >> v2:172.21.15.98:3300/0 conn(0x7f09280aee40 0x7f09280b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:38.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 -- 172.21.15.70:0/3901267286 >> v2:172.21.15.98:3300/0 conn(0x7f09280aee40 msgr2=0x7f09280b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:38.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 --2- 172.21.15.70:0/3901267286 >> v2:172.21.15.98:3300/0 conn(0x7f09280aee40 0x7f09280b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 -- 172.21.15.70:0/3901267286 >> v1:172.21.15.98:6789/0 conn(0x7f092815aa30 legacy=0x7f09280ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 -- 172.21.15.70:0/3901267286 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092806e430 con 0x7f092815bb50 2024-04-22T01:18:38.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092e945640 1 --2- 172.21.15.70:0/3901267286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092815bb50 0x7f09280b28f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f09280d07b0 tx=0x7f092001ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b8a7557104eb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:38.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092d142640 1 -- 172.21.15.70:0/3901267286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09200a7ce0 con 0x7f092815bb50 2024-04-22T01:18:38.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092d142640 1 -- 172.21.15.70:0/3901267286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f092000b0d0 con 0x7f092815bb50 2024-04-22T01:18:38.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.992+0000 7f092d142640 1 -- 172.21.15.70:0/3901267286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920009600 con 0x7f092815bb50 2024-04-22T01:18:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/3901267286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092815bb50 msgr2=0x7f09280b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 --2- 172.21.15.70:0/3901267286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092815bb50 0x7f09280b28f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f09280d07b0 tx=0x7f092001ebf0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/3901267286 shutdown_connections 2024-04-22T01:18:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 --2- 172.21.15.70:0/3901267286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092815bb50 0x7f09280b28f0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 --2- 172.21.15.70:0/3901267286 >> v2:172.21.15.98:3300/0 conn(0x7f09280aee40 0x7f09280b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/3901267286 >> 172.21.15.70:0/3901267286 conn(0x7f092806b1f0 msgr2=0x7f092806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:38.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/3901267286 shutdown_connections 2024-04-22T01:18:38.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/3901267286 wait complete. 2024-04-22T01:18:39.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 Processor -- start 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- start start 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 0x7f0928150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092815aa30 0x7f0928155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f092815bb50 0x7f09281fabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f092806ead0 con 0x7f09280aee40 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f092806e950 con 0x7f092815aa30 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0928071b20 con 0x7f092815bb50 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 0x7f0928150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 0x7f0928150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57784/0 (socket says 172.21.15.70:57784) 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 -- 172.21.15.70:0/1207165549 learned_addr learned my addr 172.21.15.70:0/1207165549 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:39.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 -- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f092815bb50 msgr2=0x7f09281fabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e945640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f092815bb50 0x7f09281fabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f092815bb50 0x7f09281fabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092d943640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092815aa30 0x7f0928155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 -- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092815aa30 msgr2=0x7f0928155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092815aa30 0x7f0928155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 -- 172.21.15.70:0/1207165549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09281fb630 con 0x7f09280aee40 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e945640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f092815bb50 0x7f09281fabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f092e144640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 0x7f0928150b70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f092400c530 tx=0x7f092400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0924016020 con 0x7f09280aee40 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0924005220 con 0x7f09280aee40 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09281fb910 con 0x7f09280aee40 2024-04-22T01:18:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0924007030 con 0x7f09280aee40 2024-04-22T01:18:39.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0928154f20 con 0x7f09280aee40 2024-04-22T01:18:39.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:38.996+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0928154a50 con 0x7f09280aee40 2024-04-22T01:18:39.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.000+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f09240071d0 con 0x7f09280aee40 2024-04-22T01:18:39.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.000+0000 7f091effd640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.000+0000 7f092d943640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.000+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f0924085810 con 0x7f09280aee40 2024-04-22T01:18:39.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.000+0000 7f092d943640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0928155e30 tx=0x7f0918008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.004+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0924020020 con 0x7f09280aee40 2024-04-22T01:18:39.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09281fbae0 con 0x7f09280aee40 2024-04-22T01:18:39.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f091effd640 1 -- 172.21.15.70:0/1207165549 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7f09240074c0 con 0x7f09280aee40 2024-04-22T01:18:39.198 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:39.198 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:22.684405+0000","modified":"2024-04-22T01:18:24.723325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7215":{"gid":7215,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2337533615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2337533615},{"type":"v1","addr":"172.21.15.70:6837","nonce":2337533615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7215,"qdb_cluster":[7215]},"id":6}]} 2024-04-22T01:18:39.198 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-04-22T01:18:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 msgr2=0x7f090c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0928155e30 tx=0x7f0918008040 comp rx=0 tx=0).stop 2024-04-22T01:18:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 msgr2=0x7f0928150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 0x7f0928150b70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f092400c530 tx=0x7f092400c9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 shutdown_connections 2024-04-22T01:18:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f090c0678c0 0x7f090c069d80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f092815bb50 0x7f09281fabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092815aa30 0x7f0928155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 --2- 172.21.15.70:0/1207165549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09280aee40 0x7f0928150b70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 >> 172.21.15.70:0/1207165549 conn(0x7f092806b1f0 msgr2=0x7f092815d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:39.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 shutdown_connections 2024-04-22T01:18:39.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.196+0000 7f09303a8640 1 -- 172.21.15.70:0/1207165549 wait complete. 2024-04-22T01:18:39.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:39.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 Processor -- start 2024-04-22T01:18:39.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 -- start start 2024-04-22T01:18:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e2c15dfb0 0x7f2e2c0b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0cea80 0x7f2e2c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e2c0b5730 0x7f2e2c0b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2e2c0cf420 con 0x7f2e2c0b5730 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2e2c15e390 con 0x7f2e2c15dfb0 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e33810640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e2c161e00 con 0x7f2e2c0cea80 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e2c0b5730 0x7f2e2c0b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e2c0b5730 0x7f2e2c0b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57792/0 (socket says 172.21.15.70:57792) 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 -- 172.21.15.70:0/710230380 learned_addr learned my addr 172.21.15.70:0/710230380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e30dab640 1 --2- 172.21.15.70:0/710230380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0cea80 0x7f2e2c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e315ac640 1 --2- 172.21.15.70:0/710230380 >> v2:172.21.15.98:3300/0 conn(0x7f2e2c15dfb0 0x7f2e2c0b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 -- 172.21.15.70:0/710230380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0cea80 msgr2=0x7f2e2c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 --2- 172.21.15.70:0/710230380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0cea80 0x7f2e2c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 -- 172.21.15.70:0/710230380 >> v2:172.21.15.98:3300/0 conn(0x7f2e2c15dfb0 msgr2=0x7f2e2c0b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 --2- 172.21.15.70:0/710230380 >> v2:172.21.15.98:3300/0 conn(0x7f2e2c15dfb0 0x7f2e2c0b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 -- 172.21.15.70:0/710230380 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2c1620e0 con 0x7f2e2c0b5730 2024-04-22T01:18:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.336+0000 7f2e31dad640 1 --2- 172.21.15.70:0/710230380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0b5730 0x7f2e2c0b5b90 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0d4d80 tx=0x7f2e2401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c81670b75259b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e23fff640 1 -- 172.21.15.70:0/710230380 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e240a5780 con 0x7f2e2c0b5730 2024-04-22T01:18:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e23fff640 1 -- 172.21.15.70:0/710230380 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e240a5920 con 0x7f2e2c0b5730 2024-04-22T01:18:39.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e23fff640 1 -- 172.21.15.70:0/710230380 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e240a5cf0 con 0x7f2e2c0b5730 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- 172.21.15.70:0/710230380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0b5730 msgr2=0x7f2e2c0b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- 172.21.15.70:0/710230380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0b5730 0x7f2e2c0b5b90 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0d4d80 tx=0x7f2e2401c9f0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- 172.21.15.70:0/710230380 shutdown_connections 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- 172.21.15.70:0/710230380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0b5730 0x7f2e2c0b5b90 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- 172.21.15.70:0/710230380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0cea80 0x7f2e2c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- 172.21.15.70:0/710230380 >> v2:172.21.15.98:3300/0 conn(0x7f2e2c15dfb0 0x7f2e2c0b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- 172.21.15.70:0/710230380 >> 172.21.15.70:0/710230380 conn(0x7f2e2c06c9b0 msgr2=0x7f2e2c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- 172.21.15.70:0/710230380 shutdown_connections 2024-04-22T01:18:39.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- 172.21.15.70:0/710230380 wait complete. 2024-04-22T01:18:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 Processor -- start 2024-04-22T01:18:39.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- start start 2024-04-22T01:18:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0b5730 0x7f2e2c15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 0x7f2e2c153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2c15dfb0 0x7f2e2c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c161a90 con 0x7f2e2c0cea80 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c0b9b10 con 0x7f2e2c15dfb0 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e33810640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e2c0d2900 con 0x7f2e2c0b5730 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e30dab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 0x7f2e2c153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e30dab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 0x7f2e2c153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57804/0 (socket says 172.21.15.70:57804) 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e30dab640 1 -- 172.21.15.70:0/324533717 learned_addr learned my addr 172.21.15.70:0/324533717 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.340+0000 7f2e315ac640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0b5730 0x7f2e2c15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e31dad640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2c15dfb0 0x7f2e2c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e30dab640 1 -- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0b5730 msgr2=0x7f2e2c15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e30dab640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0b5730 0x7f2e2c15ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e30dab640 1 -- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2c15dfb0 msgr2=0x7f2e2c1544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e30dab640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2c15dfb0 0x7f2e2c1544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e30dab640 1 -- 172.21.15.70:0/324533717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2c1fb370 con 0x7f2e2c0cea80 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e30dab640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 0x7f2e2c153f70 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00d830 tx=0x7f2e1c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c018020 con 0x7f2e2c0cea80 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e1c0096c0 con 0x7f2e2c0cea80 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c009ab0 con 0x7f2e2c0cea80 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2c15ea70 con 0x7f2e2c0cea80 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e2c15f030 con 0x7f2e2c0cea80 2024-04-22T01:18:39.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e2c068210 con 0x7f2e2c0cea80 2024-04-22T01:18:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e1c01b020 con 0x7f2e2c0cea80 2024-04-22T01:18:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e21ffb640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e100678c0 0x7f2e10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f2e1c0866b0 con 0x7f2e2c0cea80 2024-04-22T01:18:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e315ac640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e100678c0 0x7f2e10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.344+0000 7f2e315ac640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e100678c0 0x7f2e10069d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800a8d0 tx=0x7f2e28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.348+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2e1c022020 con 0x7f2e2c0cea80 2024-04-22T01:18:39.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.512+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e2c0d1950 con 0x7f2e2c0cea80 2024-04-22T01:18:39.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.512+0000 7f2e21ffb640 1 -- 172.21.15.70:0/324533717 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+17042 (secure 0 0 0) 0x7f2e1c011340 con 0x7f2e2c0cea80 2024-04-22T01:18:39.518 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:39.518 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:29.400245+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:39.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e100678c0 msgr2=0x7f2e10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e100678c0 0x7f2e10069d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800a8d0 tx=0x7f2e28008040 comp rx=0 tx=0).stop 2024-04-22T01:18:39.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 msgr2=0x7f2e2c153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 0x7f2e2c153f70 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00d830 tx=0x7f2e1c00dcf0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 shutdown_connections 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e100678c0 0x7f2e10069d80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2c15dfb0 0x7f2e2c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e2c0cea80 0x7f2e2c153f70 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 --2- 172.21.15.70:0/324533717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e2c0b5730 0x7f2e2c15ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 >> 172.21.15.70:0/324533717 conn(0x7f2e2c06c9b0 msgr2=0x7f2e2c15ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 shutdown_connections 2024-04-22T01:18:39.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.516+0000 7f2e33810640 1 -- 172.21.15.70:0/324533717 wait complete. 2024-04-22T01:18:39.535 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:18:39.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:39.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 Processor -- start 2024-04-22T01:18:39.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 -- start start 2024-04-22T01:18:39.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71a00cd740 0x7f71a006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71a006b730 0x7f71a00d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71a006bb50 con 0x7f71a015bc70 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f71a00cfd10 con 0x7f71a00cd740 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a72a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f71a015c050 con 0x7f71a006b730 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a5042640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f71a015bc70 0x7f71a015f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35030/0 (socket says 172.21.15.70:35030) 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a5042640 1 -- 172.21.15.70:0/915758507 learned_addr learned my addr 172.21.15.70:0/915758507 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3811586733 0 0) 0x7f71a006bb50 con 0x7f71a015bc70 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f718c003530 con 0x7f71a015bc70 2024-04-22T01:18:39.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a5843640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71a006b730 0x7f71a00d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f71a4841640 1 --2- 172.21.15.70:0/915758507 >> v2:172.21.15.98:3300/0 conn(0x7f71a00cd740 0x7f71a006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1092658301 0 0) 0x7f718c003530 con 0x7f71a015bc70 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71a006b730 msgr2=0x7f71a00d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71a006b730 0x7f71a00d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 >> v2:172.21.15.98:3300/0 conn(0x7f71a00cd740 msgr2=0x7f71a006b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.652+0000 7f7197fff640 1 --2- 172.21.15.70:0/915758507 >> v2:172.21.15.98:3300/0 conn(0x7f71a00cd740 0x7f71a006b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a015ff90 con 0x7f71a015bc70 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f719c0016a0 con 0x7f71a015bc70 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 >> v1:172.21.15.70:6789/0 conn(0x7f71a015bc70 legacy=0x7f71a015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c0039c0 0x7f718c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f718c004810 0x7f718c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f718c0051d0 0x7f718c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f71a00cfd10 con 0x7f718c004810 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71a015c050 con 0x7f718c0051d0 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f718c00c010 con 0x7f718c0039c0 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a5843640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71a006b730 0x7f71a00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f718c004810 0x7f718c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 -- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c0039c0 msgr2=0x7f718c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c0039c0 0x7f718c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 -- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f718c0051d0 msgr2=0x7f718c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f718c0051d0 0x7f718c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 -- 172.21.15.70:0/915758507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718c003530 con 0x7f718c004810 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a4841640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f718c004810 0x7f718c004c90 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f719000ba60 tx=0x7f719001c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf243df06c2f5dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719000b600 con 0x7f718c004810 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71900aa630 con 0x7f718c004810 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f7197fff640 1 -- 172.21.15.70:0/915758507 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71900aaa20 con 0x7f718c004810 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a72a6640 1 -- 172.21.15.70:0/915758507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f718c004810 msgr2=0x7f718c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a72a6640 1 --2- 172.21.15.70:0/915758507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f718c004810 0x7f718c004c90 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f719000ba60 tx=0x7f719001c6c0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a5843640 1 -- 172.21.15.70:0/915758507 reap_dead start 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a72a6640 1 -- 172.21.15.70:0/915758507 shutdown_connections 2024-04-22T01:18:39.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a72a6640 1 -- 172.21.15.70:0/915758507 >> 172.21.15.70:0/915758507 conn(0x7f71a0099a60 msgr2=0x7f71a0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a72a6640 1 -- 172.21.15.70:0/915758507 shutdown_connections 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.656+0000 7f71a72a6640 1 -- 172.21.15.70:0/915758507 wait complete. 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 Processor -- start 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- start start 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c004810 0x7f71a00c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71a00c7ca0 0x7f71a00b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 0x7f71a00ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f71a00cff60 con 0x7f71a00c6810 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71a0160550 con 0x7f71a00c7ca0 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f71a00d0480 con 0x7f718c004810 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 0x7f71a00ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 0x7f71a00ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57822/0 (socket says 172.21.15.70:57822) 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 -- 172.21.15.70:0/583563064 learned_addr learned my addr 172.21.15.70:0/583563064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5042640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c004810 0x7f71a00c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a4841640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71a00c7ca0 0x7f71a00b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 -- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c004810 msgr2=0x7f71a00c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c004810 0x7f71a00c8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 -- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71a00c7ca0 msgr2=0x7f71a00b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71a00c7ca0 0x7f71a00b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 -- 172.21.15.70:0/583563064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a00c8120 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5843640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 0x7f71a00ca120 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f71a0073950 tx=0x7f719800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719801a070 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719800dcd0 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a006ecf0 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198003be0 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a006ef90 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a0068920 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f719801d070 con 0x7f71a00c6810 2024-04-22T01:18:39.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f7195ffb640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71840678c0 0x7f7184069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:39.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f7198085e90 con 0x7f71a00c6810 2024-04-22T01:18:39.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.660+0000 7f71a5042640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71840678c0 0x7f7184069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:39.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.664+0000 7f71a5042640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71840678c0 0x7f7184069d80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f719c000c10 tx=0x7f719c0015a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:39.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.668+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7198010070 con 0x7f71a00c6810 2024-04-22T01:18:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.864+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71a015de40 con 0x7f71a00c6810 2024-04-22T01:18:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.864+0000 7f7195ffb640 1 -- 172.21.15.70:0/583563064 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7f71980567d0 con 0x7f71a00c6810 2024-04-22T01:18:39.866 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:39.867 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:22.684405+0000","modified":"2024-04-22T01:18:24.723325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7215":{"gid":7215,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2337533615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2337533615},{"type":"v1","addr":"172.21.15.70:6837","nonce":2337533615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7215,"qdb_cluster":[7215]},"id":6}]} 2024-04-22T01:18:39.867 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71840678c0 msgr2=0x7f7184069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71840678c0 0x7f7184069d80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f719c000c10 tx=0x7f719c0015a0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 msgr2=0x7f71a00ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 0x7f71a00ca120 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f71a0073950 tx=0x7f719800d4a0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 shutdown_connections 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f71840678c0 0x7f7184069d80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71a00c6810 0x7f71a00ca120 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71a00c7ca0 0x7f71a00b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 --2- 172.21.15.70:0/583563064 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f718c004810 0x7f71a00c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 >> 172.21.15.70:0/583563064 conn(0x7f71a0099a60 msgr2=0x7f71a00cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 shutdown_connections 2024-04-22T01:18:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:39.868+0000 7f71a72a6640 1 -- 172.21.15.70:0/583563064 wait complete. 2024-04-22T01:18:39.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:40.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.004+0000 7ff5bc01b640 1 Processor -- start 2024-04-22T01:18:40.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.004+0000 7ff5bc01b640 1 -- start start 2024-04-22T01:18:40.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5bc01b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff5b41455b0 0x7ff5b40cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5bc01b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b40d0250 con 0x7ff5b40cc150 2024-04-22T01:18:40.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5bc01b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b40cd7b0 con 0x7ff5b4072230 2024-04-22T01:18:40.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5bc01b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff5b40cc710 con 0x7ff5b41455b0 2024-04-22T01:18:40.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b9db7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff5b41455b0 0x7ff5b40cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b95b6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff5b40cc150 0x7ff5b4071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35032/0 (socket says 172.21.15.70:35032) 2024-04-22T01:18:40.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b95b6640 1 -- 172.21.15.70:0/774943036 learned_addr learned my addr 172.21.15.70:0/774943036 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:40.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b9db7640 1 -- 172.21.15.70:0/774943036 >> v1:172.21.15.98:6789/0 conn(0x7ff5b4072230 legacy=0x7ff5b406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:40.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b9db7640 1 -- 172.21.15.70:0/774943036 >> v1:172.21.15.70:6789/0 conn(0x7ff5b40cc150 legacy=0x7ff5b4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b9db7640 1 -- 172.21.15.70:0/774943036 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b40cc9f0 con 0x7ff5b41455b0 2024-04-22T01:18:40.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b9db7640 1 --2- 172.21.15.70:0/774943036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b40cfd10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00097c0 tx=0x7ff5b001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edca57294727006 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b8db5640 1 -- 172.21.15.70:0/774943036 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b00b0070 con 0x7ff5b41455b0 2024-04-22T01:18:40.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b8db5640 1 -- 172.21.15.70:0/774943036 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5b00ab600 con 0x7ff5b41455b0 2024-04-22T01:18:40.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.008+0000 7ff5b8db5640 1 -- 172.21.15.70:0/774943036 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b00ab9d0 con 0x7ff5b41455b0 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/774943036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 msgr2=0x7ff5b40cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/774943036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b40cfd10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00097c0 tx=0x7ff5b001c340 comp rx=0 tx=0).stop 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/774943036 shutdown_connections 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/774943036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b40cfd10 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/774943036 >> 172.21.15.70:0/774943036 conn(0x7ff5b406b1f0 msgr2=0x7ff5b406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/774943036 shutdown_connections 2024-04-22T01:18:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/774943036 wait complete. 2024-04-22T01:18:40.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 Processor -- start 2024-04-22T01:18:40.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- start start 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5b4072230 0x7ff5b4158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b40cc150 0x7ff5b4159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b415c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b4169ee0 con 0x7ff5b41455b0 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b4169d60 con 0x7ff5b40cc150 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b416a060 con 0x7ff5b4072230 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b415c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b415c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57830/0 (socket says 172.21.15.70:57830) 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 -- 172.21.15.70:0/3958937679 learned_addr learned my addr 172.21.15.70:0/3958937679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:40.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5b9db7640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5b4072230 0x7ff5b4158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5b95b6640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b40cc150 0x7ff5b4159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 -- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5b4072230 msgr2=0x7ff5b4158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5b4072230 0x7ff5b4158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 -- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b40cc150 msgr2=0x7ff5b4159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b40cc150 0x7ff5b4159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 -- 172.21.15.70:0/3958937679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b4144e10 con 0x7ff5b41455b0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5b95b6640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b40cc150 0x7ff5b4159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5ba5b8640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b415c5b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c780 tx=0x7ff5ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac004b70 con 0x7ff5b41455b0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ac004d10 con 0x7ff5b41455b0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00aa90 con 0x7ff5b41455b0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b41387d0 con 0x7ff5b41455b0 2024-04-22T01:18:40.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.012+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b4138d50 con 0x7ff5b41455b0 2024-04-22T01:18:40.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.016+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b4068210 con 0x7ff5b41455b0 2024-04-22T01:18:40.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.016+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff5ac00ac30 con 0x7ff5b41455b0 2024-04-22T01:18:40.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.016+0000 7ff5aa7fc640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5980678c0 0x7ff598069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.016+0000 7ff5b9db7640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5980678c0 0x7ff598069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.016+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7ff5ac0860a0 con 0x7ff5b41455b0 2024-04-22T01:18:40.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.016+0000 7ff5b9db7640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5980678c0 0x7ff598069d80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00ad000 tx=0x7ff5b00a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.020+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff5ac020070 con 0x7ff5b41455b0 2024-04-22T01:18:40.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.212+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5b415f510 con 0x7ff5b41455b0 2024-04-22T01:18:40.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5aa7fc640 1 -- 172.21.15.70:0/3958937679 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5180 (secure 0 0 0) 0x7ff5ac056a70 con 0x7ff5b41455b0 2024-04-22T01:18:40.217 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:40.217 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:22.684405+0000","modified":"2024-04-22T01:18:24.723325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7215":{"gid":7215,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/2337533615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2337533615},{"type":"v1","addr":"172.21.15.70:6837","nonce":2337533615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7215,"qdb_cluster":[7215]},"id":6}]} 2024-04-22T01:18:40.217 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 44 2024-04-22T01:18:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5980678c0 msgr2=0x7ff598069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5980678c0 0x7ff598069d80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00ad000 tx=0x7ff5b00a6000 comp rx=0 tx=0).stop 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 msgr2=0x7ff5b415c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b415c5b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c780 tx=0x7ff5ac00cc40 comp rx=0 tx=0).stop 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 shutdown_connections 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff5980678c0 0x7ff598069d80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5b41455b0 0x7ff5b415c5b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b40cc150 0x7ff5b4159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 --2- 172.21.15.70:0/3958937679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5b4072230 0x7ff5b4158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 >> 172.21.15.70:0/3958937679 conn(0x7ff5b406b1f0 msgr2=0x7ff5b40b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 shutdown_connections 2024-04-22T01:18:40.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.216+0000 7ff5bc01b640 1 -- 172.21.15.70:0/3958937679 wait complete. 2024-04-22T01:18:40.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd60dcd9640 1 Processor -- start 2024-04-22T01:18:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd60dcd9640 1 -- start start 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd60dcd9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 0x7fd608166b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd60dcd9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6080cd7b0 con 0x7fd6080cc150 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd60dcd9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6080cc710 con 0x7fd6081455b0 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd60dcd9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6080cc9f0 con 0x7fd608072230 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd6080cc150 0x7fd608071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35046/0 (socket says 172.21.15.70:35046) 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 -- 172.21.15.70:0/3876355773 learned_addr learned my addr 172.21.15.70:0/3876355773 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd607fff640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 0x7fd608166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603992878 0 0) 0x7fd6080cd7b0 con 0x7fd6080cc150 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f0003530 con 0x7fd6080cc150 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3403391703 0 0) 0x7fd5f0003530 con 0x7fd6080cc150 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 msgr2=0x7fd608166b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 0x7fd608166b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 >> v1:172.21.15.98:6789/0 conn(0x7fd6081455b0 legacy=0x7fd6080cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080cccd0 con 0x7fd6080cc150 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fd5fc001680 con 0x7fd6080cc150 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 >> v1:172.21.15.70:6789/0 conn(0x7fd6080cc150 legacy=0x7fd608071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5f00039f0 0x7fd5f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f0004810 0x7fd5f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 0x7fd5f00052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080cc9f0 con 0x7fd5f0004810 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f000bbc0 con 0x7fd5f00039f0 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f000bdc0 con 0x7fd608072230 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f0004810 0x7fd5f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd6077fe640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5f00039f0 0x7fd5f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 -- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 msgr2=0x7fd5f00052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:40.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd608072230 0x7fd5f00052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 -- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5f00039f0 msgr2=0x7fd5f0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:40.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5f00039f0 0x7fd5f0003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.356+0000 7fd606ffd640 1 -- 172.21.15.70:0/3876355773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f0003530 con 0x7fd5f0004810 2024-04-22T01:18:40.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd606ffd640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f0004810 0x7fd5f0004c90 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c240 tx=0x7fd5fc01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1ae1cf6d5322f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc007d20 con 0x7fd5f0004810 2024-04-22T01:18:40.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc0a9070 con 0x7fd5f0004810 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd6067fc640 1 -- 172.21.15.70:0/3876355773 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc005d00 con 0x7fd5f0004810 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3876355773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f0004810 msgr2=0x7fd5f0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3876355773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f0004810 0x7fd5f0004c90 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c240 tx=0x7fd5fc01cc80 comp rx=0 tx=0).stop 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd607fff640 1 -- 172.21.15.70:0/3876355773 reap_dead start 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3876355773 shutdown_connections 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3876355773 >> 172.21.15.70:0/3876355773 conn(0x7fd60806b1f0 msgr2=0x7fd60806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3876355773 shutdown_connections 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3876355773 wait complete. 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 Processor -- start 2024-04-22T01:18:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- start start 2024-04-22T01:18:40.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd5f0004810 0x7fd6081f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd60815de00 0x7fd60813d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 0x7fd60813d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081676d0 con 0x7fd60815e280 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd608167550 con 0x7fd60815de00 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.360+0000 7fd60dcd9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd608167850 con 0x7fd5f0004810 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 0x7fd60813d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 0x7fd60813d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57848/0 (socket says 172.21.15.70:57848) 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 -- 172.21.15.70:0/3769691737 learned_addr learned my addr 172.21.15.70:0/3769691737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd606ffd640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd60815de00 0x7fd60813d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd6077fe640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd5f0004810 0x7fd6081f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 -- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd5f0004810 msgr2=0x7fd6081f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd5f0004810 0x7fd6081f53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 -- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd60815de00 msgr2=0x7fd60813d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd60815de00 0x7fd60813d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 -- 172.21.15.70:0/3769691737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd608149bc0 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd607fff640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 0x7fd60813d620 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd6080cbd40 tx=0x7fd60000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60000bce0 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60001a070 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6081498e0 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600014640 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6081f4bc0 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd608068210 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd600013070 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd5effff640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd5f40678c0 0x7fd5f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7fd600086080 con 0x7fd60815e280 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd6077fe640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd5f40678c0 0x7fd5f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.364+0000 7fd6077fe640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd5f40678c0 0x7fd5f4069d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8001660 tx=0x7fd5f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.368+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd6000569c0 con 0x7fd60815e280 2024-04-22T01:18:40.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.532+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd60815f510 con 0x7fd60815e280 2024-04-22T01:18:40.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.536+0000 7fd5effff640 1 -- 172.21.15.70:0/3769691737 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+17042 (secure 0 0 0) 0x7fd6000562c0 con 0x7fd60815e280 2024-04-22T01:18:40.539 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:40.539 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:29.400245+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:20.962202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:21.681268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.24","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:40.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.536+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd5f40678c0 msgr2=0x7fd5f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.536+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd5f40678c0 0x7fd5f4069d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8001660 tx=0x7fd5f8009040 comp rx=0 tx=0).stop 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.536+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 msgr2=0x7fd60813d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.536+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 0x7fd60813d620 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd6080cbd40 tx=0x7fd60000b480 comp rx=0 tx=0).stop 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 shutdown_connections 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd5f40678c0 0x7fd5f4069d80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd60815e280 0x7fd60813d620 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd60815de00 0x7fd60813d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 --2- 172.21.15.70:0/3769691737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd5f0004810 0x7fd6081f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 >> 172.21.15.70:0/3769691737 conn(0x7fd60806b1f0 msgr2=0x7fd608144810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 shutdown_connections 2024-04-22T01:18:40.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.540+0000 7fd60dcd9640 1 -- 172.21.15.70:0/3769691737 wait complete. 2024-04-22T01:18:40.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:18:40.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 Processor -- start 2024-04-22T01:18:40.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 -- start start 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3840b12f0 0x7fd384099c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b1750 0x7fd38409a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3840cc890 con 0x7fd38415f460 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd3840d0860 con 0x7fd3840b12f0 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd38b5b7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3840cd1c0 con 0x7fd3840b1750 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd388b52640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3840b12f0 0x7fd384099c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd389353640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd38415f460 0x7fd3840d0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35054/0 (socket says 172.21.15.70:35054) 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd388b52640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd3840b12f0 0x7fd384099c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57862/0 (socket says 172.21.15.70:57862) 2024-04-22T01:18:40.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd389353640 1 -- 172.21.15.70:0/2754994097 learned_addr learned my addr 172.21.15.70:0/2754994097 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd389b54640 1 --2- 172.21.15.70:0/2754994097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b1750 0x7fd38409a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd388b52640 1 -- 172.21.15.70:0/2754994097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b1750 msgr2=0x7fd38409a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd388b52640 1 --2- 172.21.15.70:0/2754994097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b1750 0x7fd38409a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd388b52640 1 -- 172.21.15.70:0/2754994097 >> v1:172.21.15.70:6789/0 conn(0x7fd38415f460 legacy=0x7fd3840d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd388b52640 1 -- 172.21.15.70:0/2754994097 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3840ccb70 con 0x7fd3840b12f0 2024-04-22T01:18:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.672+0000 7fd37bfff640 1 -- 172.21.15.70:0/2754994097 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806445881 0 0) 0x7fd3840cc890 con 0x7fd38415f460 2024-04-22T01:18:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd388b52640 1 --2- 172.21.15.70:0/2754994097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840b12f0 0x7fd384099c50 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd3740097c0 tx=0x7fd37401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eead003c8524cdf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd37bfff640 1 -- 172.21.15.70:0/2754994097 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740b0070 con 0x7fd3840b12f0 2024-04-22T01:18:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd37bfff640 1 -- 172.21.15.70:0/2754994097 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3740ab600 con 0x7fd3840b12f0 2024-04-22T01:18:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd37bfff640 1 -- 172.21.15.70:0/2754994097 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740ab9d0 con 0x7fd3840b12f0 2024-04-22T01:18:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- 172.21.15.70:0/2754994097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840b12f0 msgr2=0x7fd384099c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/2754994097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840b12f0 0x7fd384099c50 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd3740097c0 tx=0x7fd37401c4a0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- 172.21.15.70:0/2754994097 shutdown_connections 2024-04-22T01:18:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/2754994097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b1750 0x7fd38409a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/2754994097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3840b12f0 0x7fd384099c50 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- 172.21.15.70:0/2754994097 >> 172.21.15.70:0/2754994097 conn(0x7fd3840b3660 msgr2=0x7fd3840b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- 172.21.15.70:0/2754994097 shutdown_connections 2024-04-22T01:18:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- 172.21.15.70:0/2754994097 wait complete. 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 Processor -- start 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- start start 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b12f0 0x7fd384148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840b1750 0x7fd38409eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 0x7fd384070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd384168da0 con 0x7fd38415f460 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd384168c20 con 0x7fd3840b1750 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd38b5b7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd384168f20 con 0x7fd3840b12f0 2024-04-22T01:18:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd389b54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 0x7fd384070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd389b54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 0x7fd384070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57872/0 (socket says 172.21.15.70:57872) 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd389b54640 1 -- 172.21.15.70:0/567487698 learned_addr learned my addr 172.21.15.70:0/567487698 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.676+0000 7fd389353640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b12f0 0x7fd384148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd388b52640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840b1750 0x7fd38409eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389b54640 1 -- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b12f0 msgr2=0x7fd384148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389b54640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b12f0 0x7fd384148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389b54640 1 -- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840b1750 msgr2=0x7fd38409eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389b54640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840b1750 0x7fd38409eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389b54640 1 -- 172.21.15.70:0/567487698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38409d7c0 con 0x7fd38415f460 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389b54640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 0x7fd384070290 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009fd0 tx=0x7fd37c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c014420 con 0x7fd38415f460 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c019070 con 0x7fd38415f460 2024-04-22T01:18:40.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd384158510 con 0x7fd38415f460 2024-04-22T01:18:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c014800 con 0x7fd38415f460 2024-04-22T01:18:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38409e020 con 0x7fd38415f460 2024-04-22T01:18:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38404f510 con 0x7fd38415f460 2024-04-22T01:18:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd37c0149a0 con 0x7fd38415f460 2024-04-22T01:18:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd379ffb640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389353640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7fd37c0866f0 con 0x7fd38415f460 2024-04-22T01:18:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.680+0000 7fd389353640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd380000f70 tx=0x7fd380007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:40.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.684+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd37c057030 con 0x7fd38415f460 2024-04-22T01:18:40.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.876+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd38409a780 con 0x7fd38415f460 2024-04-22T01:18:40.878 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:18:40.876+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:18:40.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.880+0000 7fd379ffb640 1 -- 172.21.15.70:0/567487698 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7fd37c056930 con 0x7fd38415f460 2024-04-22T01:18:40.881 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:18:40.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.880+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 msgr2=0x7fd368069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.880+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd380000f70 tx=0x7fd380007b60 comp rx=0 tx=0).stop 2024-04-22T01:18:40.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.880+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 msgr2=0x7fd384070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:40.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.880+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 0x7fd384070290 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009fd0 tx=0x7fd37c00e4b0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 shutdown_connections 2024-04-22T01:18:40.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd38415f460 0x7fd384070290 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3840b1750 0x7fd38409eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 --2- 172.21.15.70:0/567487698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3840b12f0 0x7fd384148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:40.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 >> 172.21.15.70:0/567487698 conn(0x7fd3840b3660 msgr2=0x7fd3840cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:40.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 shutdown_connections 2024-04-22T01:18:40.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:40.884+0000 7fd38b5b7640 1 -- 172.21.15.70:0/567487698 wait complete. 2024-04-22T01:18:40.893 INFO:tasks.ceph.mds.a.smithi070.stderr: -1968> 2024-04-22T01:18:29.296+0000 7fe0e8cf2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:18:40.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:18:40.931 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:18:41.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 Processor -- start 2024-04-22T01:18:41.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 -- start start 2024-04-22T01:18:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d84071b70 0x7f3d84072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d840cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8406da70 con 0x7f3d8415a320 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3d8406f860 con 0x7f3d84071b70 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d88a38640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3d8406dd50 con 0x7f3d840cd410 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d8259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3d8415a320 0x7f3d8406f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35080/0 (socket says 172.21.15.70:35080) 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d8259c640 1 -- 172.21.15.70:0/2462458027 learned_addr learned my addr 172.21.15.70:0/2462458027 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d82d9d640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d840cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649279316 0 0) 0x7f3d8406da70 con 0x7f3d8415a320 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d70003530 con 0x7f3d8415a320 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d7bfff640 1 --2- 172.21.15.70:0/2462458027 >> v2:172.21.15.98:3300/0 conn(0x7f3d84071b70 0x7f3d84072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 428852443 0 0) 0x7f3d70003530 con 0x7f3d8415a320 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 msgr2=0x7f3d840cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d840cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 >> v2:172.21.15.98:3300/0 conn(0x7f3d84071b70 msgr2=0x7f3d84072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 --2- 172.21.15.70:0/2462458027 >> v2:172.21.15.98:3300/0 conn(0x7f3d84071b70 0x7f3d84072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8406e030 con 0x7f3d8415a320 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f3d74001680 con 0x7f3d8415a320 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d82d9d640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d840cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 >> v1:172.21.15.70:6789/0 conn(0x7f3d8415a320 legacy=0x7f3d8406f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d70003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d70004810 0x7f3d70004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8406f860 con 0x7f3d700039c0 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8406dd50 con 0x7f3d70004810 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7000dbe0 con 0x7f3d840cd410 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.020+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d70003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d7bfff640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d70004810 0x7f3d70004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d82d9d640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d8259c640 1 -- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 msgr2=0x7f3d700051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d840cd410 0x7f3d700051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d8259c640 1 -- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d70004810 msgr2=0x7f3d70004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d70004810 0x7f3d70004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d8259c640 1 -- 172.21.15.70:0/2462458027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d70003530 con 0x7f3d700039c0 2024-04-22T01:18:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d7bfff640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d70004810 0x7f3d70004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d70003de0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400c270 tx=0x7f3d7401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44737dde62cc48fb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74007d60 con 0x7f3d700039c0 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d740a9020 con 0x7f3d700039c0 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d81d9b640 1 -- 172.21.15.70:0/2462458027 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74005ae0 con 0x7f3d700039c0 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d88a38640 1 -- 172.21.15.70:0/2462458027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 msgr2=0x7f3d70003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2462458027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d70003de0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400c270 tx=0x7f3d7401ccc0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d82d9d640 1 -- 172.21.15.70:0/2462458027 reap_dead start 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d88a38640 1 -- 172.21.15.70:0/2462458027 shutdown_connections 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d88a38640 1 -- 172.21.15.70:0/2462458027 >> 172.21.15.70:0/2462458027 conn(0x7f3d840b1d40 msgr2=0x7f3d840cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d88a38640 1 -- 172.21.15.70:0/2462458027 shutdown_connections 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.024+0000 7f3d88a38640 1 -- 172.21.15.70:0/2462458027 wait complete. 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 Processor -- start 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- start start 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d84142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8415a960 0x7f3d84147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d84147890 0x7f3d8414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d840cf5a0 con 0x7f3d700039c0 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d840cf420 con 0x7f3d8415a960 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3d840cd820 con 0x7f3d84147890 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d84142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d84142d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57912/0 (socket says 172.21.15.70:57912) 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 -- 172.21.15.70:0/2344209013 learned_addr learned my addr 172.21.15.70:0/2344209013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d82d9d640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d84147890 0x7f3d8414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d7bfff640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8415a960 0x7f3d84147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 -- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d84147890 msgr2=0x7f3d8414e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d84147890 0x7f3d8414e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 -- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8415a960 msgr2=0x7f3d84147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8415a960 0x7f3d84147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 -- 172.21.15.70:0/2344209013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8414eed0 con 0x7f3d700039c0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d8259c640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d84142d70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400c190 tx=0x7f3d70005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74004ab0 con 0x7f3d700039c0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d74004c50 con 0x7f3d700039c0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d740070d0 con 0x7f3d700039c0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d84157920 con 0x7f3d700039c0 2024-04-22T01:18:41.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d84157ee0 con 0x7f3d700039c0 2024-04-22T01:18:41.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3d74007270 con 0x7f3d700039c0 2024-04-22T01:18:41.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.028+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d84147ca0 con 0x7f3d700039c0 2024-04-22T01:18:41.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.032+0000 7f3d7affd640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d640678f0 0x7f3d64069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:41.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.032+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6303+0+0 (secure 0 0 0) 0x7f3d7411f380 con 0x7f3d700039c0 2024-04-22T01:18:41.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.032+0000 7f3d7bfff640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d640678f0 0x7f3d64069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:41.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.032+0000 7f3d7bfff640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d640678f0 0x7f3d64069db0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3d841469f0 tx=0x7f3d6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:41.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.036+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3d84147ca0 con 0x7f3d700039c0 2024-04-22T01:18:41.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.232+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3d84068210 con 0x7f3d700039c0 2024-04-22T01:18:41.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.880+0000 7f3d7affd640 1 -- 172.21.15.70:0/2344209013 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) ==== 104+0+0 (secure 0 0 0) 0x7f3d740169d0 con 0x7f3d700039c0 2024-04-22T01:18:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d640678f0 msgr2=0x7f3d64069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d640678f0 0x7f3d64069db0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3d841469f0 tx=0x7f3d6c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 msgr2=0x7f3d84142d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d84142d70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400c190 tx=0x7f3d70005840 comp rx=0 tx=0).stop 2024-04-22T01:18:41.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 shutdown_connections 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3d640678f0 0x7f3d64069db0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d84147890 0x7f3d8414e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8415a960 0x7f3d84147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 --2- 172.21.15.70:0/2344209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d700039c0 0x7f3d84142d70 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 >> 172.21.15.70:0/2344209013 conn(0x7f3d840b1d40 msgr2=0x7f3d840cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 shutdown_connections 2024-04-22T01:18:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:41.884+0000 7f3d88a38640 1 -- 172.21.15.70:0/2344209013 wait complete. 2024-04-22T01:18:41.906 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:18:42.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa868c94640 1 Processor -- start 2024-04-22T01:18:42.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa868c94640 1 -- start start 2024-04-22T01:18:42.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa868c94640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa864072000 0x7fa8640cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:42.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa868c94640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86406da70 con 0x7fa864071b70 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa868c94640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86406dd50 con 0x7fa86415a320 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa868c94640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa86406f860 con 0x7fa864072000 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa862d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa864072000 0x7fa8640cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa862d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa864072000 0x7fa8640cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57922/0 (socket says 172.21.15.70:57922) 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa862d9d640 1 -- 172.21.15.70:0/1686405773 learned_addr learned my addr 172.21.15.70:0/1686405773 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa862d9d640 1 -- 172.21.15.70:0/1686405773 >> v1:172.21.15.98:6789/0 conn(0x7fa86415a320 legacy=0x7fa86406f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa862d9d640 1 -- 172.21.15.70:0/1686405773 >> v1:172.21.15.70:6789/0 conn(0x7fa864071b70 legacy=0x7fa8640cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:42.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.080+0000 7fa862d9d640 1 -- 172.21.15.70:0/1686405773 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86406e030 con 0x7fa864072000 2024-04-22T01:18:42.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa862d9d640 1 --2- 172.21.15.70:0/1686405773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa864072000 0x7fa8640cdb20 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fa85400aa10 tx=0x7fa85401d780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9e2bea14352727 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:42.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa86159a640 1 -- 172.21.15.70:0/1686405773 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8540b3070 con 0x7fa864072000 2024-04-22T01:18:42.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa86159a640 1 -- 172.21.15.70:0/1686405773 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8540a6830 con 0x7fa864072000 2024-04-22T01:18:42.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa86159a640 1 -- 172.21.15.70:0/1686405773 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85400b040 con 0x7fa864072000 2024-04-22T01:18:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 -- 172.21.15.70:0/1686405773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa864072000 msgr2=0x7fa8640cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 --2- 172.21.15.70:0/1686405773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa864072000 0x7fa8640cdb20 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fa85400aa10 tx=0x7fa85401d780 comp rx=0 tx=0).stop 2024-04-22T01:18:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 -- 172.21.15.70:0/1686405773 shutdown_connections 2024-04-22T01:18:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 --2- 172.21.15.70:0/1686405773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa864072000 0x7fa8640cdb20 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 -- 172.21.15.70:0/1686405773 >> 172.21.15.70:0/1686405773 conn(0x7fa8640b1d40 msgr2=0x7fa8640cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 -- 172.21.15.70:0/1686405773 shutdown_connections 2024-04-22T01:18:42.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 -- 172.21.15.70:0/1686405773 wait complete. 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 Processor -- start 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.084+0000 7fa868c94640 1 -- start start 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa864071b70 0x7fa8641477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa864072000 0x7fa8641f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 0x7fa8640ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa86415a700 con 0x7fa86415a320 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa86406e5f0 con 0x7fa864072000 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa864072430 con 0x7fa864071b70 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 0x7fa8640ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 0x7fa8640ae680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57932/0 (socket says 172.21.15.70:57932) 2024-04-22T01:18:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 -- 172.21.15.70:0/1939555370 learned_addr learned my addr 172.21.15.70:0/1939555370 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa86259c640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa864071b70 0x7fa8641477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 -- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa864071b70 msgr2=0x7fa8641477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa864071b70 0x7fa8641477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa861d9b640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa864072000 0x7fa8641f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 -- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa864072000 msgr2=0x7fa8641f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa864072000 0x7fa8641f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 -- 172.21.15.70:0/1939555370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8640aeeb0 con 0x7fa86415a320 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa862d9d640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 0x7fa8640ae680 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fa85400a1c0 tx=0x7fa85401bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8540be070 con 0x7fa86415a320 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8540b9440 con 0x7fa86415a320 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8640af190 con 0x7fa86415a320 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8540b3070 con 0x7fa86415a320 2024-04-22T01:18:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8641f74a0 con 0x7fa86415a320 2024-04-22T01:18:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa864068210 con 0x7fa86415a320 2024-04-22T01:18:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8540c1070 con 0x7fa86415a320 2024-04-22T01:18:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa852ffd640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8440678c0 0x7fa844069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6303+0+0 (secure 0 0 0) 0x7fa854120220 con 0x7fa86415a320 2024-04-22T01:18:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa86259c640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8440678c0 0x7fa844069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.088+0000 7fa86259c640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8440678c0 0x7fa844069d80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00a750 tx=0x7fa85c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.092+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa854017440 con 0x7fa86415a320 2024-04-22T01:18:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.256+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa86406be50 con 0x7fa86415a320 2024-04-22T01:18:42.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.888+0000 7fa852ffd640 1 -- 172.21.15.70:0/1939555370 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) ==== 181+0+0 (secure 0 0 0) 0x7fa8540f0ba0 con 0x7fa86415a320 2024-04-22T01:18:42.891 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:18:42.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.892+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8440678c0 msgr2=0x7fa844069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:42.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.892+0000 7fa868c94640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8440678c0 0x7fa844069d80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00a750 tx=0x7fa85c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:42.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.892+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 msgr2=0x7fa8640ae680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:42.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.892+0000 7fa868c94640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 0x7fa8640ae680 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fa85400a1c0 tx=0x7fa85401bfc0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 shutdown_connections 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa8440678c0 0x7fa844069d80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa86415a320 0x7fa8640ae680 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa864072000 0x7fa8641f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 --2- 172.21.15.70:0/1939555370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa864071b70 0x7fa8641477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 >> 172.21.15.70:0/1939555370 conn(0x7fa8640b1d40 msgr2=0x7fa8640cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 shutdown_connections 2024-04-22T01:18:42.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:42.896+0000 7fa868c94640 1 -- 172.21.15.70:0/1939555370 wait complete. 2024-04-22T01:18:42.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:18:43.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.044+0000 7f17f318d640 1 Processor -- start 2024-04-22T01:18:43.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.044+0000 7f17f318d640 1 -- start start 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f318d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 0x7f17ec16d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f318d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17ec06e340 con 0x7f17ec06f3e0 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f318d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17ec06e620 con 0x7f17ec162350 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f318d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17ec06e900 con 0x7f17ec072280 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17ebfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f17ec06f3e0 0x7f17ec071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35100/0 (socket says 172.21.15.70:35100) 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17ebfff640 1 -- 172.21.15.70:0/4252325458 learned_addr learned my addr 172.21.15.70:0/4252325458 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f172a640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 0x7f17ec16d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2714106409 0 0) 0x7f17ec06e340 con 0x7f17ec06f3e0 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d4003530 con 0x7f17ec06f3e0 2024-04-22T01:18:43.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3304804969 0 0) 0x7f17d4003530 con 0x7f17ec06f3e0 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 msgr2=0x7f17ec16d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 0x7f17ec16d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 >> v1:172.21.15.98:6789/0 conn(0x7f17ec162350 legacy=0x7f17ec06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec06ebe0 con 0x7f17ec06f3e0 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f17e0001680 con 0x7f17ec06f3e0 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 >> v1:172.21.15.70:6789/0 conn(0x7f17ec06f3e0 legacy=0x7f17ec071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d40039f0 0x7f17d4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d4004810 0x7f17d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 0x7f17d40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec06e900 con 0x7f17d40039f0 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17d400bab0 con 0x7f17d4004810 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17d400bcb0 con 0x7f17ec072280 2024-04-22T01:18:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f0f29640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d40039f0 0x7f17d4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17ebfff640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d4004810 0x7f17d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f172a640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 0x7f17d40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f0f29640 1 -- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 msgr2=0x7f17d40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f0f29640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec072280 0x7f17d40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f0f29640 1 -- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d4004810 msgr2=0x7f17d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f0f29640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d4004810 0x7f17d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.048+0000 7f17f0f29640 1 -- 172.21.15.70:0/4252325458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d4003530 con 0x7f17d40039f0 2024-04-22T01:18:43.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f0f29640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d40039f0 0x7f17d4003e10 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001750 tx=0x7f17dc01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f679e1f79de0c700 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:43.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0013a0 con 0x7f17d40039f0 2024-04-22T01:18:43.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17dc0052f0 con 0x7f17d40039f0 2024-04-22T01:18:43.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17eb7fe640 1 -- 172.21.15.70:0/4252325458 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc003840 con 0x7f17d40039f0 2024-04-22T01:18:43.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- 172.21.15.70:0/4252325458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d40039f0 msgr2=0x7f17d4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 --2- 172.21.15.70:0/4252325458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17d40039f0 0x7f17d4003e10 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001750 tx=0x7f17dc01c390 comp rx=0 tx=0).stop 2024-04-22T01:18:43.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f172a640 1 -- 172.21.15.70:0/4252325458 reap_dead start 2024-04-22T01:18:43.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- 172.21.15.70:0/4252325458 shutdown_connections 2024-04-22T01:18:43.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- 172.21.15.70:0/4252325458 >> 172.21.15.70:0/4252325458 conn(0x7f17ec0b1d40 msgr2=0x7f17ec0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:43.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- 172.21.15.70:0/4252325458 shutdown_connections 2024-04-22T01:18:43.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- 172.21.15.70:0/4252325458 wait complete. 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 Processor -- start 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- start start 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d40039f0 0x7f17ec1603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec1487b0 0x7f17ec1ff230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 0x7f17ec0b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec16dab0 con 0x7f17ec0b0080 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec16d930 con 0x7f17d40039f0 2024-04-22T01:18:43.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f318d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17ec16dc30 con 0x7f17ec1487b0 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f172a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 0x7f17ec0b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f172a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 0x7f17ec0b0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57956/0 (socket says 172.21.15.70:57956) 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f172a640 1 -- 172.21.15.70:0/1514036687 learned_addr learned my addr 172.21.15.70:0/1514036687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17ebfff640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec1487b0 0x7f17ec1ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.052+0000 7f17f0f29640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d40039f0 0x7f17ec1603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f172a640 1 -- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec1487b0 msgr2=0x7f17ec1ff230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f172a640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec1487b0 0x7f17ec1ff230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f172a640 1 -- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d40039f0 msgr2=0x7f17ec1603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f172a640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d40039f0 0x7f17ec1603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f172a640 1 -- 172.21.15.70:0/1514036687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec160ab0 con 0x7f17ec0b0080 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f0f29640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d40039f0 0x7f17ec1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f172a640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 0x7f17ec0b0500 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f17ec072e80 tx=0x7f17e40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e40039a0 con 0x7f17ec0b0080 2024-04-22T01:18:43.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e401b070 con 0x7f17ec0b0080 2024-04-22T01:18:43.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec160d90 con 0x7f17ec0b0080 2024-04-22T01:18:43.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4003b80 con 0x7f17ec0b0080 2024-04-22T01:18:43.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec161350 con 0x7f17ec0b0080 2024-04-22T01:18:43.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ec068210 con 0x7f17ec0b0080 2024-04-22T01:18:43.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f17e4004030 con 0x7f17ec0b0080 2024-04-22T01:18:43.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17e97fa640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17d80678c0 0x7f17d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:43.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5865+0+0 (secure 0 0 0) 0x7f17e4086520 con 0x7f17ec0b0080 2024-04-22T01:18:43.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f0f29640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17d80678c0 0x7f17d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:43.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.056+0000 7f17f0f29640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17d80678c0 0x7f17d8069d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f17d40057f0 tx=0x7f17dc0011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:43.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.060+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f17e4025020 con 0x7f17ec0b0080 2024-04-22T01:18:43.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:18:43.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:18:43.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.224+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f17ec0d0300 con 0x7f17ec0b0080 2024-04-22T01:18:43.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.892+0000 7f17e97fa640 1 -- 172.21.15.70:0/1514036687 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) ==== 169+0+0 (secure 0 0 0) 0x7f17e4057050 con 0x7f17ec0b0080 2024-04-22T01:18:43.893 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17d80678c0 msgr2=0x7f17d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17d80678c0 0x7f17d8069d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f17d40057f0 tx=0x7f17dc0011a0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 msgr2=0x7f17ec0b0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 0x7f17ec0b0500 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f17ec072e80 tx=0x7f17e40115a0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 shutdown_connections 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f17d80678c0 0x7f17d8069d80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17ec0b0080 0x7f17ec0b0500 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ec1487b0 0x7f17ec1ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 --2- 172.21.15.70:0/1514036687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d40039f0 0x7f17ec1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:43.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 >> 172.21.15.70:0/1514036687 conn(0x7f17ec0b1d40 msgr2=0x7f17ec0cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:43.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 shutdown_connections 2024-04-22T01:18:43.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:43.896+0000 7f17f318d640 1 -- 172.21.15.70:0/1514036687 wait complete. 2024-04-22T01:18:43.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-22T01:18:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 Processor -- start 2024-04-22T01:18:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 -- start start 2024-04-22T01:18:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84fc072280 0x7f84fc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f84fc0cd050 0x7f84fc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84fc15c290 con 0x7f84fc0b4450 2024-04-22T01:18:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f84fc15c570 con 0x7f84fc0cd050 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f8501e3b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f84fc15c850 con 0x7f84fc072280 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f84fbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f84fc0cd050 0x7f84fc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f84fbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f84fc0cd050 0x7f84fc0b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57970/0 (socket says 172.21.15.70:57970) 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f84fb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f84fc0b4450 0x7f84fc071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35106/0 (socket says 172.21.15.70:35106) 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.088+0000 7f84fbfff640 1 -- 172.21.15.70:0/191276349 learned_addr learned my addr 172.21.15.70:0/191276349 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84faffd640 1 --2- 172.21.15.70:0/191276349 >> v2:172.21.15.98:3300/0 conn(0x7f84fc072280 0x7f84fc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fbfff640 1 -- 172.21.15.70:0/191276349 >> v2:172.21.15.98:3300/0 conn(0x7f84fc072280 msgr2=0x7f84fc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fbfff640 1 --2- 172.21.15.70:0/191276349 >> v2:172.21.15.98:3300/0 conn(0x7f84fc072280 0x7f84fc0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fbfff640 1 -- 172.21.15.70:0/191276349 >> v1:172.21.15.70:6789/0 conn(0x7f84fc0b4450 legacy=0x7f84fc071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fbfff640 1 -- 172.21.15.70:0/191276349 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc15cb30 con 0x7f84fc0cd050 2024-04-22T01:18:44.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fa7fc640 1 -- 172.21.15.70:0/191276349 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1294484673 0 0) 0x7f84fc15c290 con 0x7f84fc0b4450 2024-04-22T01:18:44.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fbfff640 1 --2- 172.21.15.70:0/191276349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc0b2a40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0cf9a0 tx=0x7f84f401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffce3dde7b643b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fa7fc640 1 -- 172.21.15.70:0/191276349 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40a5750 con 0x7f84fc0cd050 2024-04-22T01:18:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fa7fc640 1 -- 172.21.15.70:0/191276349 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f40a58f0 con 0x7f84fc0cd050 2024-04-22T01:18:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f84fa7fc640 1 -- 172.21.15.70:0/191276349 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40a5cc0 con 0x7f84fc0cd050 2024-04-22T01:18:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 -- 172.21.15.70:0/191276349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 msgr2=0x7f84fc0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 --2- 172.21.15.70:0/191276349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc0b2a40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0cf9a0 tx=0x7f84f401c960 comp rx=0 tx=0).stop 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 -- 172.21.15.70:0/191276349 shutdown_connections 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 --2- 172.21.15.70:0/191276349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc0b2a40 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 --2- 172.21.15.70:0/191276349 >> v2:172.21.15.98:3300/0 conn(0x7f84fc072280 0x7f84fc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 -- 172.21.15.70:0/191276349 >> 172.21.15.70:0/191276349 conn(0x7f84fc0b1d40 msgr2=0x7f84fc0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 -- 172.21.15.70:0/191276349 shutdown_connections 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.092+0000 7f8501e3b640 1 -- 172.21.15.70:0/191276349 wait complete. 2024-04-22T01:18:44.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 Processor -- start 2024-04-22T01:18:44.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- start start 2024-04-22T01:18:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84fc072280 0x7f84fc152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84fc0b4450 0x7f84fc152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc13f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc0cd980 con 0x7f84fc0cd050 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc0cd800 con 0x7f84fc072280 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84fc0b3790 con 0x7f84fc0b4450 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc13f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc13f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57976/0 (socket says 172.21.15.70:57976) 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 -- 172.21.15.70:0/3872394007 learned_addr learned my addr 172.21.15.70:0/3872394007 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fb7fe640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84fc072280 0x7f84fc152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84faffd640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84fc0b4450 0x7f84fc152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 -- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84fc0b4450 msgr2=0x7f84fc152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84fc0b4450 0x7f84fc152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 -- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84fc072280 msgr2=0x7f84fc152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84fc072280 0x7f84fc152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 -- 172.21.15.70:0/3872394007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc13f920 con 0x7f84fc0cd050 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84fbfff640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc13f3e0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f84f4019570 tx=0x7f84f4019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40b8670 con 0x7f84fc0cd050 2024-04-22T01:18:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f40be070 con 0x7f84fc0cd050 2024-04-22T01:18:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84fc06d990 con 0x7f84fc0cd050 2024-04-22T01:18:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40b8a50 con 0x7f84fc0cd050 2024-04-22T01:18:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84fc06df50 con 0x7f84fc0cd050 2024-04-22T01:18:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84fc068210 con 0x7f84fc0cd050 2024-04-22T01:18:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.096+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f84f40b8bf0 con 0x7f84fc0cd050 2024-04-22T01:18:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.100+0000 7f84f3fff640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f84dc0678c0 0x7f84dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.100+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7f84f4120c50 con 0x7f84fc0cd050 2024-04-22T01:18:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.100+0000 7f84fb7fe640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f84dc0678c0 0x7f84dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.100+0000 7f84fb7fe640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f84dc0678c0 0x7f84dc069d80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000f70 tx=0x7f84e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:44.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.104+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f84f4126020 con 0x7f84fc0cd050 2024-04-22T01:18:44.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.264+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f84fc0d01e0 con 0x7f84fc0cd050 2024-04-22T01:18:44.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.268+0000 7f84f3fff640 1 -- 172.21.15.70:0/3872394007 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f84f40165e0 con 0x7f84fc0cd050 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f84dc0678c0 msgr2=0x7f84dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f84dc0678c0 0x7f84dc069d80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000f70 tx=0x7f84e4009040 comp rx=0 tx=0).stop 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 msgr2=0x7f84fc13f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc13f3e0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f84f4019570 tx=0x7f84f4019700 comp rx=0 tx=0).stop 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 shutdown_connections 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f84dc0678c0 0x7f84dc069d80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84fc0cd050 0x7f84fc13f3e0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84fc0b4450 0x7f84fc152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 --2- 172.21.15.70:0/3872394007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84fc072280 0x7f84fc152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 >> 172.21.15.70:0/3872394007 conn(0x7f84fc0b1d40 msgr2=0x7f84fc0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:44.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 shutdown_connections 2024-04-22T01:18:44.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.272+0000 7f8501e3b640 1 -- 172.21.15.70:0/3872394007 wait complete. 2024-04-22T01:18:44.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr' 2024-04-22T01:18:44.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 Processor -- start 2024-04-22T01:18:44.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 -- start start 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc63015dcf0 0x7fc6300d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300b28f0 0x7fc6300b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc630161cf0 con 0x7fc63015e0d0 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc630161fd0 con 0x7fc63015dcf0 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.404+0000 7fc635251640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc6301622b0 con 0x7fc6300b28f0 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc63015dcf0 0x7fc6300d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc63015dcf0 0x7fc6300d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57982/0 (socket says 172.21.15.70:57982) 2024-04-22T01:18:44.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 -- 172.21.15.70:0/2090784094 learned_addr learned my addr 172.21.15.70:0/2090784094 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:44.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 -- 172.21.15.70:0/2090784094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300b28f0 msgr2=0x7fc6300b3520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T01:18:44.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 --2- 172.21.15.70:0/2090784094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300b28f0 0x7fc6300b3520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 -- 172.21.15.70:0/2090784094 >> v1:172.21.15.98:6789/0 conn(0x7fc63015e0d0 legacy=0x7fc630071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 -- 172.21.15.70:0/2090784094 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc630162590 con 0x7fc63015dcf0 2024-04-22T01:18:44.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62ed9d640 1 --2- 172.21.15.70:0/2090784094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63015dcf0 0x7fc6300d0130 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fc620014420 tx=0x7fc620025570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4e91edcf666937 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:44.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62dd9b640 1 -- 172.21.15.70:0/2090784094 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6200b35f0 con 0x7fc63015dcf0 2024-04-22T01:18:44.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62dd9b640 1 -- 172.21.15.70:0/2090784094 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6200b8070 con 0x7fc63015dcf0 2024-04-22T01:18:44.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.408+0000 7fc62dd9b640 1 -- 172.21.15.70:0/2090784094 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6200b39b0 con 0x7fc63015dcf0 2024-04-22T01:18:44.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/2090784094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63015dcf0 msgr2=0x7fc6300d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 --2- 172.21.15.70:0/2090784094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63015dcf0 0x7fc6300d0130 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fc620014420 tx=0x7fc620025570 comp rx=0 tx=0).stop 2024-04-22T01:18:44.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/2090784094 shutdown_connections 2024-04-22T01:18:44.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 --2- 172.21.15.70:0/2090784094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300b28f0 0x7fc6300b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 --2- 172.21.15.70:0/2090784094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63015dcf0 0x7fc6300d0130 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/2090784094 >> 172.21.15.70:0/2090784094 conn(0x7fc630067550 msgr2=0x7fc63006b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:44.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/2090784094 shutdown_connections 2024-04-22T01:18:44.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/2090784094 wait complete. 2024-04-22T01:18:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 Processor -- start 2024-04-22T01:18:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- start start 2024-04-22T01:18:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 0x7fc6301f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63015dcf0 0x7fc630160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63015e0d0 0x7fc63006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300aa860 con 0x7fc6300b28f0 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300aa6e0 con 0x7fc63015dcf0 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc6300aa9e0 con 0x7fc63015e0d0 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 0x7fc6301f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 0x7fc6301f5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57990/0 (socket says 172.21.15.70:57990) 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 -- 172.21.15.70:0/1450543810 learned_addr learned my addr 172.21.15.70:0/1450543810 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62f59e640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63015e0d0 0x7fc63006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62e59c640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63015dcf0 0x7fc630160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 -- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63015e0d0 msgr2=0x7fc63006e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63015e0d0 0x7fc63006e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 -- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63015dcf0 msgr2=0x7fc630160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63015dcf0 0x7fc630160da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 -- 172.21.15.70:0/1450543810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6301f3a90 con 0x7fc6300b28f0 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62e59c640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63015dcf0 0x7fc630160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc62ed9d640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 0x7fc6301f5e20 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc6200b5040 tx=0x7fc620022310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6200c8450 con 0x7fc6300b28f0 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6200c85f0 con 0x7fc6300b28f0 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6301f3d30 con 0x7fc6300b28f0 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6200c89c0 con 0x7fc6300b28f0 2024-04-22T01:18:44.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.412+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc630153740 con 0x7fc6300b28f0 2024-04-22T01:18:44.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.416+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc630068210 con 0x7fc6300b28f0 2024-04-22T01:18:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.416+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6200c8b60 con 0x7fc6300b28f0 2024-04-22T01:18:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.416+0000 7fc6177fe640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc61c0678c0 0x7fc61c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.416+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7fc620129940 con 0x7fc6300b28f0 2024-04-22T01:18:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.416+0000 7fc62e59c640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc61c0678c0 0x7fc61c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:44.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.416+0000 7fc62e59c640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc61c0678c0 0x7fc61c069d80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:44.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.420+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc6300b3e70 con 0x7fc6300b28f0 2024-04-22T01:18:44.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.640+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]} v 0) -- 0x7fc63006e050 con 0x7fc6300b28f0 2024-04-22T01:18:44.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.908+0000 7fc6177fe640 1 -- 172.21.15.70:0/1450543810 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]}]=0 v238) ==== 140+0+0 (secure 0 0 0) 0x7fc62001f160 con 0x7fc6300b28f0 2024-04-22T01:18:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc61c0678c0 msgr2=0x7fc61c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc61c0678c0 0x7fc61c069d80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).stop 2024-04-22T01:18:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 msgr2=0x7fc6301f5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 0x7fc6301f5e20 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc6200b5040 tx=0x7fc620022310 comp rx=0 tx=0).stop 2024-04-22T01:18:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 shutdown_connections 2024-04-22T01:18:44.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc61c0678c0 0x7fc61c069d80 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63015e0d0 0x7fc63006e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63015dcf0 0x7fc630160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 --2- 172.21.15.70:0/1450543810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6300b28f0 0x7fc6301f5e20 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:44.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 >> 172.21.15.70:0/1450543810 conn(0x7fc630067550 msgr2=0x7fc6301592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:44.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 shutdown_connections 2024-04-22T01:18:44.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:44.912+0000 7fc635251640 1 -- 172.21.15.70:0/1450543810 wait complete. 2024-04-22T01:18:44.934 INFO:tasks.cephfs_test_runner:That quota changes made from one client are visible to another ... ok 2024-04-22T01:18:44.935 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_write (tasks.cephfs.test_quota.TestQuota) That when a client modifies the quota on a directory used 2024-04-22T01:18:44.935 INFO:tasks.cephfs_test_runner:test_remote_update_write (tasks.cephfs.test_quota.TestQuota) 2024-04-22T01:18:44.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.052+0000 7ffa9d672640 1 Processor -- start 2024-04-22T01:18:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.052+0000 7ffa9d672640 1 -- start start 2024-04-22T01:18:45.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa9d672640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffa980cc150 0x7ffa98071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa9d672640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa980cd7b0 con 0x7ffa98072060 2024-04-22T01:18:45.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa9d672640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa980cc710 con 0x7ffa981455b0 2024-04-22T01:18:45.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa9d672640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ffa98072490 con 0x7ffa980cc150 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa967fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffa980cc150 0x7ffa98071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa977fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ffa98072060 0x7ffa9806e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35114/0 (socket says 172.21.15.70:35114) 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa977fe640 1 -- 172.21.15.70:0/3937716649 learned_addr learned my addr 172.21.15.70:0/3937716649 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa95ffb640 1 -- 172.21.15.70:0/3937716649 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563454792 0 0) 0x7ffa980cd7b0 con 0x7ffa98072060 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa95ffb640 1 -- 172.21.15.70:0/3937716649 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa80003530 con 0x7ffa98072060 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa967fc640 1 -- 172.21.15.70:0/3937716649 >> v1:172.21.15.98:6789/0 conn(0x7ffa981455b0 legacy=0x7ffa980cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa967fc640 1 -- 172.21.15.70:0/3937716649 >> v1:172.21.15.70:6789/0 conn(0x7ffa98072060 legacy=0x7ffa9806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa967fc640 1 -- 172.21.15.70:0/3937716649 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa980cc9f0 con 0x7ffa980cc150 2024-04-22T01:18:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa967fc640 1 --2- 172.21.15.70:0/3937716649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa980cc150 0x7ffa98071b20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0097c0 tx=0x7ffa8c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c211565078317c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa95ffb640 1 -- 172.21.15.70:0/3937716649 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c0b0070 con 0x7ffa980cc150 2024-04-22T01:18:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa95ffb640 1 -- 172.21.15.70:0/3937716649 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa8c0ab600 con 0x7ffa980cc150 2024-04-22T01:18:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa95ffb640 1 -- 172.21.15.70:0/3937716649 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c0ab9b0 con 0x7ffa980cc150 2024-04-22T01:18:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa9d672640 1 -- 172.21.15.70:0/3937716649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa980cc150 msgr2=0x7ffa98071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.056+0000 7ffa9d672640 1 --2- 172.21.15.70:0/3937716649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa980cc150 0x7ffa98071b20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0097c0 tx=0x7ffa8c01c380 comp rx=0 tx=0).stop 2024-04-22T01:18:45.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- 172.21.15.70:0/3937716649 shutdown_connections 2024-04-22T01:18:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 --2- 172.21.15.70:0/3937716649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa980cc150 0x7ffa98071b20 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- 172.21.15.70:0/3937716649 >> 172.21.15.70:0/3937716649 conn(0x7ffa9806b1f0 msgr2=0x7ffa9806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- 172.21.15.70:0/3937716649 shutdown_connections 2024-04-22T01:18:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- 172.21.15.70:0/3937716649 wait complete. 2024-04-22T01:18:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 Processor -- start 2024-04-22T01:18:45.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- start start 2024-04-22T01:18:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 0x7ffa980b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa980cc150 0x7ffa9813acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa981455b0 0x7ffa9813b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa98169ee0 con 0x7ffa98072060 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa98169d60 con 0x7ffa980cc150 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffa9816a060 con 0x7ffa981455b0 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 0x7ffa980b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 0x7ffa980b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58014/0 (socket says 172.21.15.70:58014) 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 -- 172.21.15.70:0/2557022548 learned_addr learned my addr 172.21.15.70:0/2557022548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 -- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa981455b0 msgr2=0x7ffa9813b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa977fe640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa981455b0 0x7ffa9813b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa981455b0 0x7ffa9813b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa967fc640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa980cc150 0x7ffa9813acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 -- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa980cc150 msgr2=0x7ffa9813acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa980cc150 0x7ffa9813acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 -- 172.21.15.70:0/2557022548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9815f110 con 0x7ffa98072060 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa96ffd640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 0x7ffa980b4310 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800c3c0 tx=0x7ffa8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88017070 con 0x7ffa98072060 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa880052d0 con 0x7ffa98072060 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88003840 con 0x7ffa98072060 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa981fa940 con 0x7ffa98072060 2024-04-22T01:18:45.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.060+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa981faf00 con 0x7ffa98072060 2024-04-22T01:18:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.064+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa98068210 con 0x7ffa98072060 2024-04-22T01:18:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.064+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffa8801a070 con 0x7ffa98072060 2024-04-22T01:18:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.064+0000 7ffa7f7fe640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffa840678c0 0x7ffa84069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.064+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7ffa88084e60 con 0x7ffa98072060 2024-04-22T01:18:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.064+0000 7ffa967fc640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffa840678c0 0x7ffa84069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.064+0000 7ffa967fc640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffa840678c0 0x7ffa84069d80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0ad000 tx=0x7ffa8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.068+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ffa88005480 con 0x7ffa98072060 2024-04-22T01:18:45.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.216+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa9815f510 con 0x7ffa98072060 2024-04-22T01:18:45.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa7f7fe640 1 -- 172.21.15.70:0/2557022548 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+13258 (secure 0 0 0) 0x7ffa88055b10 con 0x7ffa98072060 2024-04-22T01:18:45.221 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:45.221 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":94,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:43.893338+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:6837/2337533615":"2024-04-23T01:18:40.881362+0000","172.21.15.70:6836/2337533615":"2024-04-23T01:18:40.881362+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffa840678c0 msgr2=0x7ffa84069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffa840678c0 0x7ffa84069d80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0ad000 tx=0x7ffa8c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 msgr2=0x7ffa980b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 0x7ffa980b4310 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800c3c0 tx=0x7ffa8800c880 comp rx=0 tx=0).stop 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 shutdown_connections 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ffa840678c0 0x7ffa84069d80 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa981455b0 0x7ffa9813b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa980cc150 0x7ffa9813acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 --2- 172.21.15.70:0/2557022548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa98072060 0x7ffa980b4310 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 >> 172.21.15.70:0/2557022548 conn(0x7ffa9806b1f0 msgr2=0x7ffa9806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:45.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 shutdown_connections 2024-04-22T01:18:45.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.220+0000 7ffa9d672640 1 -- 172.21.15.70:0/2557022548 wait complete. 2024-04-22T01:18:45.239 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:45.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:45.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.356+0000 7fdb9a4a9640 1 Processor -- start 2024-04-22T01:18:45.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.356+0000 7fdb9a4a9640 1 -- start start 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb9a4a9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb9415a320 0x7fdb9406f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb9a4a9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb94071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb9a4a9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb9406da70 con 0x7fdb940b3d60 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb9a4a9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdb94072510 con 0x7fdb9415a320 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb9a4a9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9406dd50 con 0x7fdb94071b70 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb98a46640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdb940b3d60 0x7fdb940b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35124/0 (socket says 172.21.15.70:35124) 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb98a46640 1 -- 172.21.15.70:0/1930038492 learned_addr learned my addr 172.21.15.70:0/1930038492 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:45.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb93fff640 1 --2- 172.21.15.70:0/1930038492 >> v2:172.21.15.98:3300/0 conn(0x7fdb9415a320 0x7fdb9406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb94071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118466348 0 0) 0x7fdb9406da70 con 0x7fdb940b3d60 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb7c003530 con 0x7fdb940b3d60 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1292124119 0 0) 0x7fdb7c003530 con 0x7fdb940b3d60 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 msgr2=0x7fdb94071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb94071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 >> v2:172.21.15.98:3300/0 conn(0x7fdb9415a320 msgr2=0x7fdb9406f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 --2- 172.21.15.70:0/1930038492 >> v2:172.21.15.98:3300/0 conn(0x7fdb9415a320 0x7fdb9406f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9406e030 con 0x7fdb940b3d60 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fdb8c001a00 con 0x7fdb940b3d60 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 >> v1:172.21.15.70:6789/0 conn(0x7fdb940b3d60 legacy=0x7fdb940b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb7c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c003f40 0x7fdb7c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c004d50 0x7fdb7c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9406dd50 con 0x7fdb7c003f40 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c00b9d0 con 0x7fdb7c004d50 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c00bb50 con 0x7fdb94071b70 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c003f40 0x7fdb7c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb98a46640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c004d50 0x7fdb7c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb93fff640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb7c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 -- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 msgr2=0x7fdb7c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb7c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 -- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c004d50 msgr2=0x7fdb7c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c004d50 0x7fdb7c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 -- 172.21.15.70:0/1930038492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c003530 con 0x7fdb7c003f40 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb98a46640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb7c004d50 0x7fdb7c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb93fff640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb94071b70 0x7fdb7c0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c003f40 0x7fdb7c004810 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009ae0 tx=0x7fdb8401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac4d3c555497a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb840b2070 con 0x7fdb7c003f40 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb840a5660 con 0x7fdb7c003f40 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.360+0000 7fdb92ffd640 1 -- 172.21.15.70:0/1930038492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb840096b0 con 0x7fdb7c003f40 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1930038492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c003f40 msgr2=0x7fdb7c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1930038492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb7c003f40 0x7fdb7c004810 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009ae0 tx=0x7fdb8401ca10 comp rx=0 tx=0).stop 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb98a46640 1 -- 172.21.15.70:0/1930038492 reap_dead start 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1930038492 shutdown_connections 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1930038492 >> 172.21.15.70:0/1930038492 conn(0x7fdb940b1d40 msgr2=0x7fdb940cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1930038492 shutdown_connections 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1930038492 wait complete. 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 Processor -- start 2024-04-22T01:18:45.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- start start 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c003f40 0x7fdb9414c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 0x7fdb9414e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94154c80 0x7fdb94155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9415a700 con 0x7fdb9414e020 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9406e5f0 con 0x7fdb94154c80 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb940cf420 con 0x7fdb7c003f40 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 0x7fdb9414e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 0x7fdb9414e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58026/0 (socket says 172.21.15.70:58026) 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 -- 172.21.15.70:0/1661308108 learned_addr learned my addr 172.21.15.70:0/1661308108 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb93fff640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c003f40 0x7fdb9414c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb98a46640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94154c80 0x7fdb94155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 -- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c003f40 msgr2=0x7fdb9414c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c003f40 0x7fdb9414c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 -- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94154c80 msgr2=0x7fdb94155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94154c80 0x7fdb94155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 -- 172.21.15.70:0/1661308108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9414ea60 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb98a46640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94154c80 0x7fdb94155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb937fe640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 0x7fdb9414e480 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fdb8401c840 tx=0x7fdb7c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb840bd020 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb840a5e30 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9414ed40 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb840b2040 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9414aaa0 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.364+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb94068210 con 0x7fdb9414e020 2024-04-22T01:18:45.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.368+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb840c0070 con 0x7fdb9414e020 2024-04-22T01:18:45.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.368+0000 7fdb90ff9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb800678c0 0x7fdb80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.368+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7fdb8411f5a0 con 0x7fdb9414e020 2024-04-22T01:18:45.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.368+0000 7fdb93fff640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb800678c0 0x7fdb80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.368+0000 7fdb93fff640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb800678c0 0x7fdb80069d80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800ace0 tx=0x7fdb8800a2a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.372+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdb84016780 con 0x7fdb9414e020 2024-04-22T01:18:45.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.528+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb940cdb30 con 0x7fdb9414e020 2024-04-22T01:18:45.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.528+0000 7fdb90ff9640 1 -- 172.21.15.70:0/1661308108 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fdb840f0250 con 0x7fdb9414e020 2024-04-22T01:18:45.531 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb800678c0 msgr2=0x7fdb80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb800678c0 0x7fdb80069d80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800ace0 tx=0x7fdb8800a2a0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 msgr2=0x7fdb9414e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 0x7fdb9414e480 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fdb8401c840 tx=0x7fdb7c005840 comp rx=0 tx=0).stop 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 shutdown_connections 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdb800678c0 0x7fdb80069d80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94154c80 0x7fdb94155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb9414e020 0x7fdb9414e480 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 --2- 172.21.15.70:0/1661308108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb7c003f40 0x7fdb9414c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 >> 172.21.15.70:0/1661308108 conn(0x7fdb940b1d40 msgr2=0x7fdb940b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:45.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 shutdown_connections 2024-04-22T01:18:45.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.532+0000 7fdb9a4a9640 1 -- 172.21.15.70:0/1661308108 wait complete. 2024-04-22T01:18:45.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write' 2024-04-22T01:18:45.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.652+0000 7facc6d79640 1 Processor -- start 2024-04-22T01:18:45.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.652+0000 7facc6d79640 1 -- start start 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facc6d79640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7facc00cc150 0x7facc0071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facc6d79640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc00cd7b0 con 0x7facc0072060 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facc6d79640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc00cc710 con 0x7facc01455b0 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facc6d79640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7facc0072490 con 0x7facc00cc150 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7facc00cc150 0x7facc0071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7facc00cc150 0x7facc0071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58042/0 (socket says 172.21.15.70:58042) 2024-04-22T01:18:45.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 -- 172.21.15.70:0/995695497 learned_addr learned my addr 172.21.15.70:0/995695497 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facc5316640 1 --1- 172.21.15.70:0/995695497 >> v1:172.21.15.70:6789/0 conn(0x7facc0072060 0x7facc006e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35140/0 (socket says 172.21.15.70:35140) 2024-04-22T01:18:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 -- 172.21.15.70:0/995695497 >> v1:172.21.15.98:6789/0 conn(0x7facc01455b0 legacy=0x7facc00cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 -- 172.21.15.70:0/995695497 >> v1:172.21.15.70:6789/0 conn(0x7facc0072060 legacy=0x7facc006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbf7fe640 1 -- 172.21.15.70:0/995695497 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156022479 0 0) 0x7facc00cd7b0 con 0x7facc0072060 2024-04-22T01:18:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 -- 172.21.15.70:0/995695497 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc00cc9f0 con 0x7facc00cc150 2024-04-22T01:18:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbffff640 1 --2- 172.21.15.70:0/995695497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0071b20 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7facb4009870 tx=0x7facb401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b42e3ee5bfab51f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbf7fe640 1 -- 172.21.15.70:0/995695497 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40b0070 con 0x7facc00cc150 2024-04-22T01:18:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbf7fe640 1 -- 172.21.15.70:0/995695497 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb40ab600 con 0x7facc00cc150 2024-04-22T01:18:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.656+0000 7facbf7fe640 1 -- 172.21.15.70:0/995695497 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40ab9d0 con 0x7facc00cc150 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/995695497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 msgr2=0x7facc0071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 --2- 172.21.15.70:0/995695497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0071b20 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7facb4009870 tx=0x7facb401c400 comp rx=0 tx=0).stop 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/995695497 shutdown_connections 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 --2- 172.21.15.70:0/995695497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0071b20 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/995695497 >> 172.21.15.70:0/995695497 conn(0x7facc006b1f0 msgr2=0x7facc006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/995695497 shutdown_connections 2024-04-22T01:18:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/995695497 wait complete. 2024-04-22T01:18:45.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 Processor -- start 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- start start 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facc0072060 0x7facc0137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facc01455b0 0x7facc0157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7facc0169ee0 con 0x7facc00cc150 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facc0169d60 con 0x7facc01455b0 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7facc016a060 con 0x7facc0072060 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0138150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58050/0 (socket says 172.21.15.70:58050) 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 -- 172.21.15.70:0/135771270 learned_addr learned my addr 172.21.15.70:0/135771270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc5316640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facc01455b0 0x7facc0157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc4b15640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facc0072060 0x7facc0137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 -- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facc0072060 msgr2=0x7facc0137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facc0072060 0x7facc0137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 -- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facc01455b0 msgr2=0x7facc0157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facc01455b0 0x7facc0157000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 -- 172.21.15.70:0/135771270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc0155170 con 0x7facc00cc150 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc5316640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facc01455b0 0x7facc0157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbffff640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0138150 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7facb40ae040 tx=0x7facb40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40b0040 con 0x7facc00cc150 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb40b9070 con 0x7facc00cc150 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40b4880 con 0x7facc00cc150 2024-04-22T01:18:45.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc0155450 con 0x7facc00cc150 2024-04-22T01:18:45.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.660+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc0155a10 con 0x7facc00cc150 2024-04-22T01:18:45.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.664+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc0068210 con 0x7facc00cc150 2024-04-22T01:18:45.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.664+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7facb40ab4c0 con 0x7facc00cc150 2024-04-22T01:18:45.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.664+0000 7facbd7fa640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facac0678c0 0x7facac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:45.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.664+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7facb411f8d0 con 0x7facc00cc150 2024-04-22T01:18:45.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.664+0000 7facc4b15640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facac0678c0 0x7facac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:45.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.664+0000 7facc4b15640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facac0678c0 0x7facac069d80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7facc015bb50 tx=0x7facb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:45.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.668+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7facb4016110 con 0x7facc00cc150 2024-04-22T01:18:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.872+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]} v 0) -- 0x7facc015f510 con 0x7facc00cc150 2024-04-22T01:18:45.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.908+0000 7facbd7fa640 1 -- 172.21.15.70:0/135771270 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]}]=0 v239) ==== 140+0+0 (secure 0 0 0) 0x7facb40f04e0 con 0x7facc00cc150 2024-04-22T01:18:45.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facac0678c0 msgr2=0x7facac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facac0678c0 0x7facac069d80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7facc015bb50 tx=0x7facb0008040 comp rx=0 tx=0).stop 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 msgr2=0x7facc0138150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0138150 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7facb40ae040 tx=0x7facb40191a0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 shutdown_connections 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7facac0678c0 0x7facac069d80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facc01455b0 0x7facc0157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7facc00cc150 0x7facc0138150 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 --2- 172.21.15.70:0/135771270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7facc0072060 0x7facc0137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 >> 172.21.15.70:0/135771270 conn(0x7facc006b1f0 msgr2=0x7facc00b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 shutdown_connections 2024-04-22T01:18:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:45.912+0000 7facc6d79640 1 -- 172.21.15.70:0/135771270 wait complete. 2024-04-22T01:18:45.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-22T01:18:46.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.048+0000 7fab64130640 1 Processor -- start 2024-04-22T01:18:46.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.048+0000 7fab64130640 1 -- start start 2024-04-22T01:18:46.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab64130640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c162350 0x7fab5c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab64130640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c06e340 con 0x7fab5c06de30 2024-04-22T01:18:46.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab64130640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c06e620 con 0x7fab5c072280 2024-04-22T01:18:46.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab64130640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fab5c06e900 con 0x7fab5c162350 2024-04-22T01:18:46.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab616cb640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fab5c06de30 0x7fab5c071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:35142/0 (socket says 172.21.15.70:35142) 2024-04-22T01:18:46.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab616cb640 1 -- 172.21.15.70:0/4015359374 learned_addr learned my addr 172.21.15.70:0/4015359374 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c162350 0x7fab5c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104732431 0 0) 0x7fab5c06e340 con 0x7fab5c06de30 2024-04-22T01:18:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab48003530 con 0x7fab5c06de30 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2088258754 0 0) 0x7fab48003530 con 0x7fab5c06de30 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c162350 msgr2=0x7fab5c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c162350 0x7fab5c06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 >> v1:172.21.15.98:6789/0 conn(0x7fab5c072280 legacy=0x7fab5c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c06ebe0 con 0x7fab5c06de30 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fab4c001680 con 0x7fab5c06de30 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 >> v1:172.21.15.70:6789/0 conn(0x7fab5c06de30 legacy=0x7fab5c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab480039f0 0x7fab48003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab48004810 0x7fab48004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab480051d0 0x7fab4800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c06e900 con 0x7fab480039f0 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab4800bff0 con 0x7fab48004810 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fab4800c1f0 con 0x7fab480051d0 2024-04-22T01:18:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab480039f0 0x7fab48003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab616cb640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab48004810 0x7fab48004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab626cd640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab480051d0 0x7fab4800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 -- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab480051d0 msgr2=0x7fab4800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab480051d0 0x7fab4800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 -- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab48004810 msgr2=0x7fab48004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab48004810 0x7fab48004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.052+0000 7fab61ecc640 1 -- 172.21.15.70:0/4015359374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab48003530 con 0x7fab480039f0 2024-04-22T01:18:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab61ecc640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab480039f0 0x7fab48003e10 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fab5c070010 tx=0x7fab5401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ca328ba4535a32 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540b2070 con 0x7fab480039f0 2024-04-22T01:18:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab540a5510 con 0x7fab480039f0 2024-04-22T01:18:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab60eca640 1 -- 172.21.15.70:0/4015359374 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54009670 con 0x7fab480039f0 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- 172.21.15.70:0/4015359374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab480039f0 msgr2=0x7fab48003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 --2- 172.21.15.70:0/4015359374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab480039f0 0x7fab48003e10 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fab5c070010 tx=0x7fab5401c920 comp rx=0 tx=0).stop 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab626cd640 1 -- 172.21.15.70:0/4015359374 reap_dead start 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- 172.21.15.70:0/4015359374 shutdown_connections 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- 172.21.15.70:0/4015359374 >> 172.21.15.70:0/4015359374 conn(0x7fab5c0b1d40 msgr2=0x7fab5c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- 172.21.15.70:0/4015359374 shutdown_connections 2024-04-22T01:18:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- 172.21.15.70:0/4015359374 wait complete. 2024-04-22T01:18:46.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 Processor -- start 2024-04-22T01:18:46.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- start start 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab480039f0 0x7fab5c149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c1609a0 0x7fab5c160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 0x7fab5c16cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c16e4c0 con 0x7fab5c16c660 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c16e340 con 0x7fab480039f0 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.056+0000 7fab64130640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fab5c16e640 con 0x7fab5c1609a0 2024-04-22T01:18:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 0x7fab5c16cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 0x7fab5c16cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58080/0 (socket says 172.21.15.70:58080) 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 -- 172.21.15.70:0/2691443316 learned_addr learned my addr 172.21.15.70:0/2691443316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab61ecc640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab480039f0 0x7fab5c149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 -- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c1609a0 msgr2=0x7fab5c160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab616cb640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c1609a0 0x7fab5c160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c1609a0 0x7fab5c160e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 -- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab480039f0 msgr2=0x7fab5c149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab480039f0 0x7fab5c149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 -- 172.21.15.70:0/2691443316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c16d3d0 con 0x7fab5c16c660 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab61ecc640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab480039f0 0x7fab5c149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab626cd640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 0x7fab5c16cae0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fab5c166e50 tx=0x7fab5800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab58003980 con 0x7fab5c16c660 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab58018070 con 0x7fab5c16c660 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c16d6b0 con 0x7fab5c16c660 2024-04-22T01:18:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab58003d60 con 0x7fab5c16c660 2024-04-22T01:18:46.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c14a0a0 con 0x7fab5c16c660 2024-04-22T01:18:46.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c161360 con 0x7fab5c16c660 2024-04-22T01:18:46.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab580050d0 con 0x7fab5c16c660 2024-04-22T01:18:46.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab527fc640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab400678c0 0x7fab40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7fab58085760 con 0x7fab5c16c660 2024-04-22T01:18:46.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab61ecc640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab400678c0 0x7fab40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.060+0000 7fab61ecc640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab400678c0 0x7fab40069d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fab540af000 tx=0x7fab540046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.064+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fab580564a0 con 0x7fab5c16c660 2024-04-22T01:18:46.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.216+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fab5c16dbd0 con 0x7fab5c16c660 2024-04-22T01:18:46.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.224+0000 7fab527fc640 1 -- 172.21.15.70:0/2691443316 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7fab58055da0 con 0x7fab5c16c660 2024-04-22T01:18:46.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab400678c0 msgr2=0x7fab40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab400678c0 0x7fab40069d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fab540af000 tx=0x7fab540046d0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 msgr2=0x7fab5c16cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 0x7fab5c16cae0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fab5c166e50 tx=0x7fab5800ec10 comp rx=0 tx=0).stop 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 shutdown_connections 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fab400678c0 0x7fab40069d80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab5c16c660 0x7fab5c16cae0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab5c1609a0 0x7fab5c160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 --2- 172.21.15.70:0/2691443316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab480039f0 0x7fab5c149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 >> 172.21.15.70:0/2691443316 conn(0x7fab5c0b1d40 msgr2=0x7fab5c0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 shutdown_connections 2024-04-22T01:18:46.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.228+0000 7fab64130640 1 -- 172.21.15.70:0/2691443316 wait complete. 2024-04-22T01:18:46.245 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:18:46.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:46.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:18:46.371 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:18:46.372 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:46.372 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:46.377 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:18:46.377 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:46.377 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:46.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:18:46.383 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:18:46.383 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:46.425 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:18:46.425 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:18:46.425 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:18:46.431 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:18:46.431 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-22T01:18:46.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.584+0000 7f20f4fef640 1 Processor -- start 2024-04-22T01:18:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.584+0000 7f20f4fef640 1 -- start start 2024-04-22T01:18:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f01455b0 0x7f20f00cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20f00cc150 0x7f20f0071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f00d0250 con 0x7f20f0072060 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f20f00724d0 con 0x7f20f00cc150 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00cd7b0 con 0x7f20f01455b0 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20f00cc150 0x7f20f0071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20eed9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f20f0072060 0x7f20f015f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49812/0 (socket says 172.21.15.70:49812) 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20f00cc150 0x7f20f0071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60090/0 (socket says 172.21.15.70:60090) 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20eed9d640 1 -- 172.21.15.70:0/3572250587 learned_addr learned my addr 172.21.15.70:0/3572250587 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20ee59c640 1 --2- 172.21.15.70:0/3572250587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f01455b0 0x7f20f00cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 -- 172.21.15.70:0/3572250587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f01455b0 msgr2=0x7f20f00cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 --2- 172.21.15.70:0/3572250587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f01455b0 0x7f20f00cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 -- 172.21.15.70:0/3572250587 >> v1:172.21.15.70:6789/0 conn(0x7f20f0072060 legacy=0x7f20f015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20ed59a640 1 -- 172.21.15.70:0/3572250587 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474889650 0 0) 0x7f20f00d0250 con 0x7f20f0072060 2024-04-22T01:18:46.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 -- 172.21.15.70:0/3572250587 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f00cc710 con 0x7f20f00cc150 2024-04-22T01:18:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20edd9b640 1 --2- 172.21.15.70:0/3572250587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f0071b20 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f20e40097c0 tx=0x7f20e401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f32741315fc4c43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20ed59a640 1 -- 172.21.15.70:0/3572250587 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e40b0070 con 0x7f20f00cc150 2024-04-22T01:18:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20ed59a640 1 -- 172.21.15.70:0/3572250587 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e40ab600 con 0x7f20f00cc150 2024-04-22T01:18:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20ed59a640 1 -- 172.21.15.70:0/3572250587 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e40ab9d0 con 0x7f20f00cc150 2024-04-22T01:18:46.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 -- 172.21.15.70:0/3572250587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 msgr2=0x7f20f0071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.588+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3572250587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f0071b20 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f20e40097c0 tx=0x7f20e401c380 comp rx=0 tx=0).stop 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3572250587 shutdown_connections 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3572250587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f0071b20 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3572250587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f01455b0 0x7f20f00cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3572250587 >> 172.21.15.70:0/3572250587 conn(0x7f20f006b1f0 msgr2=0x7f20f006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3572250587 shutdown_connections 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3572250587 wait complete. 2024-04-22T01:18:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 Processor -- start 2024-04-22T01:18:46.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- start start 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f0072060 0x7f20f0164c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f01424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f01455b0 0x7f20f01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0168b40 con 0x7f20f00cc150 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f20f006f210 con 0x7f20f01455b0 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f20f0168cc0 con 0x7f20f0072060 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f01424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f01424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60106/0 (socket says 172.21.15.70:60106) 2024-04-22T01:18:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 -- 172.21.15.70:0/3043954445 learned_addr learned my addr 172.21.15.70:0/3043954445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20ee59c640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f0072060 0x7f20f0164c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20eed9d640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f01455b0 0x7f20f01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 -- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f0072060 msgr2=0x7f20f0164c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f0072060 0x7f20f0164c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 -- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f01455b0 msgr2=0x7f20f01593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f01455b0 0x7f20f01593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 -- 172.21.15.70:0/3043954445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f0165300 con 0x7f20f00cc150 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20ee59c640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f0072060 0x7f20f0164c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20edd9b640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f01424b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f20e40ae040 tx=0x7f20e4019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e40b8030 con 0x7f20f00cc150 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e40ab400 con 0x7f20f00cc150 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e40b0040 con 0x7f20f00cc150 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f01655e0 con 0x7f20f00cc150 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f0165ba0 con 0x7f20f00cc150 2024-04-22T01:18:46.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.592+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f0068210 con 0x7f20f00cc150 2024-04-22T01:18:46.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.596+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f20e40bb070 con 0x7f20f00cc150 2024-04-22T01:18:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.596+0000 7f20deffd640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20d00678c0 0x7f20d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.596+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7f20e411f2f0 con 0x7f20f00cc150 2024-04-22T01:18:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.596+0000 7f20ee59c640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20d00678c0 0x7f20d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.596+0000 7f20ee59c640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20d00678c0 0x7f20d0069d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f20e0007330 tx=0x7f20e000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.600+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f20e40c2070 con 0x7f20f00cc150 2024-04-22T01:18:46.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.780+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20f015f510 con 0x7f20f00cc150 2024-04-22T01:18:46.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.780+0000 7f20deffd640 1 -- 172.21.15.70:0/3043954445 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+3866 (secure 0 0 0) 0x7f20e4016090 con 0x7f20f00cc150 2024-04-22T01:18:46.783 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:46.783 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5402,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/510540534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":510540534},{"type":"v1","addr":"172.21.15.98:6835","nonce":510540534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5815,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3775134070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3775134070},{"type":"v1","addr":"172.21.15.98:6833","nonce":3775134070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3939653906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3939653906},{"type":"v1","addr":"172.21.15.70:6835","nonce":3939653906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":7545,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/1569869995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":1569869995},{"type":"v1","addr":"172.21.15.70:6837","nonce":1569869995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[]} 2024-04-22T01:18:46.783 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 46 2024-04-22T01:18:46.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20d00678c0 msgr2=0x7f20d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20d00678c0 0x7f20d0069d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f20e0007330 tx=0x7f20e000b040 comp rx=0 tx=0).stop 2024-04-22T01:18:46.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 msgr2=0x7f20f01424b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f01424b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f20e40ae040 tx=0x7f20e4019120 comp rx=0 tx=0).stop 2024-04-22T01:18:46.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 shutdown_connections 2024-04-22T01:18:46.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f20d00678c0 0x7f20d0069d80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f01455b0 0x7f20f01593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f00cc150 0x7f20f01424b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 --2- 172.21.15.70:0/3043954445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f0072060 0x7f20f0164c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 >> 172.21.15.70:0/3043954445 conn(0x7f20f006b1f0 msgr2=0x7f20f013f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:46.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 shutdown_connections 2024-04-22T01:18:46.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.784+0000 7f20f4fef640 1 -- 172.21.15.70:0/3043954445 wait complete. 2024-04-22T01:18:46.800 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-22T01:18:46.801 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-22T01:18:46.801 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:18:46.801 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:46.801 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-22T01:18:46.801 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-22T01:18:46.801 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:18:46.801 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:46.802 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-22T01:18:46.802 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-22T01:18:46.802 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:18:46.802 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:46.802 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-22T01:18:46.802 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-22T01:18:46.803 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:18:46.803 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:18:46.811 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:18:46.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-22T01:18:46.812 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:18:46.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-22T01:18:46.816 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:18:46.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-22T01:18:46.821 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:18:46.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-22T01:18:46.857 INFO:tasks.ceph.mds.c:Started 2024-04-22T01:18:46.857 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:46.858 INFO:tasks.ceph.mds.a:Started 2024-04-22T01:18:46.858 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:46.866 INFO:tasks.ceph.mds.d:Started 2024-04-22T01:18:46.866 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:46.867 INFO:tasks.ceph.mds.b:Started 2024-04-22T01:18:46.867 DEBUG:teuthology.parallel:result is None 2024-04-22T01:18:46.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-22T01:18:46.919 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-22T01:18:46.923 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:18:46.938 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-04-22T01:18:46.940 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-04-22T01:18:46.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 Processor -- start 2024-04-22T01:18:46.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 -- start start 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c1c15bf30 0x7f5c1c0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5c1c099ad0 0x7f5c1c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c0b1340 con 0x7f5c1c0d0010 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5c1c0673a0 con 0x7f5c1c099ad0 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c21c92640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5c1c0b15e0 con 0x7f5c1c15bf30 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c1affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5c1c099ad0 0x7f5c1c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.976+0000 7f5c1affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5c1c099ad0 0x7f5c1c0ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60140/0 (socket says 172.21.15.70:60140) 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1affd640 1 -- 172.21.15.70:0/1001136667 learned_addr learned my addr 172.21.15.70:0/1001136667 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1b7fe640 1 --2- 172.21.15.70:0/1001136667 >> v2:172.21.15.98:3300/0 conn(0x7f5c1c15bf30 0x7f5c1c0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1affd640 1 -- 172.21.15.70:0/1001136667 >> v2:172.21.15.98:3300/0 conn(0x7f5c1c15bf30 msgr2=0x7f5c1c0cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1affd640 1 --2- 172.21.15.70:0/1001136667 >> v2:172.21.15.98:3300/0 conn(0x7f5c1c15bf30 0x7f5c1c0cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1affd640 1 -- 172.21.15.70:0/1001136667 >> v1:172.21.15.98:6789/0 conn(0x7f5c1c0d0010 legacy=0x7f5c1c0af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1affd640 1 -- 172.21.15.70:0/1001136667 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1c0b18c0 con 0x7f5c1c099ad0 2024-04-22T01:18:46.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1b7fe640 1 --2- 172.21.15.70:0/1001136667 >> v2:172.21.15.98:3300/0 conn(0x7f5c1c15bf30 0x7f5c1c0cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:46.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1affd640 1 --2- 172.21.15.70:0/1001136667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c099ad0 0x7f5c1c0ccae0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0b5680 tx=0x7f5c0c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0d1a1a4627e838 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1a7fc640 1 -- 172.21.15.70:0/1001136667 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0b0070 con 0x7f5c1c099ad0 2024-04-22T01:18:46.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1a7fc640 1 -- 172.21.15.70:0/1001136667 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c0c0ab600 con 0x7f5c1c099ad0 2024-04-22T01:18:46.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c1a7fc640 1 -- 172.21.15.70:0/1001136667 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0ab9d0 con 0x7f5c1c099ad0 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 -- 172.21.15.70:0/1001136667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c099ad0 msgr2=0x7f5c1c0ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1001136667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c099ad0 0x7f5c1c0ccae0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0b5680 tx=0x7f5c0c01c520 comp rx=0 tx=0).stop 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 -- 172.21.15.70:0/1001136667 shutdown_connections 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1001136667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c099ad0 0x7f5c1c0ccae0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1001136667 >> v2:172.21.15.98:3300/0 conn(0x7f5c1c15bf30 0x7f5c1c0cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 -- 172.21.15.70:0/1001136667 >> 172.21.15.70:0/1001136667 conn(0x7f5c1c0b3c80 msgr2=0x7f5c1c0b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 -- 172.21.15.70:0/1001136667 shutdown_connections 2024-04-22T01:18:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.980+0000 7f5c21c92640 1 -- 172.21.15.70:0/1001136667 wait complete. 2024-04-22T01:18:46.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 Processor -- start 2024-04-22T01:18:46.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- start start 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c1c099ad0 0x7f5c1c148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 0x7f5c1c14f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c1c15bf30 0x7f5c1c14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1c0cd1a0 con 0x7f5c1c0d0010 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1c0cd020 con 0x7f5c1c15bf30 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5c1c169310 con 0x7f5c1c099ad0 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 0x7f5c1c14f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 0x7f5c1c14f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60146/0 (socket says 172.21.15.70:60146) 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 -- 172.21.15.70:0/1050500020 learned_addr learned my addr 172.21.15.70:0/1050500020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1b7fe640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c1c099ad0 0x7f5c1c148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1bfff640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c1c15bf30 0x7f5c1c14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 -- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c1c099ad0 msgr2=0x7f5c1c148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c1c099ad0 0x7f5c1c148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 -- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c1c15bf30 msgr2=0x7f5c1c14b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c1c15bf30 0x7f5c1c14b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 -- 172.21.15.70:0/1050500020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1c157df0 con 0x7f5c1c0d0010 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1affd640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 0x7f5c1c14f1f0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c01ca00 tx=0x7f5c0c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0b0040 con 0x7f5c1c0d0010 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c0c0b4630 con 0x7f5c1c0d0010 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1c1580d0 con 0x7f5c1c0d0010 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0b4a20 con 0x7f5c1c0d0010 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c1c158690 con 0x7f5c1c0d0010 2024-04-22T01:18:46.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c1c04f510 con 0x7f5c1c0d0010 2024-04-22T01:18:46.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5c0c0ab4c0 con 0x7f5c1c0d0010 2024-04-22T01:18:46.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c13fff640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5bfc0678c0 0x7f5bfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:46.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5335+0+0 (secure 0 0 0) 0x7f5c0c11f5e0 con 0x7f5c1c0d0010 2024-04-22T01:18:46.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.984+0000 7f5c1b7fe640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5bfc0678c0 0x7f5bfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:46.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.988+0000 7f5c1b7fe640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5bfc0678c0 0x7f5bfc069d80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009950 tx=0x7f5c04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:46.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:46.992+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5c0c016110 con 0x7f5c1c0d0010 2024-04-22T01:18:47.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:47.152+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5c1c074640 con 0x7f5c1c0d0010 2024-04-22T01:18:48.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c13fff640 1 -- 172.21.15.70:0/1050500020 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v99) ==== 97+0+0 (secure 0 0 0) 0x7f5c0c0efcd0 con 0x7f5c1c0d0010 2024-04-22T01:18:48.253 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-22T01:18:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5bfc0678c0 msgr2=0x7f5bfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5bfc0678c0 0x7f5bfc069d80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009950 tx=0x7f5c04008040 comp rx=0 tx=0).stop 2024-04-22T01:18:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 msgr2=0x7f5c1c14f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 0x7f5c1c14f1f0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c01ca00 tx=0x7f5c0c0192c0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 shutdown_connections 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5bfc0678c0 0x7f5bfc069d80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c1c15bf30 0x7f5c1c14b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c1c0d0010 0x7f5c1c14f1f0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 --2- 172.21.15.70:0/1050500020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c1c099ad0 0x7f5c1c148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.251+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 >> 172.21.15.70:0/1050500020 conn(0x7f5c1c0b3c80 msgr2=0x7f5c1c06e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.255+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 shutdown_connections 2024-04-22T01:18:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.255+0000 7f5c21c92640 1 -- 172.21.15.70:0/1050500020 wait complete. 2024-04-22T01:18:48.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-22T01:18:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.399+0000 7ff628c6b640 1 Processor -- start 2024-04-22T01:18:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.399+0000 7ff628c6b640 1 -- start start 2024-04-22T01:18:48.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff628c6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 0x7ff624071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff628c6b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6240cd7b0 con 0x7ff6241455b0 2024-04-22T01:18:48.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff628c6b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6240cc710 con 0x7ff624072060 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff628c6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff6240cc9f0 con 0x7ff6240cc150 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff6241455b0 0x7ff6240cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49822/0 (socket says 172.21.15.70:49822) 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62259c640 1 -- 172.21.15.70:0/1593935932 learned_addr learned my addr 172.21.15.70:0/1593935932 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690117350 0 0) 0x7ff6240cd7b0 con 0x7ff6241455b0 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff60c003530 con 0x7ff6241455b0 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 0x7ff624071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1702516445 0 0) 0x7ff60c003530 con 0x7ff6241455b0 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 msgr2=0x7ff624071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 0x7ff624071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 >> v1:172.21.15.98:6789/0 conn(0x7ff624072060 legacy=0x7ff62406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6240cccd0 con 0x7ff6241455b0 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7ff614001680 con 0x7ff6241455b0 2024-04-22T01:18:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 >> v1:172.21.15.70:6789/0 conn(0x7ff6241455b0 legacy=0x7ff6240cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 0x7ff60c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff60c004020 0x7ff60c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff60c004da0 0x7ff60c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6240cc9f0 con 0x7ff60c004020 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c00bcb0 con 0x7ff60c004da0 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff60c00beb0 con 0x7ff6240cc150 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff60c004020 0x7ff60c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff62259c640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 0x7ff60c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 -- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 msgr2=0x7ff60c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6240cc150 0x7ff60c0039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 -- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff60c004da0 msgr2=0x7ff60c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff622d9d640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff60c004da0 0x7ff60c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff60c004da0 0x7ff60c005220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.403+0000 7ff621d9b640 1 -- 172.21.15.70:0/1593935932 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60c003530 con 0x7ff60c004020 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff621d9b640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff60c004020 0x7ff60c004810 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ff61800ba00 tx=0x7ff61801c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80dbecbfc9babd93 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6180b2070 con 0x7ff60c004020 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6180ad410 con 0x7ff60c004020 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff62159a640 1 -- 172.21.15.70:0/1593935932 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61800b600 con 0x7ff60c004020 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- 172.21.15.70:0/1593935932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff60c004020 msgr2=0x7ff60c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1593935932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff60c004020 0x7ff60c004810 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ff61800ba00 tx=0x7ff61801c660 comp rx=0 tx=0).stop 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff622d9d640 1 -- 172.21.15.70:0/1593935932 reap_dead start 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- 172.21.15.70:0/1593935932 shutdown_connections 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- 172.21.15.70:0/1593935932 >> 172.21.15.70:0/1593935932 conn(0x7ff62406b1f0 msgr2=0x7ff62406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- 172.21.15.70:0/1593935932 shutdown_connections 2024-04-22T01:18:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- 172.21.15.70:0/1593935932 wait complete. 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 Processor -- start 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- start start 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff60c004020 0x7ff62415b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62415ee80 0x7ff624165b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 0x7ff6241660b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff624167d50 con 0x7ff62415d870 2024-04-22T01:18:48.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff624072470 con 0x7ff62415ee80 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff628c6b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff624167ed0 con 0x7ff60c004020 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff622d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 0x7ff6241660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff622d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 0x7ff6241660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60154/0 (socket says 172.21.15.70:60154) 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff622d9d640 1 -- 172.21.15.70:0/1384890506 learned_addr learned my addr 172.21.15.70:0/1384890506 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff621d9b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62415ee80 0x7ff624165b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.407+0000 7ff62259c640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff60c004020 0x7ff62415b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff622d9d640 1 -- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff60c004020 msgr2=0x7ff62415b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff622d9d640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff60c004020 0x7ff62415b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff622d9d640 1 -- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62415ee80 msgr2=0x7ff624165b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff622d9d640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62415ee80 0x7ff624165b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff622d9d640 1 -- 172.21.15.70:0/1384890506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff624141700 con 0x7ff62415d870 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff62259c640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff60c004020 0x7ff62415b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff622d9d640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 0x7ff6241660b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7ff624070000 tx=0x7ff61c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c018070 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff61c003b80 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62415dde0 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c005070 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62415e3a0 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff624068210 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff61c01b070 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff612ffd640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6040678c0 0x7ff604069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7ff61c085270 con 0x7ff62415d870 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff62259c640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6040678c0 0x7ff604069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.411+0000 7ff62259c640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6040678c0 0x7ff604069d80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff614000c10 tx=0x7ff614009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.415+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff61c055920 con 0x7ff62415d870 2024-04-22T01:18:48.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.615+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff62415f510 con 0x7ff62415d870 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.615+0000 7ff612ffd640 1 -- 172.21.15.70:0/1384890506 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) ==== 80+0+5304 (secure 0 0 0) 0x7ff61c059c20 con 0x7ff62415d870 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmusATHRAAvELcY3TCmjcJ2CUrvJVmHw==", 2024-04-22T01:18:48.619 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmTTb7KxAAutDCQxFRRgdq6Eo+3CaG6w==", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmXkizJRAAeT7d8i60aMgK8hhpJs/hpQ==", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm6MiEMRAAtrmxeoOxEBPDIw7XuwMxzA==", 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.621 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDKuSVm7bREDRAADO7fl0UHmLZiKhn9Y5E0lA==", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDNuSVmkKmNERAAuwDuGnjbYDORmfNss9vOtA==", 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDQuSVmqdmEFxAAgApFlVXICq2U0WOl+y+Rlg==", 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.623 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-22T01:18:48.624 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDTuSVmq+P8HBAA2Axno1Yw4ZBVS5IX5r/Acg==", 2024-04-22T01:18:48.625 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.625 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.625 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.625 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDZuSVmgPPWHBAAVJ++EZvnajn5hhS6Xo2Ong==", 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.626 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDcuSVmXLcKIxAA5tne9qPEjYS5COr54jABKQ==", 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDfuSVm/WcFKhAAmW6r5yiNjmAFoYjEQThmjA==", 2024-04-22T01:18:48.627 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDiuSVm8kwGMBAA+oj+sZb/vizWwWqJWMC8tQ==", 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.628 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw==", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow r", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmCw93AxAASgS7senwAP4BBYtiOCcLbQ==", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:18:48.629 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlnDEABAA3wH6HXhSdLu/DS0UZp2FBA==", 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDJuSVmlpCjCBAAITDf5B7b6xtnqwerF61Iuw==", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm2O5GBRAA/JxaogwD4Hjmtr9eQvMY0g==", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-22T01:18:48.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVmRTEmGRAAupLbFRAPYnoEe3Mxk+BPvw==", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQDIuSVm5amRFhAAHs7EMrC6FfhxpyC80FqGYg==", 2024-04-22T01:18:48.632 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-22T01:18:48.633 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6040678c0 msgr2=0x7ff604069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6040678c0 0x7ff604069d80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff614000c10 tx=0x7ff614009040 comp rx=0 tx=0).stop 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 msgr2=0x7ff6241660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 0x7ff6241660b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7ff624070000 tx=0x7ff61c00dc00 comp rx=0 tx=0).stop 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 shutdown_connections 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff6040678c0 0x7ff604069d80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff62415d870 0x7ff6241660b0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62415ee80 0x7ff624165b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 --2- 172.21.15.70:0/1384890506 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff60c004020 0x7ff62415b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 >> 172.21.15.70:0/1384890506 conn(0x7ff62406b1f0 msgr2=0x7ff6240cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:48.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 shutdown_connections 2024-04-22T01:18:48.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.619+0000 7ff628c6b640 1 -- 172.21.15.70:0/1384890506 wait complete. 2024-04-22T01:18:48.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.743+0000 7fb0bd1a2640 1 Processor -- start 2024-04-22T01:18:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.743+0000 7fb0bd1a2640 1 -- start start 2024-04-22T01:18:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0bd1a2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0b815aa30 0x7fb0b80b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0bd1a2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 0x7fb0b80b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0bd1a2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0b806dbd0 con 0x7fb0b815bb50 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0bd1a2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b806de70 con 0x7fb0b815aa30 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0bd1a2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b806e150 con 0x7fb0b80aee40 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb0b815bb50 0x7fb0b80ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49832/0 (socket says 172.21.15.70:49832) 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 -- 172.21.15.70:0/1654440847 learned_addr learned my addr 172.21.15.70:0/1654440847 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b6d9d640 1 --2- 172.21.15.70:0/1654440847 >> v2:172.21.15.98:3300/0 conn(0x7fb0b815aa30 0x7fb0b80b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485640695 0 0) 0x7fb0b806dbd0 con 0x7fb0b815bb50 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a0003530 con 0x7fb0b815bb50 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b759e640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 0x7fb0b80b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4073545512 0 0) 0x7fb0a0003530 con 0x7fb0b815bb50 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 msgr2=0x7fb0b80b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 0x7fb0b80b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 >> v2:172.21.15.98:3300/0 conn(0x7fb0b815aa30 msgr2=0x7fb0b80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 --2- 172.21.15.70:0/1654440847 >> v2:172.21.15.98:3300/0 conn(0x7fb0b815aa30 0x7fb0b80b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b806e430 con 0x7fb0b815bb50 2024-04-22T01:18:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fb0ac001680 con 0x7fb0b815bb50 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 >> v1:172.21.15.70:6789/0 conn(0x7fb0b815bb50 legacy=0x7fb0b80ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 0x7fb0a00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0a0003fb0 0x7fb0a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0a0004d50 0x7fb0a00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b806de70 con 0x7fb0a0003fb0 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b806e150 con 0x7fb0a0004d50 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb0a000ba50 con 0x7fb0b80aee40 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b6d9d640 1 --2- 172.21.15.70:0/1654440847 >> v2:172.21.15.98:3300/0 conn(0x7fb0b815aa30 0x7fb0b80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0a0003fb0 0x7fb0a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 -- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 msgr2=0x7fb0a00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b80aee40 0x7fb0a00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 -- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0a0004d50 msgr2=0x7fb0a00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0a0004d50 0x7fb0a00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 -- 172.21.15.70:0/1654440847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a0003530 con 0x7fb0a0003fb0 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b659c640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0a0003fb0 0x7fb0a0004810 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005150 tx=0x7fb0ac01fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba2267c3693a325 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac004d50 con 0x7fb0a0003fb0 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ac005b00 con 0x7fb0a0003fb0 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.747+0000 7fb0b5d9b640 1 -- 172.21.15.70:0/1654440847 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac008110 con 0x7fb0a0003fb0 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/1654440847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0a0003fb0 msgr2=0x7fb0a0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/1654440847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0a0003fb0 0x7fb0a0004810 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005150 tx=0x7fb0ac01fad0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 -- 172.21.15.70:0/1654440847 reap_dead start 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/1654440847 shutdown_connections 2024-04-22T01:18:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/1654440847 >> 172.21.15.70:0/1654440847 conn(0x7fb0b806b1f0 msgr2=0x7fb0b806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/1654440847 shutdown_connections 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/1654440847 wait complete. 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 Processor -- start 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- start start 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b8147320 0x7fb0b8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0b8147700 0x7fb0b8072060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 0x7fb0b81549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b815ae10 con 0x7fb0b8154530 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b80b4d00 con 0x7fb0b8147700 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b815af90 con 0x7fb0b8147320 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 0x7fb0b81549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 0x7fb0b81549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60174/0 (socket says 172.21.15.70:60174) 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 -- 172.21.15.70:0/3734147718 learned_addr learned my addr 172.21.15.70:0/3734147718 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b659c640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0b8147700 0x7fb0b8072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b6d9d640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b8147320 0x7fb0b8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 -- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b8147320 msgr2=0x7fb0b8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b8147320 0x7fb0b8071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 -- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0b8147700 msgr2=0x7fb0b8072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0b8147700 0x7fb0b8072060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 -- 172.21.15.70:0/3734147718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b81530a0 con 0x7fb0b8154530 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b659c640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0b8147700 0x7fb0b8072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0b759e640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 0x7fb0b81549b0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8072ea0 tx=0x7fb0b000d760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00125f0 con 0x7fb0b8154530 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b0019070 con 0x7fb0b8154530 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b8153380 con 0x7fb0b8154530 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00129f0 con 0x7fb0b8154530 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b8153940 con 0x7fb0b8154530 2024-04-22T01:18:48.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.751+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b8068210 con 0x7fb0b8154530 2024-04-22T01:18:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.755+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0b0012b90 con 0x7fb0b8154530 2024-04-22T01:18:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.755+0000 7fb09f7fe640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0a40678f0 0x7fb0a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.755+0000 7fb0b6d9d640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0a40678f0 0x7fb0a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.755+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7fb0b0085840 con 0x7fb0b8154530 2024-04-22T01:18:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.755+0000 7fb0b6d9d640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0a40678f0 0x7fb0a4069db0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80070b0 tx=0x7fb0a800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:48.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.759+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb0b001b310 con 0x7fb0b8154530 2024-04-22T01:18:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.915+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0b806f190 con 0x7fb0b8154530 2024-04-22T01:18:48.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.915+0000 7fb09f7fe640 1 -- 172.21.15.70:0/3734147718 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+13129 (secure 0 0 0) 0x7fb0b00565d0 con 0x7fb0b8154530 2024-04-22T01:18:48.920 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:48.920 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":99,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:48.253139+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0a40678f0 msgr2=0x7fb0a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0a40678f0 0x7fb0a4069db0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80070b0 tx=0x7fb0a800b040 comp rx=0 tx=0).stop 2024-04-22T01:18:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 msgr2=0x7fb0b81549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 0x7fb0b81549b0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8072ea0 tx=0x7fb0b000d760 comp rx=0 tx=0).stop 2024-04-22T01:18:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 shutdown_connections 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb0a40678f0 0x7fb0a4069db0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b8154530 0x7fb0b81549b0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0b8147700 0x7fb0b8072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 --2- 172.21.15.70:0/3734147718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0b8147320 0x7fb0b8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 >> 172.21.15.70:0/3734147718 conn(0x7fb0b806b1f0 msgr2=0x7fb0b8155c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 shutdown_connections 2024-04-22T01:18:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:48.919+0000 7fb0bd1a2640 1 -- 172.21.15.70:0/3734147718 wait complete. 2024-04-22T01:18:48.939 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:18:48.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:18:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.047+0000 7fcaf93b8640 1 Processor -- start 2024-04-22T01:18:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.047+0000 7fcaf93b8640 1 -- start start 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcaf40b4450 0x7fcaf4071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcaf40cd220 0x7fcaf40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf415c290 con 0x7fcaf40720b0 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf415c570 con 0x7fcaf40cd220 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf415c850 con 0x7fcaf40b4450 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcaf40cd220 0x7fcaf40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcaf40cd220 0x7fcaf40b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60184/0 (socket says 172.21.15.70:60184) 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 -- 172.21.15.70:0/2689641942 learned_addr learned my addr 172.21.15.70:0/2689641942 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 -- 172.21.15.70:0/2689641942 >> v2:172.21.15.98:3300/0 conn(0x7fcaf40b4450 msgr2=0x7fcaf4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 --2- 172.21.15.70:0/2689641942 >> v2:172.21.15.98:3300/0 conn(0x7fcaf40b4450 0x7fcaf4071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 -- 172.21.15.70:0/2689641942 >> v1:172.21.15.98:6789/0 conn(0x7fcaf40720b0 legacy=0x7fcaf40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 -- 172.21.15.70:0/2689641942 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf415cb30 con 0x7fcaf40cd220 2024-04-22T01:18:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf37fe640 1 --2- 172.21.15.70:0/2689641942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40cd220 0x7fcaf40b2a40 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcaf406fce0 tx=0x7fcaec01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9735493e79e48e36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf1ffb640 1 -- 172.21.15.70:0/2689641942 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0a5670 con 0x7fcaf40cd220 2024-04-22T01:18:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf1ffb640 1 -- 172.21.15.70:0/2689641942 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaec0a5810 con 0x7fcaf40cd220 2024-04-22T01:18:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf1ffb640 1 -- 172.21.15.70:0/2689641942 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0a5be0 con 0x7fcaf40cd220 2024-04-22T01:18:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- 172.21.15.70:0/2689641942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40cd220 msgr2=0x7fcaf40b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/2689641942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40cd220 0x7fcaf40b2a40 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcaf406fce0 tx=0x7fcaec01c9b0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- 172.21.15.70:0/2689641942 shutdown_connections 2024-04-22T01:18:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/2689641942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40cd220 0x7fcaf40b2a40 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/2689641942 >> v2:172.21.15.98:3300/0 conn(0x7fcaf40b4450 0x7fcaf4071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- 172.21.15.70:0/2689641942 >> 172.21.15.70:0/2689641942 conn(0x7fcaf40b1d40 msgr2=0x7fcaf40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- 172.21.15.70:0/2689641942 shutdown_connections 2024-04-22T01:18:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.051+0000 7fcaf93b8640 1 -- 172.21.15.70:0/2689641942 wait complete. 2024-04-22T01:18:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 Processor -- start 2024-04-22T01:18:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- start start 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcaf40720b0 0x7fcaf4145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 0x7fcaf415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40cd220 0x7fcaf415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf40b3910 con 0x7fcaf40b4450 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf40b3790 con 0x7fcaf40cd220 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf4168b10 con 0x7fcaf40720b0 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 0x7fcaf415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 0x7fcaf415ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60196/0 (socket says 172.21.15.70:60196) 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 -- 172.21.15.70:0/1948559897 learned_addr learned my addr 172.21.15.70:0/1948559897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 -- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcaf40720b0 msgr2=0x7fcaf4145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcaf40720b0 0x7fcaf4145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 -- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40cd220 msgr2=0x7fcaf415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf37fe640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40cd220 0x7fcaf415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40cd220 0x7fcaf415b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 -- 172.21.15.70:0/1948559897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf415e800 con 0x7fcaf40b4450 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf27fc640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 0x7fcaf415ac20 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c490 tx=0x7fcae8001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8017070 con 0x7fcaf40b4450 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae8003600 con 0x7fcaf40b4450 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf41f6fa0 con 0x7fcaf40b4450 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8003a30 con 0x7fcaf40b4450 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf41f74f0 con 0x7fcaf40b4450 2024-04-22T01:18:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.055+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf4068210 con 0x7fcaf40b4450 2024-04-22T01:18:49.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.059+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcae801a070 con 0x7fcaf40b4450 2024-04-22T01:18:49.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.059+0000 7fcadb7fe640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcae00678f0 0x7fcae0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.059+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7fcae8084e70 con 0x7fcaf40b4450 2024-04-22T01:18:49.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.059+0000 7fcaf2ffd640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcae00678f0 0x7fcae0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.059+0000 7fcaf2ffd640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcae00678f0 0x7fcae0069db0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009730 tx=0x7fcae4006ca0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.063+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcae8055c00 con 0x7fcaf40b4450 2024-04-22T01:18:49.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.211+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcaf40ce3c0 con 0x7fcaf40b4450 2024-04-22T01:18:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.211+0000 7fcadb7fe640 1 -- 172.21.15.70:0/1948559897 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7fcae8055500 con 0x7fcaf40b4450 2024-04-22T01:18:49.215 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:18:49.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcae00678f0 msgr2=0x7fcae0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcae00678f0 0x7fcae0069db0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009730 tx=0x7fcae4006ca0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 msgr2=0x7fcaf415ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 0x7fcaf415ac20 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c490 tx=0x7fcae8001e80 comp rx=0 tx=0).stop 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 shutdown_connections 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fcae00678f0 0x7fcae0069db0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcaf40cd220 0x7fcaf415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcaf40b4450 0x7fcaf415ac20 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 --2- 172.21.15.70:0/1948559897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcaf40720b0 0x7fcaf4145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 >> 172.21.15.70:0/1948559897 conn(0x7fcaf40b1d40 msgr2=0x7fcaf414fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 shutdown_connections 2024-04-22T01:18:49.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.215+0000 7fcaf93b8640 1 -- 172.21.15.70:0/1948559897 wait complete. 2024-04-22T01:18:49.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:18:49.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc5d57e640 1 Processor -- start 2024-04-22T01:18:49.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc5d57e640 1 -- start start 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc5d57e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdc5809a130 0x7fdc5806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc5d57e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc58161920 con 0x7fdc58099cc0 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc5d57e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc5806c300 con 0x7fdc58161540 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc5d57e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdc580cdea0 con 0x7fdc5809a130 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdc5809a130 0x7fdc5806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdc5809a130 0x7fdc5806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60212/0 (socket says 172.21.15.70:60212) 2024-04-22T01:18:49.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc4e7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdc58099cc0 0x7fdc5806cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49848/0 (socket says 172.21.15.70:49848) 2024-04-22T01:18:49.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 -- 172.21.15.70:0/2935768954 learned_addr learned my addr 172.21.15.70:0/2935768954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:49.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 -- 172.21.15.70:0/2935768954 >> v1:172.21.15.98:6789/0 conn(0x7fdc58161540 legacy=0x7fdc580cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:49.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc567fc640 1 -- 172.21.15.70:0/2935768954 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047393831 0 0) 0x7fdc58161920 con 0x7fdc58099cc0 2024-04-22T01:18:49.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 -- 172.21.15.70:0/2935768954 >> v1:172.21.15.70:6789/0 conn(0x7fdc58099cc0 legacy=0x7fdc5806cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 -- 172.21.15.70:0/2935768954 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc580ce180 con 0x7fdc5809a130 2024-04-22T01:18:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.351+0000 7fdc577fe640 1 --2- 172.21.15.70:0/2935768954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc5809a130 0x7fdc5806d4d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009e50 tx=0x7fdc5001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7de77189d5505e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc567fc640 1 -- 172.21.15.70:0/2935768954 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc500a58d0 con 0x7fdc5809a130 2024-04-22T01:18:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc567fc640 1 -- 172.21.15.70:0/2935768954 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc500a5a70 con 0x7fdc5809a130 2024-04-22T01:18:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc567fc640 1 -- 172.21.15.70:0/2935768954 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc500a5e40 con 0x7fdc5809a130 2024-04-22T01:18:49.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/2935768954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc5809a130 msgr2=0x7fdc5806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/2935768954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc5809a130 0x7fdc5806d4d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009e50 tx=0x7fdc5001ca00 comp rx=0 tx=0).stop 2024-04-22T01:18:49.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/2935768954 shutdown_connections 2024-04-22T01:18:49.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/2935768954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc5809a130 0x7fdc5806d4d0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/2935768954 >> 172.21.15.70:0/2935768954 conn(0x7fdc580b35d0 msgr2=0x7fdc580b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:49.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/2935768954 shutdown_connections 2024-04-22T01:18:49.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/2935768954 wait complete. 2024-04-22T01:18:49.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 Processor -- start 2024-04-22T01:18:49.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- start start 2024-04-22T01:18:49.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc58099cc0 0x7fdc580cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc5809a130 0x7fdc580b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 0x7fdc580c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc580cdaf0 con 0x7fdc58161540 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5809a540 con 0x7fdc5809a130 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdc580b7910 con 0x7fdc58099cc0 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 0x7fdc580c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 0x7fdc580c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60224/0 (socket says 172.21.15.70:60224) 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 -- 172.21.15.70:0/92912712 learned_addr learned my addr 172.21.15.70:0/92912712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc4e7fc640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc5809a130 0x7fdc580b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc56ffd640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc58099cc0 0x7fdc580cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 -- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc58099cc0 msgr2=0x7fdc580cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc58099cc0 0x7fdc580cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 -- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc5809a130 msgr2=0x7fdc580b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc5809a130 0x7fdc580b46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 -- 172.21.15.70:0/92912712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5814ecd0 con 0x7fdc58161540 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc4e7fc640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc5809a130 0x7fdc580b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc577fe640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 0x7fdc580c9c00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fdc50019610 tx=0x7fdc5806e220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc500bc020 con 0x7fdc58161540 2024-04-22T01:18:49.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc500b8470 con 0x7fdc58161540 2024-04-22T01:18:49.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc580b7350 con 0x7fdc58161540 2024-04-22T01:18:49.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc500a58d0 con 0x7fdc58161540 2024-04-22T01:18:49.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.355+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc5814d220 con 0x7fdc58161540 2024-04-22T01:18:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.359+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc5806e650 con 0x7fdc58161540 2024-04-22T01:18:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.359+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc500bf020 con 0x7fdc58161540 2024-04-22T01:18:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.359+0000 7fdc4ffff640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc380678c0 0x7fdc38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.359+0000 7fdc56ffd640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc380678c0 0x7fdc38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.359+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7fdc50120f20 con 0x7fdc58161540 2024-04-22T01:18:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.359+0000 7fdc56ffd640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc380678c0 0x7fdc38069d80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdc48000f70 tx=0x7fdc48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.363+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdc50016620 con 0x7fdc58161540 2024-04-22T01:18:49.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.511+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdc580cfe30 con 0x7fdc58161540 2024-04-22T01:18:49.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.511+0000 7fdc4ffff640 1 -- 172.21.15.70:0/92912712 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) ==== 84+0+65 (secure 0 0 0) 0x7fdc500f1e90 con 0x7fdc58161540 2024-04-22T01:18:49.516 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:49.516 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-22T01:18:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-22T01:18:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-22T01:18:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-22T01:18:49.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:49.517 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-22T01:18:49.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc380678c0 msgr2=0x7fdc38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc380678c0 0x7fdc38069d80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdc48000f70 tx=0x7fdc48009040 comp rx=0 tx=0).stop 2024-04-22T01:18:49.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 msgr2=0x7fdc580c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 0x7fdc580c9c00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fdc50019610 tx=0x7fdc5806e220 comp rx=0 tx=0).stop 2024-04-22T01:18:49.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 shutdown_connections 2024-04-22T01:18:49.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdc380678c0 0x7fdc38069d80 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc58161540 0x7fdc580c9c00 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc5809a130 0x7fdc580b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 --2- 172.21.15.70:0/92912712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc58099cc0 0x7fdc580cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 >> 172.21.15.70:0/92912712 conn(0x7fdc580b35d0 msgr2=0x7fdc580d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:49.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 shutdown_connections 2024-04-22T01:18:49.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.515+0000 7fdc5d57e640 1 -- 172.21.15.70:0/92912712 wait complete. 2024-04-22T01:18:49.535 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:18:49.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:18:49.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 Processor -- start 2024-04-22T01:18:49.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 -- start start 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f58b00b4450 0x7f58b0071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58b00cd220 0x7f58b00b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b015c290 con 0x7f58b00720b0 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f58b015c570 con 0x7f58b00b4450 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58b53e0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f58b015c850 con 0x7f58b00cd220 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f58b00b4450 0x7f58b0071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f58b00b4450 0x7f58b0071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60240/0 (socket says 172.21.15.70:60240) 2024-04-22T01:18:49.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 -- 172.21.15.70:0/478783762 learned_addr learned my addr 172.21.15.70:0/478783762 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:49.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 -- 172.21.15.70:0/478783762 >> v2:172.21.15.98:3300/0 conn(0x7f58b00cd220 msgr2=0x7f58b00b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:49.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58af7fe640 1 --2- 172.21.15.70:0/478783762 >> v2:172.21.15.98:3300/0 conn(0x7f58b00cd220 0x7f58b00b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 --2- 172.21.15.70:0/478783762 >> v2:172.21.15.98:3300/0 conn(0x7f58b00cd220 0x7f58b00b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 -- 172.21.15.70:0/478783762 >> v1:172.21.15.98:6789/0 conn(0x7f58b00720b0 legacy=0x7f58b00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:49.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.651+0000 7f58aeffd640 1 -- 172.21.15.70:0/478783762 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b015cb30 con 0x7f58b00b4450 2024-04-22T01:18:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58aeffd640 1 --2- 172.21.15.70:0/478783762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b0071b70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f58b015f6b0 tx=0x7f58a001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e1f68e4c7d49b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58ae7fc640 1 -- 172.21.15.70:0/478783762 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a00b0070 con 0x7f58b00b4450 2024-04-22T01:18:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58ae7fc640 1 -- 172.21.15.70:0/478783762 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a00ab600 con 0x7f58b00b4450 2024-04-22T01:18:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58ae7fc640 1 -- 172.21.15.70:0/478783762 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a00ab9d0 con 0x7f58b00b4450 2024-04-22T01:18:49.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- 172.21.15.70:0/478783762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 msgr2=0x7f58b0071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 --2- 172.21.15.70:0/478783762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b0071b70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f58b015f6b0 tx=0x7f58a001c4e0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- 172.21.15.70:0/478783762 shutdown_connections 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 --2- 172.21.15.70:0/478783762 >> v2:172.21.15.98:3300/0 conn(0x7f58b00cd220 0x7f58b00b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 --2- 172.21.15.70:0/478783762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b0071b70 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- 172.21.15.70:0/478783762 >> 172.21.15.70:0/478783762 conn(0x7f58b00b1d40 msgr2=0x7f58b00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- 172.21.15.70:0/478783762 shutdown_connections 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- 172.21.15.70:0/478783762 wait complete. 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 Processor -- start 2024-04-22T01:18:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- start start 2024-04-22T01:18:49.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58b00720b0 0x7f58b014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b01495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58b00cd220 0x7f58b0149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58b00b3910 con 0x7f58b00b4450 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58b00b3790 con 0x7f58b00720b0 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58b53e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f58b006d8f0 con 0x7f58b00cd220 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.655+0000 7f58a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b01495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b01495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60254/0 (socket says 172.21.15.70:60254) 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 -- 172.21.15.70:0/2130636176 learned_addr learned my addr 172.21.15.70:0/2130636176 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58aeffd640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58b00720b0 0x7f58b014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 -- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58b00cd220 msgr2=0x7f58b0149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58af7fe640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58b00cd220 0x7f58b0149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58b00cd220 0x7f58b0149af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 -- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58b00720b0 msgr2=0x7f58b014cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58b00720b0 0x7f58b014cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 -- 172.21.15.70:0/2130636176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b014d1c0 con 0x7f58b00b4450 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58aeffd640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58b00720b0 0x7f58b014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a7fff640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b01495b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f589800c450 tx=0x7f589800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898012830 con 0x7f58b00b4450 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5898017070 con 0x7f58b00b4450 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b014d4a0 con 0x7f58b00b4450 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898012c30 con 0x7f58b00b4450 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b014da60 con 0x7f58b00b4450 2024-04-22T01:18:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b0068210 con 0x7f58b00b4450 2024-04-22T01:18:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f589801c460 con 0x7f58b00b4450 2024-04-22T01:18:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a77fe640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f58900678c0 0x7f5890069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7f5898085880 con 0x7f58b00b4450 2024-04-22T01:18:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58aeffd640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f58900678c0 0x7f5890069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.659+0000 7f58aeffd640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f58900678c0 0x7f5890069d80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f58a00ad000 tx=0x7f58a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.663+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5898056580 con 0x7f58b00b4450 2024-04-22T01:18:49.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:49.827+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f58b00cc190 con 0x7f58b00b4450 2024-04-22T01:18:50.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58a77fe640 1 -- 172.21.15.70:0/2130636176 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) ==== 138+0+0 (secure 0 0 0) 0x7f5898055e80 con 0x7f58b00b4450 2024-04-22T01:18:50.258 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f58900678c0 msgr2=0x7f5890069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f58900678c0 0x7f5890069d80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f58a00ad000 tx=0x7f58a0008040 comp rx=0 tx=0).stop 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 msgr2=0x7f58b01495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b01495b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f589800c450 tx=0x7f589800c910 comp rx=0 tx=0).stop 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 shutdown_connections 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f58900678c0 0x7f5890069d80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58b00cd220 0x7f58b0149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58b00b4450 0x7f58b01495b0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.255+0000 7f58b53e0640 1 --2- 172.21.15.70:0/2130636176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58b00720b0 0x7f58b014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.259+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 >> 172.21.15.70:0/2130636176 conn(0x7f58b00b1d40 msgr2=0x7f58b015d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.259+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 shutdown_connections 2024-04-22T01:18:50.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.259+0000 7f58b53e0640 1 -- 172.21.15.70:0/2130636176 wait complete. 2024-04-22T01:18:50.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:18:50.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21cd239640 1 Processor -- start 2024-04-22T01:18:50.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21cd239640 1 -- start start 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21cd239640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f21c806efb0 0x7f21c806f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21cd239640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21c80d1a70 con 0x7f21c806b1f0 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21cd239640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21c80d0cb0 con 0x7f21c8161970 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21cd239640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f21c80cd570 con 0x7f21c806efb0 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f21c806efb0 0x7f21c806f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f21c806efb0 0x7f21c806f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60258/0 (socket says 172.21.15.70:60258) 2024-04-22T01:18:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c659c640 1 -- 172.21.15.70:0/4089149543 learned_addr learned my addr 172.21.15.70:0/4089149543 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c659c640 1 -- 172.21.15.70:0/4089149543 >> v1:172.21.15.98:6789/0 conn(0x7f21c8161970 legacy=0x7f21c8161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c659c640 1 -- 172.21.15.70:0/4089149543 >> v1:172.21.15.70:6789/0 conn(0x7f21c806b1f0 legacy=0x7f21c806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c659c640 1 -- 172.21.15.70:0/4089149543 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c80d0f90 con 0x7f21c806efb0 2024-04-22T01:18:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.403+0000 7f21c5d9b640 1 -- 172.21.15.70:0/4089149543 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2814203034 0 0) 0x7f21c80d1a70 con 0x7f21c806b1f0 2024-04-22T01:18:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21c659c640 1 --2- 172.21.15.70:0/4089149543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c806efb0 0x7f21c806f410 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009870 tx=0x7f21b801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc575a7bc4a0e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21c5d9b640 1 -- 172.21.15.70:0/4089149543 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b80b0070 con 0x7f21c806efb0 2024-04-22T01:18:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21c5d9b640 1 -- 172.21.15.70:0/4089149543 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21b80ab600 con 0x7f21c806efb0 2024-04-22T01:18:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21c5d9b640 1 -- 172.21.15.70:0/4089149543 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b80ab9d0 con 0x7f21c806efb0 2024-04-22T01:18:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 -- 172.21.15.70:0/4089149543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c806efb0 msgr2=0x7f21c806f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:50.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 --2- 172.21.15.70:0/4089149543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c806efb0 0x7f21c806f410 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009870 tx=0x7f21b801c520 comp rx=0 tx=0).stop 2024-04-22T01:18:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 -- 172.21.15.70:0/4089149543 shutdown_connections 2024-04-22T01:18:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 --2- 172.21.15.70:0/4089149543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c806efb0 0x7f21c806f410 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 -- 172.21.15.70:0/4089149543 >> 172.21.15.70:0/4089149543 conn(0x7f21c80733e0 msgr2=0x7f21c8099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 -- 172.21.15.70:0/4089149543 shutdown_connections 2024-04-22T01:18:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 -- 172.21.15.70:0/4089149543 wait complete. 2024-04-22T01:18:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 Processor -- start 2024-04-22T01:18:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.407+0000 7f21cd239640 1 -- start start 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c806b1f0 0x7f21c8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c806efb0 0x7f21c80b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 0x7f21c8160b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21c816c530 con 0x7f21c8161970 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21c80d15e0 con 0x7f21c806efb0 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f21c816c6b0 con 0x7f21c806b1f0 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 0x7f21c8160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 0x7f21c8160b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60268/0 (socket says 172.21.15.70:60268) 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 -- 172.21.15.70:0/3549684447 learned_addr learned my addr 172.21.15.70:0/3549684447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c659c640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c806efb0 0x7f21c80b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c6d9d640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c806b1f0 0x7f21c8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 -- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c806b1f0 msgr2=0x7f21c8155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c806b1f0 0x7f21c8155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 -- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c806efb0 msgr2=0x7f21c80b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c806efb0 0x7f21c80b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 -- 172.21.15.70:0/3549684447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c8156210 con 0x7f21c8161970 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c759e640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 0x7f21c8160b90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c780 tx=0x7f21c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0004b70 con 0x7f21c8161970 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c0004d10 con 0x7f21c8161970 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c815faf0 con 0x7f21c8161970 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c000aa30 con 0x7f21c8161970 2024-04-22T01:18:50.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c81600b0 con 0x7f21c8161970 2024-04-22T01:18:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c8068920 con 0x7f21c8161970 2024-04-22T01:18:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21c000abd0 con 0x7f21c8161970 2024-04-22T01:18:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21bf7fe640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21a80678c0 0x7f21a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5619+0+0 (secure 0 0 0) 0x7f21c0085e40 con 0x7f21c8161970 2024-04-22T01:18:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c6d9d640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21a80678c0 0x7f21a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.411+0000 7f21c6d9d640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21a80678c0 0x7f21a8069d80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000f70 tx=0x7f21b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.415+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f21c0020070 con 0x7f21c8161970 2024-04-22T01:18:50.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:50.575+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f21c80cdc30 con 0x7f21c8161970 2024-04-22T01:18:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.259+0000 7f21bf7fe640 1 -- 172.21.15.70:0/3549684447 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) ==== 170+0+0 (secure 0 0 0) 0x7f21c8156210 con 0x7f21c8161970 2024-04-22T01:18:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-22T01:18:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21a80678c0 msgr2=0x7f21a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21a80678c0 0x7f21a8069d80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000f70 tx=0x7f21b0009040 comp rx=0 tx=0).stop 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 msgr2=0x7f21c8160b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 0x7f21c8160b90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c780 tx=0x7f21c000cc40 comp rx=0 tx=0).stop 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 shutdown_connections 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f21a80678c0 0x7f21a8069d80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21c8161970 0x7f21c8160b90 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c806efb0 0x7f21c80b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 --2- 172.21.15.70:0/3549684447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c806b1f0 0x7f21c8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 >> 172.21.15.70:0/3549684447 conn(0x7f21c80733e0 msgr2=0x7f21c80cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 shutdown_connections 2024-04-22T01:18:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.263+0000 7f21cd239640 1 -- 172.21.15.70:0/3549684447 wait complete. 2024-04-22T01:18:51.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:18:51.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510c240640 1 Processor -- start 2024-04-22T01:18:51.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510c240640 1 -- start start 2024-04-22T01:18:51.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510c240640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 0x7f5104067340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510c240640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51040b5060 con 0x7f51040b0450 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510c240640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51040b5340 con 0x7f510415e0e0 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510c240640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f51040b6360 con 0x7f510406c900 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510a7dd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f51040b0450 0x7f51040ad1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49878/0 (socket says 172.21.15.70:49878) 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510a7dd640 1 -- 172.21.15.70:0/3472607090 learned_addr learned my addr 172.21.15.70:0/3472607090 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3476737710 0 0) 0x7f51040b5060 con 0x7f51040b0450 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50f0003530 con 0x7f51040b0450 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 0x7f5104067340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1676767071 0 0) 0x7f50f0003530 con 0x7f51040b0450 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 msgr2=0x7f5104067340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 0x7f5104067340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 >> v1:172.21.15.98:6789/0 conn(0x7f510415e0e0 legacy=0x7f51040afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51040b6080 con 0x7f51040b0450 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f50fc001a00 con 0x7f51040b0450 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 >> v1:172.21.15.70:6789/0 conn(0x7f51040b0450 legacy=0x7f51040ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 0x7f50f00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f0003f70 0x7f50f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f0004d50 0x7f50f00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50f000bbe0 con 0x7f50f0003f70 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50f000bd90 con 0x7f50f0004d50 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50f000bf90 con 0x7f510406c900 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f0003f70 0x7f50f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f5109fdc640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 0x7f50f00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f510a7dd640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f0004d50 0x7f50f00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 -- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 msgr2=0x7f50f00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:51.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510406c900 0x7f50f00039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 -- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f0004d50 msgr2=0x7f50f00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:51.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50f0004d50 0x7f50f00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.411+0000 7f51097db640 1 -- 172.21.15.70:0/3472607090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f0003530 con 0x7f50f0003f70 2024-04-22T01:18:51.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f51097db640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f0003f70 0x7f50f0004810 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009a90 tx=0x7f50f401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6949762306da063 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:51.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4009690 con 0x7f50f0003f70 2024-04-22T01:18:51.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f40a6410 con 0x7f50f0003f70 2024-04-22T01:18:51.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f5108fda640 1 -- 172.21.15.70:0/3472607090 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f40a67e0 con 0x7f50f0003f70 2024-04-22T01:18:51.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 -- 172.21.15.70:0/3472607090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f0003f70 msgr2=0x7f50f0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 --2- 172.21.15.70:0/3472607090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f0003f70 0x7f50f0004810 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009a90 tx=0x7f50f401ca30 comp rx=0 tx=0).stop 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510a7dd640 1 -- 172.21.15.70:0/3472607090 reap_dead start 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 -- 172.21.15.70:0/3472607090 shutdown_connections 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 -- 172.21.15.70:0/3472607090 >> 172.21.15.70:0/3472607090 conn(0x7f5104099a60 msgr2=0x7f51040b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 -- 172.21.15.70:0/3472607090 shutdown_connections 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 -- 172.21.15.70:0/3472607090 wait complete. 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 Processor -- start 2024-04-22T01:18:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.415+0000 7f510c240640 1 -- start start 2024-04-22T01:18:51.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f0003f70 0x7f5104155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51041561f0 0x7f51041554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 0x7f51041637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5104169cb0 con 0x7f5104160a90 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51040d0c00 con 0x7f51041561f0 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5104169e30 con 0x7f50f0003f70 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 0x7f51041637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 0x7f51041637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60286/0 (socket says 172.21.15.70:60286) 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 -- 172.21.15.70:0/1684804447 learned_addr learned my addr 172.21.15.70:0/1684804447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f5109fdc640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f0003f70 0x7f5104155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f51097db640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51041561f0 0x7f51041554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 -- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f0003f70 msgr2=0x7f5104155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f0003f70 0x7f5104155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 -- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51041561f0 msgr2=0x7f51041554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51041561f0 0x7f51041554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:51.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 -- 172.21.15.70:0/1684804447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5104160f90 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510a7dd640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 0x7f51041637b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f510406db50 tx=0x7f50fc00e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc00eb70 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50fc00ed30 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510415f3c0 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc014670 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f510415fea0 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5104160130 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50fc014810 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f50fa7fc640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50e80678c0 0x7f50e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6001+0+0 (secure 0 0 0) 0x7f50fc085bb0 con 0x7f5104160a90 2024-04-22T01:18:51.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f5109fdc640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50e80678c0 0x7f50e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:51.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.419+0000 7f5109fdc640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50e80678c0 0x7f50e8069d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5104164c50 tx=0x7f5100009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:51.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.423+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f50fc0565c0 con 0x7f5104160a90 2024-04-22T01:18:51.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:51.619+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5104068920 con 0x7f5104160a90 2024-04-22T01:18:52.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.287+0000 7f50fa7fc640 1 -- 172.21.15.70:0/1684804447 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v48) ==== 174+0+0 (secure 0 0 0) 0x7f50fc089070 con 0x7f5104160a90 2024-04-22T01:18:52.289 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 15 and data pool 16 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50e80678c0 msgr2=0x7f50e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50e80678c0 0x7f50e8069d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5104164c50 tx=0x7f5100009040 comp rx=0 tx=0).stop 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 msgr2=0x7f51041637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 0x7f51041637b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f510406db50 tx=0x7f50fc00e160 comp rx=0 tx=0).stop 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 shutdown_connections 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50e80678c0 0x7f50e8069d80 secure :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5104164c50 tx=0x7f5100009040 comp rx=0 tx=0).stop 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5104160a90 0x7f51041637b0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51041561f0 0x7f51041554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 --2- 172.21.15.70:0/1684804447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50f0003f70 0x7f5104155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.291+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 >> 172.21.15.70:0/1684804447 conn(0x7f5104099a60 msgr2=0x7f51040ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:52.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.295+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 shutdown_connections 2024-04-22T01:18:52.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.295+0000 7f510c240640 1 -- 172.21.15.70:0/1684804447 wait complete. 2024-04-22T01:18:52.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:52.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.451+0000 7fc2ffae7640 1 Processor -- start 2024-04-22T01:18:52.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.451+0000 7fc2ffae7640 1 -- start start 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2ffae7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f80cc150 0x7fc2f8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2ffae7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2f8072060 0x7fc2f806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2ffae7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f80cd7b0 con 0x7fc2f81455b0 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2ffae7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f80cc710 con 0x7fc2f8072060 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2ffae7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80cc9f0 con 0x7fc2f80cc150 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2f8072060 0x7fc2f806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2f8072060 0x7fc2f806e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60298/0 (socket says 172.21.15.70:60298) 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 -- 172.21.15.70:0/1165818639 learned_addr learned my addr 172.21.15.70:0/1165818639 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fd883640 1 --1- 172.21.15.70:0/1165818639 >> v1:172.21.15.70:6789/0 conn(0x7fc2f81455b0 0x7fc2f80cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49892/0 (socket says 172.21.15.70:49892) 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fd082640 1 --2- 172.21.15.70:0/1165818639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f80cc150 0x7fc2f8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 -- 172.21.15.70:0/1165818639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f80cc150 msgr2=0x7fc2f8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 --2- 172.21.15.70:0/1165818639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f80cc150 0x7fc2f8071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 -- 172.21.15.70:0/1165818639 >> v1:172.21.15.70:6789/0 conn(0x7fc2f81455b0 legacy=0x7fc2f80cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 -- 172.21.15.70:0/1165818639 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f80cccd0 con 0x7fc2f8072060 2024-04-22T01:18:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fc881640 1 -- 172.21.15.70:0/1165818639 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399145122 0 0) 0x7fc2f80cd7b0 con 0x7fc2f81455b0 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fe084640 1 --2- 172.21.15.70:0/1165818639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f8072060 0x7fc2f806e200 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009870 tx=0x7fc2f001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a84c112192ca3b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fc881640 1 -- 172.21.15.70:0/1165818639 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00b0070 con 0x7fc2f8072060 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fc881640 1 -- 172.21.15.70:0/1165818639 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f00ab600 con 0x7fc2f8072060 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.455+0000 7fc2fc881640 1 -- 172.21.15.70:0/1165818639 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00ab9d0 con 0x7fc2f8072060 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/1165818639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f8072060 msgr2=0x7fc2f806e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/1165818639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f8072060 0x7fc2f806e200 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009870 tx=0x7fc2f001c440 comp rx=0 tx=0).stop 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/1165818639 shutdown_connections 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/1165818639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f8072060 0x7fc2f806e200 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/1165818639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f80cc150 0x7fc2f8071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/1165818639 >> 172.21.15.70:0/1165818639 conn(0x7fc2f806b1f0 msgr2=0x7fc2f806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:52.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/1165818639 shutdown_connections 2024-04-22T01:18:52.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/1165818639 wait complete. 2024-04-22T01:18:52.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 Processor -- start 2024-04-22T01:18:52.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- start start 2024-04-22T01:18:52.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f8072060 0x7fc2f8142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f80cc150 0x7fc2f813b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 0x7fc2f815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f806f2e0 con 0x7fc2f81455b0 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f806f160 con 0x7fc2f80cc150 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f8168b40 con 0x7fc2f8072060 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 0x7fc2f815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 0x7fc2f815b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60306/0 (socket says 172.21.15.70:60306) 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 -- 172.21.15.70:0/2201907959 learned_addr learned my addr 172.21.15.70:0/2201907959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fd082640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f80cc150 0x7fc2f813b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fd883640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f8072060 0x7fc2f8142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 -- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f8072060 msgr2=0x7fc2f8142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f8072060 0x7fc2f8142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 -- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f80cc150 msgr2=0x7fc2f813b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f80cc150 0x7fc2f813b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 -- 172.21.15.70:0/2201907959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f8143420 con 0x7fc2f81455b0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2fe084640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 0x7fc2f815b5a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fc2f001a070 tx=0x7fc2f00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00b0040 con 0x7fc2f81455b0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f00b9070 con 0x7fc2f81455b0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f8143700 con 0x7fc2f81455b0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00b3860 con 0x7fc2f81455b0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.459+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f8143cc0 con 0x7fc2f81455b0 2024-04-22T01:18:52.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.463+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f8068210 con 0x7fc2f81455b0 2024-04-22T01:18:52.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.463+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2f00b3a00 con 0x7fc2f81455b0 2024-04-22T01:18:52.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.463+0000 7fc2ee7fc640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2dc0678c0 0x7fc2dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.463+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6101+0+0 (secure 0 0 0) 0x7fc2f011fdb0 con 0x7fc2f81455b0 2024-04-22T01:18:52.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.463+0000 7fc2fd883640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2dc0678c0 0x7fc2dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.463+0000 7fc2fd883640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2dc0678c0 0x7fc2dc069d80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8000f70 tx=0x7fc2e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:52.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.467+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc2f0016150 con 0x7fc2f81455b0 2024-04-22T01:18:52.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.631+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2f815f510 con 0x7fc2f81455b0 2024-04-22T01:18:52.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.631+0000 7fc2ee7fc640 1 -- 172.21.15.70:0/2201907959 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16861 (secure 0 0 0) 0x7fc2f00f0760 con 0x7fc2f81455b0 2024-04-22T01:18:52.633 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:52.633 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":102,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:51.626230+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2dc0678c0 msgr2=0x7fc2dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2dc0678c0 0x7fc2dc069d80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8000f70 tx=0x7fc2e8009040 comp rx=0 tx=0).stop 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 msgr2=0x7fc2f815b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 0x7fc2f815b5a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fc2f001a070 tx=0x7fc2f00191e0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 shutdown_connections 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2dc0678c0 0x7fc2dc069d80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2f81455b0 0x7fc2f815b5a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f80cc150 0x7fc2f813b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 --2- 172.21.15.70:0/2201907959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2f8072060 0x7fc2f8142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 >> 172.21.15.70:0/2201907959 conn(0x7fc2f806b1f0 msgr2=0x7fc2f8074530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:52.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 shutdown_connections 2024-04-22T01:18:52.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.635+0000 7fc2ffae7640 1 -- 172.21.15.70:0/2201907959 wait complete. 2024-04-22T01:18:52.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:52.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.767+0000 7fc76f4f3640 1 Processor -- start 2024-04-22T01:18:52.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.767+0000 7fc76f4f3640 1 -- start start 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76f4f3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7680d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76f4f3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc7680cdee0 0x7fc7680b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76f4f3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc768145950 con 0x7fc7680b2ba0 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76f4f3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc7680b2ff0 con 0x7fc7680cdee0 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76f4f3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc768071420 con 0x7fc768145570 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc7680b2ba0 0x7fc76806b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49908/0 (socket says 172.21.15.70:49908) 2024-04-22T01:18:52.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 -- 172.21.15.70:0/1141100215 learned_addr learned my addr 172.21.15.70:0/1141100215 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76ca8e640 1 --2- 172.21.15.70:0/1141100215 >> v2:172.21.15.98:3300/0 conn(0x7fc7680cdee0 0x7fc7680b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76d28f640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7680d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665625034 0 0) 0x7fc7600014a0 con 0x7fc7680b2ba0 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc754003530 con 0x7fc7680b2ba0 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2366653717 0 0) 0x7fc754003530 con 0x7fc7680b2ba0 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 msgr2=0x7fc7680d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7680d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 >> v2:172.21.15.98:3300/0 conn(0x7fc7680cdee0 msgr2=0x7fc7680b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 --2- 172.21.15.70:0/1141100215 >> v2:172.21.15.98:3300/0 conn(0x7fc7680cdee0 0x7fc7680b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7680d06f0 con 0x7fc7680b2ba0 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fc760001bd0 con 0x7fc7680b2ba0 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76d28f640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7680d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 >> v1:172.21.15.70:6789/0 conn(0x7fc7680b2ba0 legacy=0x7fc76806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7540039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc754003f00 0x7fc754004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc754004d50 0x7fc7540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7680b2ff0 con 0x7fc754004d50 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc768071420 con 0x7fc754003f00 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc75400ba00 con 0x7fc768145570 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc754004d50 0x7fc7540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76d28f640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7540039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76ca8e640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc754003f00 0x7fc754004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 -- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 msgr2=0x7fc7540039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc768145570 0x7fc7540039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 -- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc754003f00 msgr2=0x7fc754004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc754003f00 0x7fc754004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76da90640 1 -- 172.21.15.70:0/1141100215 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc754003530 con 0x7fc754004d50 2024-04-22T01:18:52.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.771+0000 7fc76ca8e640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc754003f00 0x7fc754004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:52.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76da90640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc754004d50 0x7fc7540051d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fc76000aa70 tx=0x7fc76001d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40e2c908cea3d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76000a690 con 0x7fc754004d50 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600a9070 con 0x7fc754004d50 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc75ffff640 1 -- 172.21.15.70:0/1141100215 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600ae910 con 0x7fc754004d50 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 -- 172.21.15.70:0/1141100215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc754004d50 msgr2=0x7fc7540051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/1141100215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc754004d50 0x7fc7540051d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fc76000aa70 tx=0x7fc76001d3c0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76da90640 1 -- 172.21.15.70:0/1141100215 reap_dead start 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 -- 172.21.15.70:0/1141100215 shutdown_connections 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 -- 172.21.15.70:0/1141100215 >> 172.21.15.70:0/1141100215 conn(0x7fc76806c300 msgr2=0x7fc76806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 -- 172.21.15.70:0/1141100215 shutdown_connections 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 -- 172.21.15.70:0/1141100215 wait complete. 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 Processor -- start 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.775+0000 7fc76f4f3640 1 -- start start 2024-04-22T01:18:52.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc76815f230 0x7fc76815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc76813fe60 0x7fc7681402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 0x7fc768138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc76815ff20 con 0x7fc768138080 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc76815fc40 con 0x7fc76813fe60 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc768160200 con 0x7fc76815f230 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 0x7fc768138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 0x7fc768138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60322/0 (socket says 172.21.15.70:60322) 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 -- 172.21.15.70:0/4207734198 learned_addr learned my addr 172.21.15.70:0/4207734198 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76d28f640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc76815f230 0x7fc76815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76ca8e640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc76813fe60 0x7fc7681402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 -- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc76815f230 msgr2=0x7fc76815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc76815f230 0x7fc76815d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 -- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc76813fe60 msgr2=0x7fc7681402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc76813fe60 0x7fc7681402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 -- 172.21.15.70:0/4207734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc768140820 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76d28f640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc76815f230 0x7fc76815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76da90640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 0x7fc768138500 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc76000aa70 tx=0x7fc754005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76000a690 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600b4070 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76814ac80 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600aecb0 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76814b240 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc768068210 con 0x7fc768138080 2024-04-22T01:18:52.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc7600b6020 con 0x7fc768138080 2024-04-22T01:18:52.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc75dffb640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc74c0678c0 0x7fc74c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:52.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6101+0+0 (secure 0 0 0) 0x7fc76011fb70 con 0x7fc768138080 2024-04-22T01:18:52.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76d28f640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc74c0678c0 0x7fc74c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:52.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.779+0000 7fc76d28f640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc74c0678c0 0x7fc74c069d80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc76400c850 tx=0x7fc764008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:52.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.783+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc768140820 con 0x7fc768138080 2024-04-22T01:18:52.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.939+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc76815bec0 con 0x7fc768138080 2024-04-22T01:18:52.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.943+0000 7fc75dffb640 1 -- 172.21.15.70:0/4207734198 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16861 (secure 0 0 0) 0x7fc768145950 con 0x7fc768138080 2024-04-22T01:18:52.943 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:52.944 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":102,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:51.626230+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc74c0678c0 msgr2=0x7fc74c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc74c0678c0 0x7fc74c069d80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc76400c850 tx=0x7fc764008040 comp rx=0 tx=0).stop 2024-04-22T01:18:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 msgr2=0x7fc768138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 0x7fc768138500 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc76000aa70 tx=0x7fc754005840 comp rx=0 tx=0).stop 2024-04-22T01:18:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 shutdown_connections 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc74c0678c0 0x7fc74c069d80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc768138080 0x7fc768138500 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc76813fe60 0x7fc7681402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 --2- 172.21.15.70:0/4207734198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc76815f230 0x7fc76815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 >> 172.21.15.70:0/4207734198 conn(0x7fc76806c300 msgr2=0x7fc768142d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 shutdown_connections 2024-04-22T01:18:52.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:52.947+0000 7fc76f4f3640 1 -- 172.21.15.70:0/4207734198 wait complete. 2024-04-22T01:18:52.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:18:53.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 Processor -- start 2024-04-22T01:18:53.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 -- start start 2024-04-22T01:18:53.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd100b4450 0x7fcd10071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcd100720b0 0x7fcd100ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd1015c290 con 0x7fcd100cd050 2024-04-22T01:18:53.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcd100b3a50 con 0x7fcd100720b0 2024-04-22T01:18:53.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd15386640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1015c570 con 0x7fcd100b4450 2024-04-22T01:18:53.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0e7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcd100720b0 0x7fcd100ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcd100cd050 0x7fcd1015f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49922/0 (socket says 172.21.15.70:49922) 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0f7fe640 1 -- 172.21.15.70:0/4147693863 learned_addr learned my addr 172.21.15.70:0/4147693863 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0effd640 1 --2- 172.21.15.70:0/4147693863 >> v2:172.21.15.98:3300/0 conn(0x7fcd100b4450 0x7fcd10071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0e7fc640 1 -- 172.21.15.70:0/4147693863 >> v2:172.21.15.98:3300/0 conn(0x7fcd100b4450 msgr2=0x7fcd10071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0e7fc640 1 --2- 172.21.15.70:0/4147693863 >> v2:172.21.15.98:3300/0 conn(0x7fcd100b4450 0x7fcd10071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0e7fc640 1 -- 172.21.15.70:0/4147693863 >> v1:172.21.15.70:6789/0 conn(0x7fcd100cd050 legacy=0x7fcd1015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0e7fc640 1 -- 172.21.15.70:0/4147693863 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1015c850 con 0x7fcd100720b0 2024-04-22T01:18:53.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.087+0000 7fcd0e7fc640 1 --2- 172.21.15.70:0/4147693863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100720b0 0x7fcd100ccb10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcd1006fce0 tx=0x7fcd0401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8badd622550412 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:53.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4147693863 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd040b0070 con 0x7fcd100720b0 2024-04-22T01:18:53.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4147693863 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd040ab600 con 0x7fcd100720b0 2024-04-22T01:18:53.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4147693863 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd040ab9d0 con 0x7fcd100720b0 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- 172.21.15.70:0/4147693863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100720b0 msgr2=0x7fcd100ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 --2- 172.21.15.70:0/4147693863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100720b0 0x7fcd100ccb10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcd1006fce0 tx=0x7fcd0401c480 comp rx=0 tx=0).stop 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- 172.21.15.70:0/4147693863 shutdown_connections 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 --2- 172.21.15.70:0/4147693863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100720b0 0x7fcd100ccb10 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 --2- 172.21.15.70:0/4147693863 >> v2:172.21.15.98:3300/0 conn(0x7fcd100b4450 0x7fcd10071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- 172.21.15.70:0/4147693863 >> 172.21.15.70:0/4147693863 conn(0x7fcd100b1d40 msgr2=0x7fcd100cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- 172.21.15.70:0/4147693863 shutdown_connections 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- 172.21.15.70:0/4147693863 wait complete. 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 Processor -- start 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- start start 2024-04-22T01:18:53.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd100720b0 0x7fcd1014bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd100b4450 0x7fcd10151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 0x7fcd10151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd100b49b0 con 0x7fcd100cd050 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd100b4830 con 0x7fcd100720b0 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd15386640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd100cd800 con 0x7fcd100b4450 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 0x7fcd10151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 0x7fcd10151980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60330/0 (socket says 172.21.15.70:60330) 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 -- 172.21.15.70:0/3642251919 learned_addr learned my addr 172.21.15.70:0/3642251919 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:53.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0e7fc640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd100b4450 0x7fcd10151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0effd640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd100720b0 0x7fcd1014bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 -- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd100b4450 msgr2=0x7fcd10151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd100b4450 0x7fcd10151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 -- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd100720b0 msgr2=0x7fcd1014bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd100720b0 0x7fcd1014bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 -- 172.21.15.70:0/3642251919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1006da70 con 0x7fcd100cd050 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0effd640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd100720b0 0x7fcd1014bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:53.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fcd0f7fe640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 0x7fcd10151980 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcd0800cbc0 tx=0x7fcd0800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:53.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08017070 con 0x7fcd100cd050 2024-04-22T01:18:53.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0800ace0 con 0x7fcd100cd050 2024-04-22T01:18:53.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.091+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08012600 con 0x7fcd100cd050 2024-04-22T01:18:53.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd101fb0f0 con 0x7fcd100cd050 2024-04-22T01:18:53.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1006e010 con 0x7fcd100cd050 2024-04-22T01:18:53.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd10068210 con 0x7fcd100cd050 2024-04-22T01:18:53.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd0801a070 con 0x7fcd100cd050 2024-04-22T01:18:53.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fccf77fe640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccfc0678c0 0x7fccfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6101+0+0 (secure 0 0 0) 0x7fcd08085a40 con 0x7fcd100cd050 2024-04-22T01:18:53.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fcd0effd640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccfc0678c0 0x7fccfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.095+0000 7fcd0effd640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccfc0678c0 0x7fccfc069d80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009840 tx=0x7fcd00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:53.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.099+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fcd08055da0 con 0x7fcd100cd050 2024-04-22T01:18:53.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.295+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fcd100d01e0 con 0x7fcd100cd050 2024-04-22T01:18:53.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fccf77fe640 1 -- 172.21.15.70:0/3642251919 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) ==== 118+0+0 (secure 0 0 0) 0x7fcd0805a0a0 con 0x7fcd100cd050 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccfc0678c0 msgr2=0x7fccfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccfc0678c0 0x7fccfc069d80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009840 tx=0x7fcd00008040 comp rx=0 tx=0).stop 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 msgr2=0x7fcd10151980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 0x7fcd10151980 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcd0800cbc0 tx=0x7fcd0800a430 comp rx=0 tx=0).stop 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 shutdown_connections 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fccfc0678c0 0x7fccfc069d80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd100cd050 0x7fcd10151980 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd100b4450 0x7fcd10151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 --2- 172.21.15.70:0/3642251919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd100720b0 0x7fcd1014bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 >> 172.21.15.70:0/3642251919 conn(0x7fcd100b1d40 msgr2=0x7fcd100cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 shutdown_connections 2024-04-22T01:18:53.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.315+0000 7fcd15386640 1 -- 172.21.15.70:0/3642251919 wait complete. 2024-04-22T01:18:53.333 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-22T01:18:53.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-22T01:18:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 Processor -- start 2024-04-22T01:18:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 -- start start 2024-04-22T01:18:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93015a320 0x7fe93006f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe930071b70 0x7fe930071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe93006da70 con 0x7fe9300b3d60 2024-04-22T01:18:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe930072510 con 0x7fe930071b70 2024-04-22T01:18:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe9362b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe93006dd50 con 0x7fe93015a320 2024-04-22T01:18:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93015a320 0x7fe93006f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe930071b70 0x7fe930071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe934850640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe9300b3d60 0x7fe9300b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:35682/0 (socket says 172.21.15.70:35682) 2024-04-22T01:18:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe934850640 1 -- 172.21.15.70:0/3236921444 learned_addr learned my addr 172.21.15.70:0/3236921444 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92effd640 1 -- 172.21.15.70:0/3236921444 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609509432 0 0) 0x7fe93006da70 con 0x7fe9300b3d60 2024-04-22T01:18:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92effd640 1 -- 172.21.15.70:0/3236921444 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe918003530 con 0x7fe9300b3d60 2024-04-22T01:18:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92ffff640 1 -- 172.21.15.70:0/3236921444 >> v2:172.21.15.98:3300/0 conn(0x7fe930071b70 msgr2=0x7fe930071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92ffff640 1 --2- 172.21.15.70:0/3236921444 >> v2:172.21.15.98:3300/0 conn(0x7fe930071b70 0x7fe930071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92ffff640 1 -- 172.21.15.70:0/3236921444 >> v1:172.21.15.98:6789/0 conn(0x7fe9300b3d60 legacy=0x7fe9300b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.451+0000 7fe92ffff640 1 -- 172.21.15.70:0/3236921444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93006e030 con 0x7fe93015a320 2024-04-22T01:18:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe92ffff640 1 --2- 172.21.15.70:0/3236921444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93015a320 0x7fe93006f150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe9200097c0 tx=0x7fe92001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c05df717b3fd38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:53.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe92effd640 1 -- 172.21.15.70:0/3236921444 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200a7070 con 0x7fe93015a320 2024-04-22T01:18:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe92effd640 1 -- 172.21.15.70:0/3236921444 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9200b0070 con 0x7fe93015a320 2024-04-22T01:18:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe92effd640 1 -- 172.21.15.70:0/3236921444 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200ab870 con 0x7fe93015a320 2024-04-22T01:18:53.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 -- 172.21.15.70:0/3236921444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93015a320 msgr2=0x7fe93006f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 --2- 172.21.15.70:0/3236921444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93015a320 0x7fe93006f150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe9200097c0 tx=0x7fe92001c470 comp rx=0 tx=0).stop 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 -- 172.21.15.70:0/3236921444 shutdown_connections 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 --2- 172.21.15.70:0/3236921444 >> v2:172.21.15.98:3300/0 conn(0x7fe930071b70 0x7fe930071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 --2- 172.21.15.70:0/3236921444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93015a320 0x7fe93006f150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 -- 172.21.15.70:0/3236921444 >> 172.21.15.70:0/3236921444 conn(0x7fe9300b1d40 msgr2=0x7fe9300cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 -- 172.21.15.70:0/3236921444 shutdown_connections 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.455+0000 7fe9362b3640 1 -- 172.21.15.70:0/3236921444 wait complete. 2024-04-22T01:18:53.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 Processor -- start 2024-04-22T01:18:53.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- start start 2024-04-22T01:18:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe930071b70 0x7fe930151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9300b3d60 0x7fe93015cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 0x7fe93015d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe9300cf5a0 con 0x7fe93015a320 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9300cf420 con 0x7fe9300b3d60 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe93006f690 con 0x7fe930071b70 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 0x7fe93015d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 0x7fe93015d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60336/0 (socket says 172.21.15.70:60336) 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 -- 172.21.15.70:0/1269615786 learned_addr learned my addr 172.21.15.70:0/1269615786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92f7fe640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9300b3d60 0x7fe93015cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92ffff640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe930071b70 0x7fe930151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 -- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe930071b70 msgr2=0x7fe930151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe930071b70 0x7fe930151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 -- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9300b3d60 msgr2=0x7fe93015cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9300b3d60 0x7fe93015cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 -- 172.21.15.70:0/1269615786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9301578e0 con 0x7fe93015a320 2024-04-22T01:18:53.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe934850640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 0x7fe93015d070 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe92800cb70 tx=0x7fe92800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928017070 con 0x7fe93015a320 2024-04-22T01:18:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe92800ad40 con 0x7fe93015a320 2024-04-22T01:18:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928012620 con 0x7fe93015a320 2024-04-22T01:18:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93014d160 con 0x7fe93015a320 2024-04-22T01:18:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe93014d720 con 0x7fe93015a320 2024-04-22T01:18:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe930068210 con 0x7fe93015a320 2024-04-22T01:18:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe92801a070 con 0x7fe93015a320 2024-04-22T01:18:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.459+0000 7fe92cff9640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe91c0678c0 0x7fe91c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.463+0000 7fe92ffff640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe91c0678c0 0x7fe91c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.463+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fe928085bf0 con 0x7fe93015a320 2024-04-22T01:18:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.463+0000 7fe92ffff640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe91c0678c0 0x7fe91c069d80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe9200af000 tx=0x7fe9200b6040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.467+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe9301578e0 con 0x7fe93015a320 2024-04-22T01:18:53.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:53.647+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe9300b43e0 con 0x7fe93015a320 2024-04-22T01:18:54.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.323+0000 7fe92cff9640 1 -- 172.21.15.70:0/1269615786 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) ==== 122+0+0 (secure 0 0 0) 0x7fe928056630 con 0x7fe93015a320 2024-04-22T01:18:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe91c0678c0 msgr2=0x7fe91c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe91c0678c0 0x7fe91c069d80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe9200af000 tx=0x7fe9200b6040 comp rx=0 tx=0).stop 2024-04-22T01:18:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 msgr2=0x7fe93015d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 0x7fe93015d070 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe92800cb70 tx=0x7fe92800a430 comp rx=0 tx=0).stop 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 shutdown_connections 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe91c0678c0 0x7fe91c069d80 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93015a320 0x7fe93015d070 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9300b3d60 0x7fe93015cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 --2- 172.21.15.70:0/1269615786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe930071b70 0x7fe930151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 >> 172.21.15.70:0/1269615786 conn(0x7fe9300b1d40 msgr2=0x7fe9300b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 shutdown_connections 2024-04-22T01:18:54.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.327+0000 7fe9362b3640 1 -- 172.21.15.70:0/1269615786 wait complete. 2024-04-22T01:18:54.344 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-22T01:18:54.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:54.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.459+0000 7f10cbada640 1 Processor -- start 2024-04-22T01:18:54.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.459+0000 7f10cbada640 1 -- start start 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10cbada640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f10c415aa30 0x7f10c40b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10cbada640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10c40aee40 0x7f10c40b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10cbada640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10c406dbd0 con 0x7f10c415bb50 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10cbada640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f10c406de70 con 0x7f10c415aa30 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10cbada640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f10c406e150 con 0x7f10c40aee40 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f10c415aa30 0x7f10c40b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f10c415aa30 0x7f10c40b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60350/0 (socket says 172.21.15.70:60350) 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 -- 172.21.15.70:0/3178848663 learned_addr learned my addr 172.21.15.70:0/3178848663 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:54.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10ca077640 1 --2- 172.21.15.70:0/3178848663 >> v2:172.21.15.98:3300/0 conn(0x7f10c40aee40 0x7f10c40b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 -- 172.21.15.70:0/3178848663 >> v2:172.21.15.98:3300/0 conn(0x7f10c40aee40 msgr2=0x7f10c40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 --2- 172.21.15.70:0/3178848663 >> v2:172.21.15.98:3300/0 conn(0x7f10c40aee40 0x7f10c40b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 -- 172.21.15.70:0/3178848663 >> v1:172.21.15.98:6789/0 conn(0x7f10c415bb50 legacy=0x7f10c40ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:54.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.463+0000 7f10c9876640 1 -- 172.21.15.70:0/3178848663 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c406e430 con 0x7f10c415aa30 2024-04-22T01:18:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c9876640 1 --2- 172.21.15.70:0/3178848663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c415aa30 0x7f10c40b4720 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f10c406ecd0 tx=0x7f10c001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fcfcb81c748186 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c8874640 1 -- 172.21.15.70:0/3178848663 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c00aa610 con 0x7f10c415aa30 2024-04-22T01:18:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c8874640 1 -- 172.21.15.70:0/3178848663 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c00af070 con 0x7f10c415aa30 2024-04-22T01:18:54.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c8874640 1 -- 172.21.15.70:0/3178848663 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c00aa9d0 con 0x7f10c415aa30 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- 172.21.15.70:0/3178848663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c415aa30 msgr2=0x7f10c40b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 --2- 172.21.15.70:0/3178848663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c415aa30 0x7f10c40b4720 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f10c406ecd0 tx=0x7f10c001c490 comp rx=0 tx=0).stop 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- 172.21.15.70:0/3178848663 shutdown_connections 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 --2- 172.21.15.70:0/3178848663 >> v2:172.21.15.98:3300/0 conn(0x7f10c40aee40 0x7f10c40b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 --2- 172.21.15.70:0/3178848663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c415aa30 0x7f10c40b4720 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- 172.21.15.70:0/3178848663 >> 172.21.15.70:0/3178848663 conn(0x7f10c406b1f0 msgr2=0x7f10c406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- 172.21.15.70:0/3178848663 shutdown_connections 2024-04-22T01:18:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- 172.21.15.70:0/3178848663 wait complete. 2024-04-22T01:18:54.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 Processor -- start 2024-04-22T01:18:54.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- start start 2024-04-22T01:18:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 0x7f10c413a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10c415aa30 0x7f10c40ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c415bb50 0x7f10c40cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10c406ead0 con 0x7f10c40aee40 2024-04-22T01:18:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10c406e950 con 0x7f10c415bb50 2024-04-22T01:18:54.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10cbada640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f10c4165a80 con 0x7f10c415aa30 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c9876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 0x7f10c413a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c9876640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 0x7f10c413a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60362/0 (socket says 172.21.15.70:60362) 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10c9876640 1 -- 172.21.15.70:0/1464011915 learned_addr learned my addr 172.21.15.70:0/1464011915 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.467+0000 7f10ca077640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c415bb50 0x7f10c40cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9075640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10c415aa30 0x7f10c40ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9876640 1 -- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10c415aa30 msgr2=0x7f10c40ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9876640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10c415aa30 0x7f10c40ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9876640 1 -- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c415bb50 msgr2=0x7f10c40cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9876640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c415bb50 0x7f10c40cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9876640 1 -- 172.21.15.70:0/1464011915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c40b20a0 con 0x7f10c40aee40 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9876640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 0x7f10c413a6c0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f10c00ac040 tx=0x7f10c001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c00bd5f0 con 0x7f10c40aee40 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c00bd790 con 0x7f10c40aee40 2024-04-22T01:18:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c40b2380 con 0x7f10c40aee40 2024-04-22T01:18:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c00bdb40 con 0x7f10c40aee40 2024-04-22T01:18:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c413b520 con 0x7f10c40aee40 2024-04-22T01:18:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c40b2550 con 0x7f10c40aee40 2024-04-22T01:18:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10c00b33f0 con 0x7f10c40aee40 2024-04-22T01:18:54.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10ba7fc640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10a80678c0 0x7f10a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9075640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10a80678c0 0x7f10a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f10c0120b60 con 0x7f10c40aee40 2024-04-22T01:18:54.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.471+0000 7f10c9075640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10a80678c0 0x7f10a8069d80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f10c41f7fe0 tx=0x7f10b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:54.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.475+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f10c00c3060 con 0x7f10c40aee40 2024-04-22T01:18:54.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.651+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10c40b2550 con 0x7f10c40aee40 2024-04-22T01:18:54.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10ba7fc640 1 -- 172.21.15.70:0/1464011915 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f10c00a7070 con 0x7f10c40aee40 2024-04-22T01:18:54.653 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:54.653 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:51.626962+0000","modified":"2024-04-22T01:18:53.648845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7629":{"gid":7629,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3578750329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3578750329},{"type":"v1","addr":"172.21.15.70:6837","nonce":3578750329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7629,"qdb_cluster":[7629]},"id":7}]} 2024-04-22T01:18:54.654 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 50 2024-04-22T01:18:54.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10a80678c0 msgr2=0x7f10a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10a80678c0 0x7f10a8069d80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f10c41f7fe0 tx=0x7f10b4008040 comp rx=0 tx=0).stop 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 msgr2=0x7f10c413a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 0x7f10c413a6c0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f10c00ac040 tx=0x7f10c001acd0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 shutdown_connections 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f10a80678c0 0x7f10a8069d80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c415bb50 0x7f10c40cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10c415aa30 0x7f10c40ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 --2- 172.21.15.70:0/1464011915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c40aee40 0x7f10c413a6c0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 >> 172.21.15.70:0/1464011915 conn(0x7f10c406b1f0 msgr2=0x7f10c406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:54.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 shutdown_connections 2024-04-22T01:18:54.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.655+0000 7f10cbada640 1 -- 172.21.15.70:0/1464011915 wait complete. 2024-04-22T01:18:54.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:18:54.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.791+0000 7f317b982640 1 Processor -- start 2024-04-22T01:18:54.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.791+0000 7f317b982640 1 -- start start 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f317b982640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 0x7f317415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f317b982640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317406dbd0 con 0x7f31740aee40 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f317b982640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317406de70 con 0x7f317415aa30 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f317b982640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f317415c090 con 0x7f31740b4720 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f31740aee40 0x7f31740b28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49932/0 (socket says 172.21.15.70:49932) 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 -- 172.21.15.70:0/377752951 learned_addr learned my addr 172.21.15.70:0/377752951 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3179f1f640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 0x7f317415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882502892 0 0) 0x7f317406dbd0 con 0x7f31740aee40 2024-04-22T01:18:54.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3160003530 con 0x7f31740aee40 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2725842797 0 0) 0x7f3160003530 con 0x7f31740aee40 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 msgr2=0x7f317415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 0x7f317415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 >> v1:172.21.15.98:6789/0 conn(0x7f317415aa30 legacy=0x7f31740ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317406e150 con 0x7f31740aee40 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f31640016a0 con 0x7f31740aee40 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 >> v1:172.21.15.70:6789/0 conn(0x7f31740aee40 legacy=0x7f31740b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31600039f0 0x7f3160003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3160004810 0x7f3160004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 0x7f31600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f316000ba80 con 0x7f3160004810 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f316000bc00 con 0x7f31600039f0 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f316000e010 con 0x7f31740b4720 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3160004810 0x7f3160004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3179f1f640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 0x7f31600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f317971e640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31600039f0 0x7f3160003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 -- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 msgr2=0x7f31600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:54.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31740b4720 0x7f31600051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 -- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31600039f0 msgr2=0x7f3160003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:54.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31600039f0 0x7f3160003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.795+0000 7f3178f1d640 1 -- 172.21.15.70:0/377752951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3160003530 con 0x7f3160004810 2024-04-22T01:18:54.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3178f1d640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3160004810 0x7f3160004c90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f317406ffe0 tx=0x7f316401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba467e1aeaeb1111 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:54.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3164007d60 con 0x7f3160004810 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3164005b00 con 0x7f3160004810 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f316bfff640 1 -- 172.21.15.70:0/377752951 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3164007070 con 0x7f3160004810 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/377752951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3160004810 msgr2=0x7f3160004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 --2- 172.21.15.70:0/377752951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3160004810 0x7f3160004c90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f317406ffe0 tx=0x7f316401f370 comp rx=0 tx=0).stop 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 -- 172.21.15.70:0/377752951 reap_dead start 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/377752951 shutdown_connections 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/377752951 >> 172.21.15.70:0/377752951 conn(0x7f317406b1f0 msgr2=0x7f317406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/377752951 shutdown_connections 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/377752951 wait complete. 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 Processor -- start 2024-04-22T01:18:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- start start 2024-04-22T01:18:54.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3160004810 0x7f31740a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31740b0620 0x7f31741485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 0x7f31740b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f317415ae10 con 0x7f31740b0fb0 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31740b4d00 con 0x7f31740b0620 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f317415af90 con 0x7f3160004810 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 0x7f31740b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 0x7f31740b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60374/0 (socket says 172.21.15.70:60374) 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 -- 172.21.15.70:0/1587567993 learned_addr learned my addr 172.21.15.70:0/1587567993 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317971e640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3160004810 0x7f31740a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3178f1d640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31740b0620 0x7f31741485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 -- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3160004810 msgr2=0x7f31740a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3160004810 0x7f31740a86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 -- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31740b0620 msgr2=0x7f31741485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31740b0620 0x7f31741485c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 -- 172.21.15.70:0/1587567993 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31740b1d30 con 0x7f31740b0fb0 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3179f1f640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 0x7f31740b1430 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f3174072ea0 tx=0x7f3170011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:54.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170019020 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3170003620 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31740b2010 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170003a10 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.799+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31740b0aa0 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.803+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3174068210 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.803+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f317001c020 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.803+0000 7f3169ffb640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31580678c0 0x7f3158069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.803+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f3170086150 con 0x7f31740b0fb0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.803+0000 7f317971e640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31580678c0 0x7f3158069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:54.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.803+0000 7f317971e640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31580678c0 0x7f3158069d80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f316c005210 tx=0x7f316c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:54.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.807+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3170023030 con 0x7f31740b0fb0 2024-04-22T01:18:54.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.967+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f317406f190 con 0x7f31740b0fb0 2024-04-22T01:18:54.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.967+0000 7f3169ffb640 1 -- 172.21.15.70:0/1587567993 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7f3170056b90 con 0x7f31740b0fb0 2024-04-22T01:18:54.968 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:54.969 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":103,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:53.308712+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31580678c0 msgr2=0x7f3158069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31580678c0 0x7f3158069d80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f316c005210 tx=0x7f316c009040 comp rx=0 tx=0).stop 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 msgr2=0x7f31740b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 0x7f31740b1430 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f3174072ea0 tx=0x7f3170011920 comp rx=0 tx=0).stop 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 shutdown_connections 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31580678c0 0x7f3158069d80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31740b0fb0 0x7f31740b1430 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31740b0620 0x7f31741485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 --2- 172.21.15.70:0/1587567993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3160004810 0x7f31740a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 >> 172.21.15.70:0/1587567993 conn(0x7f317406b1f0 msgr2=0x7f31740ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 shutdown_connections 2024-04-22T01:18:54.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:54.971+0000 7f317b982640 1 -- 172.21.15.70:0/1587567993 wait complete. 2024-04-22T01:18:54.988 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:18:54.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:18:55.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 Processor -- start 2024-04-22T01:18:55.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 -- start start 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc38815bc70 0x7fc38815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc3880cd740 0x7fc38806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc38806b730 0x7fc3880ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc3880cfd10 con 0x7fc3880cd740 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc38806bb50 con 0x7fc38815bc70 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.107+0000 7fc38ecea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc38815fd10 con 0x7fc38806b730 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc3880cd740 0x7fc38806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc3880cd740 0x7fc38806b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60390/0 (socket says 172.21.15.70:60390) 2024-04-22T01:18:55.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 -- 172.21.15.70:0/2597745868 learned_addr learned my addr 172.21.15.70:0/2597745868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ca86640 1 --2- 172.21.15.70:0/2597745868 >> v2:172.21.15.98:3300/0 conn(0x7fc38815bc70 0x7fc38815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38d287640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc38806b730 0x7fc3880ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 -- 172.21.15.70:0/2597745868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc38806b730 msgr2=0x7fc3880ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc38806b730 0x7fc3880ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 -- 172.21.15.70:0/2597745868 >> v2:172.21.15.98:3300/0 conn(0x7fc38815bc70 msgr2=0x7fc38815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 --2- 172.21.15.70:0/2597745868 >> v2:172.21.15.98:3300/0 conn(0x7fc38815bc70 0x7fc38815f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 -- 172.21.15.70:0/2597745868 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38815c050 con 0x7fc3880cd740 2024-04-22T01:18:55.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38d287640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc38806b730 0x7fc3880ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:55.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc387fff640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3880cd740 0x7fc38806b1f0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc3880ce550 tx=0x7fc37c01c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80babec8c5d7805 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc3877fe640 1 -- 172.21.15.70:0/2597745868 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c0b0070 con 0x7fc3880cd740 2024-04-22T01:18:55.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc3877fe640 1 -- 172.21.15.70:0/2597745868 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c0ab600 con 0x7fc3880cd740 2024-04-22T01:18:55.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc3877fe640 1 -- 172.21.15.70:0/2597745868 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c0ab9d0 con 0x7fc3880cd740 2024-04-22T01:18:55.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 -- 172.21.15.70:0/2597745868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3880cd740 msgr2=0x7fc38806b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3880cd740 0x7fc38806b1f0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc3880ce550 tx=0x7fc37c01c230 comp rx=0 tx=0).stop 2024-04-22T01:18:55.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 -- 172.21.15.70:0/2597745868 shutdown_connections 2024-04-22T01:18:55.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc38806b730 0x7fc3880ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 --2- 172.21.15.70:0/2597745868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3880cd740 0x7fc38806b1f0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 --2- 172.21.15.70:0/2597745868 >> v2:172.21.15.98:3300/0 conn(0x7fc38815bc70 0x7fc38815f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 -- 172.21.15.70:0/2597745868 >> 172.21.15.70:0/2597745868 conn(0x7fc388099a60 msgr2=0x7fc388099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:55.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 -- 172.21.15.70:0/2597745868 shutdown_connections 2024-04-22T01:18:55.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.111+0000 7fc38ecea640 1 -- 172.21.15.70:0/2597745868 wait complete. 2024-04-22T01:18:55.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 Processor -- start 2024-04-22T01:18:55.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 -- start start 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 0x7fc3880c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3880cd740 0x7fc3881f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38815bc70 0x7fc3880c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc3880d0480 con 0x7fc38806b730 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3880cff60 con 0x7fc38815bc70 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc3880d0600 con 0x7fc3880cd740 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 0x7fc3880c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 0x7fc3880c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60398/0 (socket says 172.21.15.70:60398) 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 -- 172.21.15.70:0/288828571 learned_addr learned my addr 172.21.15.70:0/288828571 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38d287640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38815bc70 0x7fc3880c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc387fff640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3880cd740 0x7fc3881f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 -- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3880cd740 msgr2=0x7fc3881f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3880cd740 0x7fc3881f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 -- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38815bc70 msgr2=0x7fc3880c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38815bc70 0x7fc3880c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 -- 172.21.15.70:0/288828571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3880c8280 con 0x7fc38806b730 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc387fff640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3880cd740 0x7fc3881f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ca86640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 0x7fc3880c67f0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc378007e40 tx=0x7fc37800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc378014620 con 0x7fc38806b730 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc378019070 con 0x7fc38806b730 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3780149e0 con 0x7fc38806b730 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3881f50b0 con 0x7fc38806b730 2024-04-22T01:18:55.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.115+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3881f5670 con 0x7fc38806b730 2024-04-22T01:18:55.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.119+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3880b1ac0 con 0x7fc38806b730 2024-04-22T01:18:55.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.119+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc378014b80 con 0x7fc38806b730 2024-04-22T01:18:55.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.119+0000 7fc3857fa640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc3740678c0 0x7fc374069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.119+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc378087410 con 0x7fc38806b730 2024-04-22T01:18:55.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.119+0000 7fc387fff640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc3740678c0 0x7fc374069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.119+0000 7fc387fff640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc3740678c0 0x7fc374069d80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0ad000 tx=0x7fc37c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.123+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc3880c8280 con 0x7fc38806b730 2024-04-22T01:18:55.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.275+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc388068920 con 0x7fc3740678c0 2024-04-22T01:18:55.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.295+0000 7fc3857fa640 1 -- 172.21.15.70:0/288828571 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252454 (secure 0 0 0) 0x7fc388068920 con 0x7fc3740678c0 2024-04-22T01:18:55.296 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:55.298 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:18:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc3740678c0 msgr2=0x7fc374069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc3740678c0 0x7fc374069d80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0ad000 tx=0x7fc37c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:55.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 msgr2=0x7fc3880c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 0x7fc3880c67f0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc378007e40 tx=0x7fc37800e4b0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 shutdown_connections 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc3740678c0 0x7fc374069d80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38815bc70 0x7fc3880c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3880cd740 0x7fc3881f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 --2- 172.21.15.70:0/288828571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc38806b730 0x7fc3880c67f0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 >> 172.21.15.70:0/288828571 conn(0x7fc388099a60 msgr2=0x7fc38815f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 shutdown_connections 2024-04-22T01:18:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.299+0000 7fc38ecea640 1 -- 172.21.15.70:0/288828571 wait complete. 2024-04-22T01:18:55.314 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-04-22T01:18:54.659127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277252,"kb_used_data":11124,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754697468,"statfs":{"total":773094113280,"available":772810207232,"internally_reserved":0,"allocated":11390976,"data_stored":5857664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.397436"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318647+0000","last_change":"2024-04-22T01:18:52.314149+0000","last_active":"2024-04-22T01:18:53.318647+0000","last_peered":"2024-04-22T01:18:53.318647+0000","last_clean":"2024-04-22T01:18:53.318647+0000","last_became_active":"2024-04-22T01:18:52.313904+0000","last_became_peered":"2024-04-22T01:18:52.313904+0000","last_unstale":"2024-04-22T01:18:53.318647+0000","last_undegraded":"2024-04-22T01:18:53.318647+0000","last_fullsized":"2024-04-22T01:18:53.318647+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:01.926744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361953+0000","last_change":"2024-04-22T01:18:52.356375+0000","last_active":"2024-04-22T01:18:52.361953+0000","last_peered":"2024-04-22T01:18:52.361953+0000","last_clean":"2024-04-22T01:18:52.361953+0000","last_became_active":"2024-04-22T01:18:52.348920+0000","last_became_peered":"2024-04-22T01:18:52.348920+0000","last_unstale":"2024-04-22T01:18:52.361953+0000","last_undegraded":"2024-04-22T01:18:52.361953+0000","last_fullsized":"2024-04-22T01:18:52.361953+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318481+0000","last_change":"2024-04-22T01:18:52.307572+0000","last_active":"2024-04-22T01:18:53.318481+0000","last_peered":"2024-04-22T01:18:53.318481+0000","last_clean":"2024-04-22T01:18:53.318481+0000","last_became_active":"2024-04-22T01:18:52.307257+0000","last_became_peered":"2024-04-22T01:18:52.307257+0000","last_unstale":"2024-04-22T01:18:53.318481+0000","last_undegraded":"2024-04-22T01:18:53.318481+0000","last_fullsized":"2024-04-22T01:18:53.318481+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:28:21.608094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.347742+0000","last_change":"2024-04-22T01:18:52.347162+0000","last_active":"2024-04-22T01:18:52.347742+0000","last_peered":"2024-04-22T01:18:52.347742+0000","last_clean":"2024-04-22T01:18:52.347742+0000","last_became_active":"2024-04-22T01:18:52.346337+0000","last_became_peered":"2024-04-22T01:18:52.346337+0000","last_unstale":"2024-04-22T01:18:52.347742+0000","last_undegraded":"2024-04-22T01:18:52.347742+0000","last_fullsized":"2024-04-22T01:18:52.347742+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:01.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.347284+0000","last_change":"2024-04-22T01:18:52.346935+0000","last_active":"2024-04-22T01:18:52.347284+0000","last_peered":"2024-04-22T01:18:52.347284+0000","last_clean":"2024-04-22T01:18:52.347284+0000","last_became_active":"2024-04-22T01:18:52.346680+0000","last_became_peered":"2024-04-22T01:18:52.346680+0000","last_unstale":"2024-04-22T01:18:52.347284+0000","last_undegraded":"2024-04-22T01:18:52.347284+0000","last_fullsized":"2024-04-22T01:18:52.347284+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361925+0000","last_change":"2024-04-22T01:18:52.356098+0000","last_active":"2024-04-22T01:18:52.361925+0000","last_peered":"2024-04-22T01:18:52.361925+0000","last_clean":"2024-04-22T01:18:52.361925+0000","last_became_active":"2024-04-22T01:18:52.340731+0000","last_became_peered":"2024-04-22T01:18:52.340731+0000","last_unstale":"2024-04-22T01:18:52.361925+0000","last_undegraded":"2024-04-22T01:18:52.361925+0000","last_fullsized":"2024-04-22T01:18:52.361925+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:52.479137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361992+0000","last_change":"2024-04-22T01:18:52.356791+0000","last_active":"2024-04-22T01:18:52.361992+0000","last_peered":"2024-04-22T01:18:52.361992+0000","last_clean":"2024-04-22T01:18:52.361992+0000","last_became_active":"2024-04-22T01:18:52.355533+0000","last_became_peered":"2024-04-22T01:18:52.355533+0000","last_unstale":"2024-04-22T01:18:52.361992+0000","last_undegraded":"2024-04-22T01:18:52.361992+0000","last_fullsized":"2024-04-22T01:18:52.361992+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:01.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.329204+0000","last_change":"2024-04-22T01:18:52.327508+0000","last_active":"2024-04-22T01:18:52.329204+0000","last_peered":"2024-04-22T01:18:52.329204+0000","last_clean":"2024-04-22T01:18:52.329204+0000","last_became_active":"2024-04-22T01:18:52.327030+0000","last_became_peered":"2024-04-22T01:18:52.327030+0000","last_unstale":"2024-04-22T01:18:52.329204+0000","last_undegraded":"2024-04-22T01:18:52.329204+0000","last_fullsized":"2024-04-22T01:18:52.329204+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:35.303367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.357415+0000","last_change":"2024-04-22T01:18:52.350875+0000","last_active":"2024-04-22T01:18:52.357415+0000","last_peered":"2024-04-22T01:18:52.357415+0000","last_clean":"2024-04-22T01:18:52.357415+0000","last_became_active":"2024-04-22T01:18:52.349998+0000","last_became_peered":"2024-04-22T01:18:52.349998+0000","last_unstale":"2024-04-22T01:18:52.357415+0000","last_undegraded":"2024-04-22T01:18:52.357415+0000","last_fullsized":"2024-04-22T01:18:52.357415+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:56:41.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.359900+0000","last_change":"2024-04-22T01:18:52.359516+0000","last_active":"2024-04-22T01:18:52.359900+0000","last_peered":"2024-04-22T01:18:52.359900+0000","last_clean":"2024-04-22T01:18:52.359900+0000","last_became_active":"2024-04-22T01:18:52.359240+0000","last_became_peered":"2024-04-22T01:18:52.359240+0000","last_unstale":"2024-04-22T01:18:52.359900+0000","last_undegraded":"2024-04-22T01:18:52.359900+0000","last_fullsized":"2024-04-22T01:18:52.359900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:12.739140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323538+0000","last_change":"2024-04-22T01:18:52.358815+0000","last_active":"2024-04-22T01:18:53.323538+0000","last_peered":"2024-04-22T01:18:53.323538+0000","last_clean":"2024-04-22T01:18:53.323538+0000","last_became_active":"2024-04-22T01:18:52.358253+0000","last_became_peered":"2024-04-22T01:18:52.358253+0000","last_unstale":"2024-04-22T01:18:53.323538+0000","last_undegraded":"2024-04-22T01:18:53.323538+0000","last_fullsized":"2024-04-22T01:18:53.323538+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:10.638308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321503+0000","last_change":"2024-04-22T01:18:52.317096+0000","last_active":"2024-04-22T01:18:53.321503+0000","last_peered":"2024-04-22T01:18:53.321503+0000","last_clean":"2024-04-22T01:18:53.321503+0000","last_became_active":"2024-04-22T01:18:52.316306+0000","last_became_peered":"2024-04-22T01:18:52.316306+0000","last_unstale":"2024-04-22T01:18:53.321503+0000","last_undegraded":"2024-04-22T01:18:53.321503+0000","last_fullsized":"2024-04-22T01:18:53.321503+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:01.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.354862+0000","last_change":"2024-04-22T01:18:52.354479+0000","last_active":"2024-04-22T01:18:52.354862+0000","last_peered":"2024-04-22T01:18:52.354862+0000","last_clean":"2024-04-22T01:18:52.354862+0000","last_became_active":"2024-04-22T01:18:52.354195+0000","last_became_peered":"2024-04-22T01:18:52.354195+0000","last_unstale":"2024-04-22T01:18:52.354862+0000","last_undegraded":"2024-04-22T01:18:52.354862+0000","last_fullsized":"2024-04-22T01:18:52.354862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591761+0000","last_change":"2024-04-22T01:18:52.332159+0000","last_active":"2024-04-22T01:18:53.591761+0000","last_peered":"2024-04-22T01:18:53.591761+0000","last_clean":"2024-04-22T01:18:53.591761+0000","last_became_active":"2024-04-22T01:18:52.331379+0000","last_became_peered":"2024-04-22T01:18:52.331379+0000","last_unstale":"2024-04-22T01:18:53.591761+0000","last_undegraded":"2024-04-22T01:18:53.591761+0000","last_fullsized":"2024-04-22T01:18:53.591761+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:09:57.302694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.592091+0000","last_change":"2024-04-22T01:18:52.354786+0000","last_active":"2024-04-22T01:18:53.592091+0000","last_peered":"2024-04-22T01:18:53.592091+0000","last_clean":"2024-04-22T01:18:53.592091+0000","last_became_active":"2024-04-22T01:18:52.353998+0000","last_became_peered":"2024-04-22T01:18:52.353998+0000","last_unstale":"2024-04-22T01:18:53.592091+0000","last_undegraded":"2024-04-22T01:18:53.592091+0000","last_fullsized":"2024-04-22T01:18:53.592091+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:00:55.019846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.592054+0000","last_change":"2024-04-22T01:18:52.334389+0000","last_active":"2024-04-22T01:18:53.592054+0000","last_peered":"2024-04-22T01:18:53.592054+0000","last_clean":"2024-04-22T01:18:53.592054+0000","last_became_active":"2024-04-22T01:18:52.333487+0000","last_became_peered":"2024-04-22T01:18:52.333487+0000","last_unstale":"2024-04-22T01:18:53.592054+0000","last_undegraded":"2024-04-22T01:18:53.592054+0000","last_fullsized":"2024-04-22T01:18:53.592054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:09:12.835038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318727+0000","last_change":"2024-04-22T01:18:52.329841+0000","last_active":"2024-04-22T01:18:53.318727+0000","last_peered":"2024-04-22T01:18:53.318727+0000","last_clean":"2024-04-22T01:18:53.318727+0000","last_became_active":"2024-04-22T01:18:52.329550+0000","last_became_peered":"2024-04-22T01:18:52.329550+0000","last_unstale":"2024-04-22T01:18:53.318727+0000","last_undegraded":"2024-04-22T01:18:53.318727+0000","last_fullsized":"2024-04-22T01:18:53.318727+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:57.240873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361311+0000","last_change":"2024-04-22T01:18:52.355588+0000","last_active":"2024-04-22T01:18:52.361311+0000","last_peered":"2024-04-22T01:18:52.361311+0000","last_clean":"2024-04-22T01:18:52.361311+0000","last_became_active":"2024-04-22T01:18:52.340740+0000","last_became_peered":"2024-04-22T01:18:52.340740+0000","last_unstale":"2024-04-22T01:18:52.361311+0000","last_undegraded":"2024-04-22T01:18:52.361311+0000","last_fullsized":"2024-04-22T01:18:52.361311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:04:42.756386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318630+0000","last_change":"2024-04-22T01:18:52.358493+0000","last_active":"2024-04-22T01:18:53.318630+0000","last_peered":"2024-04-22T01:18:53.318630+0000","last_clean":"2024-04-22T01:18:53.318630+0000","last_became_active":"2024-04-22T01:18:52.358164+0000","last_became_peered":"2024-04-22T01:18:52.358164+0000","last_unstale":"2024-04-22T01:18:53.318630+0000","last_undegraded":"2024-04-22T01:18:53.318630+0000","last_fullsized":"2024-04-22T01:18:53.318630+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:54:43.782657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321327+0000","last_change":"2024-04-22T01:18:52.335923+0000","last_active":"2024-04-22T01:18:53.321327+0000","last_peered":"2024-04-22T01:18:53.321327+0000","last_clean":"2024-04-22T01:18:53.321327+0000","last_became_active":"2024-04-22T01:18:52.334164+0000","last_became_peered":"2024-04-22T01:18:52.334164+0000","last_unstale":"2024-04-22T01:18:53.321327+0000","last_undegraded":"2024-04-22T01:18:53.321327+0000","last_fullsized":"2024-04-22T01:18:53.321327+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.323451+0000","last_change":"2024-04-22T01:18:52.322252+0000","last_active":"2024-04-22T01:18:52.323451+0000","last_peered":"2024-04-22T01:18:52.323451+0000","last_clean":"2024-04-22T01:18:52.323451+0000","last_became_active":"2024-04-22T01:18:52.317433+0000","last_became_peered":"2024-04-22T01:18:52.317433+0000","last_unstale":"2024-04-22T01:18:52.323451+0000","last_undegraded":"2024-04-22T01:18:52.323451+0000","last_fullsized":"2024-04-22T01:18:52.323451+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:04.752486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333992+0000","last_change":"2024-04-22T01:18:52.327197+0000","last_active":"2024-04-22T01:18:52.333992+0000","last_peered":"2024-04-22T01:18:52.333992+0000","last_clean":"2024-04-22T01:18:52.333992+0000","last_became_active":"2024-04-22T01:18:52.326654+0000","last_became_peered":"2024-04-22T01:18:52.326654+0000","last_unstale":"2024-04-22T01:18:52.333992+0000","last_undegraded":"2024-04-22T01:18:52.333992+0000","last_fullsized":"2024-04-22T01:18:52.333992+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318262+0000","last_change":"2024-04-22T01:18:52.340870+0000","last_active":"2024-04-22T01:18:53.318262+0000","last_peered":"2024-04-22T01:18:53.318262+0000","last_clean":"2024-04-22T01:18:53.318262+0000","last_became_active":"2024-04-22T01:18:52.340582+0000","last_became_peered":"2024-04-22T01:18:52.340582+0000","last_unstale":"2024-04-22T01:18:53.318262+0000","last_undegraded":"2024-04-22T01:18:53.318262+0000","last_fullsized":"2024-04-22T01:18:53.318262+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:24:05.347394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333912+0000","last_change":"2024-04-22T01:18:52.326186+0000","last_active":"2024-04-22T01:18:52.333912+0000","last_peered":"2024-04-22T01:18:52.333912+0000","last_clean":"2024-04-22T01:18:52.333912+0000","last_became_active":"2024-04-22T01:18:52.325885+0000","last_became_peered":"2024-04-22T01:18:52.325885+0000","last_unstale":"2024-04-22T01:18:52.333912+0000","last_undegraded":"2024-04-22T01:18:52.333912+0000","last_fullsized":"2024-04-22T01:18:52.333912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:56:41.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321720+0000","last_change":"2024-04-22T01:18:52.347405+0000","last_active":"2024-04-22T01:18:53.321720+0000","last_peered":"2024-04-22T01:18:53.321720+0000","last_clean":"2024-04-22T01:18:53.321720+0000","last_became_active":"2024-04-22T01:18:52.346612+0000","last_became_peered":"2024-04-22T01:18:52.346612+0000","last_unstale":"2024-04-22T01:18:53.321720+0000","last_undegraded":"2024-04-22T01:18:53.321720+0000","last_fullsized":"2024-04-22T01:18:53.321720+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321618+0000","last_change":"2024-04-22T01:18:52.351374+0000","last_active":"2024-04-22T01:18:53.321618+0000","last_peered":"2024-04-22T01:18:53.321618+0000","last_clean":"2024-04-22T01:18:53.321618+0000","last_became_active":"2024-04-22T01:18:52.350733+0000","last_became_peered":"2024-04-22T01:18:52.350733+0000","last_unstale":"2024-04-22T01:18:53.321618+0000","last_undegraded":"2024-04-22T01:18:53.321618+0000","last_fullsized":"2024-04-22T01:18:53.321618+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323751+0000","last_change":"2024-04-22T01:18:52.342467+0000","last_active":"2024-04-22T01:18:53.323751+0000","last_peered":"2024-04-22T01:18:53.323751+0000","last_clean":"2024-04-22T01:18:53.323751+0000","last_became_active":"2024-04-22T01:18:52.341578+0000","last_became_peered":"2024-04-22T01:18:52.341578+0000","last_unstale":"2024-04-22T01:18:53.323751+0000","last_undegraded":"2024-04-22T01:18:53.323751+0000","last_fullsized":"2024-04-22T01:18:53.323751+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:22.235466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.356638+0000","last_change":"2024-04-22T01:18:52.349240+0000","last_active":"2024-04-22T01:18:52.356638+0000","last_peered":"2024-04-22T01:18:52.356638+0000","last_clean":"2024-04-22T01:18:52.356638+0000","last_became_active":"2024-04-22T01:18:52.340896+0000","last_became_peered":"2024-04-22T01:18:52.340896+0000","last_unstale":"2024-04-22T01:18:52.356638+0000","last_undegraded":"2024-04-22T01:18:52.356638+0000","last_fullsized":"2024-04-22T01:18:52.356638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:39.029774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333747+0000","last_change":"2024-04-22T01:18:52.333354+0000","last_active":"2024-04-22T01:18:52.333747+0000","last_peered":"2024-04-22T01:18:52.333747+0000","last_clean":"2024-04-22T01:18:52.333747+0000","last_became_active":"2024-04-22T01:18:52.333069+0000","last_became_peered":"2024-04-22T01:18:52.333069+0000","last_unstale":"2024-04-22T01:18:52.333747+0000","last_undegraded":"2024-04-22T01:18:52.333747+0000","last_fullsized":"2024-04-22T01:18:52.333747+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:16.653262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.320705+0000","last_change":"2024-04-22T01:18:52.346605+0000","last_active":"2024-04-22T01:18:53.320705+0000","last_peered":"2024-04-22T01:18:53.320705+0000","last_clean":"2024-04-22T01:18:53.320705+0000","last_became_active":"2024-04-22T01:18:52.346007+0000","last_became_peered":"2024-04-22T01:18:52.346007+0000","last_unstale":"2024-04-22T01:18:53.320705+0000","last_undegraded":"2024-04-22T01:18:53.320705+0000","last_fullsized":"2024-04-22T01:18:53.320705+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:52.479137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.360688+0000","last_change":"2024-04-22T01:18:52.355633+0000","last_active":"2024-04-22T01:18:52.360688+0000","last_peered":"2024-04-22T01:18:52.360688+0000","last_clean":"2024-04-22T01:18:52.360688+0000","last_became_active":"2024-04-22T01:18:52.350967+0000","last_became_peered":"2024-04-22T01:18:52.350967+0000","last_unstale":"2024-04-22T01:18:52.360688+0000","last_undegraded":"2024-04-22T01:18:52.360688+0000","last_fullsized":"2024-04-22T01:18:52.360688+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:02.573459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.340972+0000","last_change":"2024-04-22T01:18:52.340194+0000","last_active":"2024-04-22T01:18:52.340972+0000","last_peered":"2024-04-22T01:18:52.340972+0000","last_clean":"2024-04-22T01:18:52.340972+0000","last_became_active":"2024-04-22T01:18:52.339904+0000","last_became_peered":"2024-04-22T01:18:52.339904+0000","last_unstale":"2024-04-22T01:18:52.340972+0000","last_undegraded":"2024-04-22T01:18:52.340972+0000","last_fullsized":"2024-04-22T01:18:52.340972+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:51:15.255767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318396+0000","last_change":"2024-04-22T01:18:52.341326+0000","last_active":"2024-04-22T01:18:53.318396+0000","last_peered":"2024-04-22T01:18:53.318396+0000","last_clean":"2024-04-22T01:18:53.318396+0000","last_became_active":"2024-04-22T01:18:52.341102+0000","last_became_peered":"2024-04-22T01:18:52.341102+0000","last_unstale":"2024-04-22T01:18:53.318396+0000","last_undegraded":"2024-04-22T01:18:53.318396+0000","last_fullsized":"2024-04-22T01:18:53.318396+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:12.237330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"102'4","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.358352+0000","last_change":"2024-04-22T01:18:51.315283+0000","last_active":"2024-04-22T01:18:52.358352+0000","last_peered":"2024-04-22T01:18:52.358352+0000","last_clean":"2024-04-22T01:18:52.358352+0000","last_became_active":"2024-04-22T01:18:51.315068+0000","last_became_peered":"2024-04-22T01:18:51.315068+0000","last_unstale":"2024-04-22T01:18:52.358352+0000","last_undegraded":"2024-04-22T01:18:52.358352+0000","last_fullsized":"2024-04-22T01:18:52.358352+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:04:41.753177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.318274+0000","last_change":"2024-04-22T01:18:52.317724+0000","last_active":"2024-04-22T01:18:52.318274+0000","last_peered":"2024-04-22T01:18:52.318274+0000","last_clean":"2024-04-22T01:18:52.318274+0000","last_became_active":"2024-04-22T01:18:52.317404+0000","last_became_peered":"2024-04-22T01:18:52.317404+0000","last_unstale":"2024-04-22T01:18:52.318274+0000","last_undegraded":"2024-04-22T01:18:52.318274+0000","last_fullsized":"2024-04-22T01:18:52.318274+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:30:47.755545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.332994+0000","last_change":"2024-04-22T01:18:51.308359+0000","last_active":"2024-04-22T01:18:52.332994+0000","last_peered":"2024-04-22T01:18:52.332994+0000","last_clean":"2024-04-22T01:18:52.332994+0000","last_became_active":"2024-04-22T01:18:51.307681+0000","last_became_peered":"2024-04-22T01:18:51.307681+0000","last_unstale":"2024-04-22T01:18:52.332994+0000","last_undegraded":"2024-04-22T01:18:52.332994+0000","last_fullsized":"2024-04-22T01:18:52.332994+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:29.570206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321104+0000","last_change":"2024-04-22T01:18:52.327103+0000","last_active":"2024-04-22T01:18:53.321104+0000","last_peered":"2024-04-22T01:18:53.321104+0000","last_clean":"2024-04-22T01:18:53.321104+0000","last_became_active":"2024-04-22T01:18:52.326404+0000","last_became_peered":"2024-04-22T01:18:52.326404+0000","last_unstale":"2024-04-22T01:18:53.321104+0000","last_undegraded":"2024-04-22T01:18:53.321104+0000","last_fullsized":"2024-04-22T01:18:53.321104+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321206+0000","last_change":"2024-04-22T01:18:51.311531+0000","last_active":"2024-04-22T01:18:53.321206+0000","last_peered":"2024-04-22T01:18:53.321206+0000","last_clean":"2024-04-22T01:18:53.321206+0000","last_became_active":"2024-04-22T01:18:51.310780+0000","last_became_peered":"2024-04-22T01:18:51.310780+0000","last_unstale":"2024-04-22T01:18:53.321206+0000","last_undegraded":"2024-04-22T01:18:53.321206+0000","last_fullsized":"2024-04-22T01:18:53.321206+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:07.837857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323053+0000","last_change":"2024-04-22T01:18:52.347947+0000","last_active":"2024-04-22T01:18:53.323053+0000","last_peered":"2024-04-22T01:18:53.323053+0000","last_clean":"2024-04-22T01:18:53.323053+0000","last_became_active":"2024-04-22T01:18:52.347277+0000","last_became_peered":"2024-04-22T01:18:52.347277+0000","last_unstale":"2024-04-22T01:18:53.323053+0000","last_undegraded":"2024-04-22T01:18:53.323053+0000","last_fullsized":"2024-04-22T01:18:53.323053+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:38.433456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.346807+0000","last_change":"2024-04-22T01:18:51.302794+0000","last_active":"2024-04-22T01:18:52.346807+0000","last_peered":"2024-04-22T01:18:52.346807+0000","last_clean":"2024-04-22T01:18:52.346807+0000","last_became_active":"2024-04-22T01:18:51.302436+0000","last_became_peered":"2024-04-22T01:18:51.302436+0000","last_unstale":"2024-04-22T01:18:52.346807+0000","last_undegraded":"2024-04-22T01:18:52.346807+0000","last_fullsized":"2024-04-22T01:18:52.346807+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:04:41.753177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.336488+0000","last_change":"2024-04-22T01:18:52.335978+0000","last_active":"2024-04-22T01:18:52.336488+0000","last_peered":"2024-04-22T01:18:52.336488+0000","last_clean":"2024-04-22T01:18:52.336488+0000","last_became_active":"2024-04-22T01:18:52.335651+0000","last_became_peered":"2024-04-22T01:18:52.335651+0000","last_unstale":"2024-04-22T01:18:52.336488+0000","last_undegraded":"2024-04-22T01:18:52.336488+0000","last_fullsized":"2024-04-22T01:18:52.336488+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591760+0000","last_change":"2024-04-22T01:18:51.298870+0000","last_active":"2024-04-22T01:18:53.591760+0000","last_peered":"2024-04-22T01:18:53.591760+0000","last_clean":"2024-04-22T01:18:53.591760+0000","last_became_active":"2024-04-22T01:18:51.298047+0000","last_became_peered":"2024-04-22T01:18:51.298047+0000","last_unstale":"2024-04-22T01:18:53.591760+0000","last_undegraded":"2024-04-22T01:18:53.591760+0000","last_fullsized":"2024-04-22T01:18:53.591760+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:21.232256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318757+0000","last_change":"2024-04-22T01:18:52.358235+0000","last_active":"2024-04-22T01:18:53.318757+0000","last_peered":"2024-04-22T01:18:53.318757+0000","last_clean":"2024-04-22T01:18:53.318757+0000","last_became_active":"2024-04-22T01:18:52.358007+0000","last_became_peered":"2024-04-22T01:18:52.358007+0000","last_unstale":"2024-04-22T01:18:53.318757+0000","last_undegraded":"2024-04-22T01:18:53.318757+0000","last_fullsized":"2024-04-22T01:18:53.318757+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:30:16.786285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.322609+0000","last_change":"2024-04-22T01:18:51.303578+0000","last_active":"2024-04-22T01:18:53.322609+0000","last_peered":"2024-04-22T01:18:53.322609+0000","last_clean":"2024-04-22T01:18:53.322609+0000","last_became_active":"2024-04-22T01:18:51.302994+0000","last_became_peered":"2024-04-22T01:18:51.302994+0000","last_unstale":"2024-04-22T01:18:53.322609+0000","last_undegraded":"2024-04-22T01:18:53.322609+0000","last_fullsized":"2024-04-22T01:18:53.322609+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:38.026564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.362081+0000","last_change":"2024-04-22T01:18:52.357882+0000","last_active":"2024-04-22T01:18:52.362081+0000","last_peered":"2024-04-22T01:18:52.362081+0000","last_clean":"2024-04-22T01:18:52.362081+0000","last_became_active":"2024-04-22T01:18:52.348758+0000","last_became_peered":"2024-04-22T01:18:52.348758+0000","last_unstale":"2024-04-22T01:18:52.362081+0000","last_undegraded":"2024-04-22T01:18:52.362081+0000","last_fullsized":"2024-04-22T01:18:52.362081+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.320420+0000","last_change":"2024-04-22T01:18:51.309550+0000","last_active":"2024-04-22T01:18:53.320420+0000","last_peered":"2024-04-22T01:18:53.320420+0000","last_clean":"2024-04-22T01:18:53.320420+0000","last_became_active":"2024-04-22T01:18:51.303367+0000","last_became_peered":"2024-04-22T01:18:51.303367+0000","last_unstale":"2024-04-22T01:18:53.320420+0000","last_undegraded":"2024-04-22T01:18:53.320420+0000","last_fullsized":"2024-04-22T01:18:53.320420+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:40:39.157115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.320782+0000","last_change":"2024-04-22T01:18:52.313086+0000","last_active":"2024-04-22T01:18:52.320782+0000","last_peered":"2024-04-22T01:18:52.320782+0000","last_clean":"2024-04-22T01:18:52.320782+0000","last_became_active":"2024-04-22T01:18:52.312403+0000","last_became_peered":"2024-04-22T01:18:52.312403+0000","last_unstale":"2024-04-22T01:18:52.320782+0000","last_undegraded":"2024-04-22T01:18:52.320782+0000","last_fullsized":"2024-04-22T01:18:52.320782+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.304978+0000","last_change":"2024-04-22T01:18:51.282774+0000","last_active":"2024-04-22T01:18:52.304978+0000","last_peered":"2024-04-22T01:18:52.304978+0000","last_clean":"2024-04-22T01:18:52.304978+0000","last_became_active":"2024-04-22T01:18:51.282219+0000","last_became_peered":"2024-04-22T01:18:51.282219+0000","last_unstale":"2024-04-22T01:18:52.304978+0000","last_undegraded":"2024-04-22T01:18:52.304978+0000","last_fullsized":"2024-04-22T01:18:52.304978+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:01.570250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318296+0000","last_change":"2024-04-22T01:18:52.341648+0000","last_active":"2024-04-22T01:18:53.318296+0000","last_peered":"2024-04-22T01:18:53.318296+0000","last_clean":"2024-04-22T01:18:53.318296+0000","last_became_active":"2024-04-22T01:18:52.341296+0000","last_became_peered":"2024-04-22T01:18:52.341296+0000","last_unstale":"2024-04-22T01:18:53.318296+0000","last_undegraded":"2024-04-22T01:18:53.318296+0000","last_fullsized":"2024-04-22T01:18:53.318296+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:59:50.633160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321953+0000","last_change":"2024-04-22T01:18:51.311920+0000","last_active":"2024-04-22T01:18:53.321953+0000","last_peered":"2024-04-22T01:18:53.321953+0000","last_clean":"2024-04-22T01:18:53.321953+0000","last_became_active":"2024-04-22T01:18:51.310648+0000","last_became_peered":"2024-04-22T01:18:51.310648+0000","last_unstale":"2024-04-22T01:18:53.321953+0000","last_undegraded":"2024-04-22T01:18:53.321953+0000","last_fullsized":"2024-04-22T01:18:53.321953+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:03:01.467024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.334549+0000","last_change":"2024-04-22T01:18:52.323570+0000","last_active":"2024-04-22T01:18:52.334549+0000","last_peered":"2024-04-22T01:18:52.334549+0000","last_clean":"2024-04-22T01:18:52.334549+0000","last_became_active":"2024-04-22T01:18:52.322934+0000","last_became_peered":"2024-04-22T01:18:52.322934+0000","last_unstale":"2024-04-22T01:18:52.334549+0000","last_undegraded":"2024-04-22T01:18:52.334549+0000","last_fullsized":"2024-04-22T01:18:52.334549+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:39.029774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318595+0000","last_change":"2024-04-22T01:18:51.303127+0000","last_active":"2024-04-22T01:18:53.318595+0000","last_peered":"2024-04-22T01:18:53.318595+0000","last_clean":"2024-04-22T01:18:53.318595+0000","last_became_active":"2024-04-22T01:18:51.302773+0000","last_became_peered":"2024-04-22T01:18:51.302773+0000","last_unstale":"2024-04-22T01:18:53.318595+0000","last_undegraded":"2024-04-22T01:18:53.318595+0000","last_fullsized":"2024-04-22T01:18:53.318595+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:01:16.888369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.329450+0000","last_change":"2024-04-22T01:18:52.327804+0000","last_active":"2024-04-22T01:18:52.329450+0000","last_peered":"2024-04-22T01:18:52.329450+0000","last_clean":"2024-04-22T01:18:52.329450+0000","last_became_active":"2024-04-22T01:18:52.327100+0000","last_became_peered":"2024-04-22T01:18:52.327100+0000","last_unstale":"2024-04-22T01:18:52.329450+0000","last_undegraded":"2024-04-22T01:18:52.329450+0000","last_fullsized":"2024-04-22T01:18:52.329450+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:09:24.954206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"102'3","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591842+0000","last_change":"2024-04-22T01:18:51.289102+0000","last_active":"2024-04-22T01:18:53.591842+0000","last_peered":"2024-04-22T01:18:53.591842+0000","last_clean":"2024-04-22T01:18:53.591842+0000","last_became_active":"2024-04-22T01:18:51.288334+0000","last_became_peered":"2024-04-22T01:18:51.288334+0000","last_unstale":"2024-04-22T01:18:53.591842+0000","last_undegraded":"2024-04-22T01:18:53.591842+0000","last_fullsized":"2024-04-22T01:18:53.591842+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:37.430246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333950+0000","last_change":"2024-04-22T01:18:52.326573+0000","last_active":"2024-04-22T01:18:52.333950+0000","last_peered":"2024-04-22T01:18:52.333950+0000","last_clean":"2024-04-22T01:18:52.333950+0000","last_became_active":"2024-04-22T01:18:52.325971+0000","last_became_peered":"2024-04-22T01:18:52.325971+0000","last_unstale":"2024-04-22T01:18:52.333950+0000","last_undegraded":"2024-04-22T01:18:52.333950+0000","last_fullsized":"2024-04-22T01:18:52.333950+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"102'6","reported_seq":18,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.362747+0000","last_change":"2024-04-22T01:18:51.311610+0000","last_active":"2024-04-22T01:18:52.362747+0000","last_peered":"2024-04-22T01:18:52.362747+0000","last_clean":"2024-04-22T01:18:52.362747+0000","last_became_active":"2024-04-22T01:18:51.305910+0000","last_became_peered":"2024-04-22T01:18:51.305910+0000","last_unstale":"2024-04-22T01:18:52.362747+0000","last_undegraded":"2024-04-22T01:18:52.362747+0000","last_fullsized":"2024-04-22T01:18:52.362747+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:57.349352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.322994+0000","last_change":"2024-04-22T01:18:52.330283+0000","last_active":"2024-04-22T01:18:53.322994+0000","last_peered":"2024-04-22T01:18:53.322994+0000","last_clean":"2024-04-22T01:18:53.322994+0000","last_became_active":"2024-04-22T01:18:52.328732+0000","last_became_peered":"2024-04-22T01:18:52.328732+0000","last_unstale":"2024-04-22T01:18:53.322994+0000","last_undegraded":"2024-04-22T01:18:53.322994+0000","last_fullsized":"2024-04-22T01:18:53.322994+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:12.739140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337539+0000","last_change":"2024-04-22T01:18:51.299682+0000","last_active":"2024-04-22T01:18:52.337539+0000","last_peered":"2024-04-22T01:18:52.337539+0000","last_clean":"2024-04-22T01:18:52.337539+0000","last_became_active":"2024-04-22T01:18:51.299324+0000","last_became_peered":"2024-04-22T01:18:51.299324+0000","last_unstale":"2024-04-22T01:18:52.337539+0000","last_undegraded":"2024-04-22T01:18:52.337539+0000","last_fullsized":"2024-04-22T01:18:52.337539+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:51.475927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.362033+0000","last_change":"2024-04-22T01:18:52.357031+0000","last_active":"2024-04-22T01:18:52.362033+0000","last_peered":"2024-04-22T01:18:52.362033+0000","last_clean":"2024-04-22T01:18:52.362033+0000","last_became_active":"2024-04-22T01:18:52.349376+0000","last_became_peered":"2024-04-22T01:18:52.349376+0000","last_unstale":"2024-04-22T01:18:52.362033+0000","last_undegraded":"2024-04-22T01:18:52.362033+0000","last_fullsized":"2024-04-22T01:18:52.362033+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.306549+0000","last_change":"2024-04-22T01:18:51.282341+0000","last_active":"2024-04-22T01:18:52.306549+0000","last_peered":"2024-04-22T01:18:52.306549+0000","last_clean":"2024-04-22T01:18:52.306549+0000","last_became_active":"2024-04-22T01:18:51.281442+0000","last_became_peered":"2024-04-22T01:18:51.281442+0000","last_unstale":"2024-04-22T01:18:52.306549+0000","last_undegraded":"2024-04-22T01:18:52.306549+0000","last_fullsized":"2024-04-22T01:18:52.306549+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:51.475927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.341044+0000","last_change":"2024-04-22T01:18:52.340116+0000","last_active":"2024-04-22T01:18:52.341044+0000","last_peered":"2024-04-22T01:18:52.341044+0000","last_clean":"2024-04-22T01:18:52.341044+0000","last_became_active":"2024-04-22T01:18:52.339855+0000","last_became_peered":"2024-04-22T01:18:52.339855+0000","last_unstale":"2024-04-22T01:18:52.341044+0000","last_undegraded":"2024-04-22T01:18:52.341044+0000","last_fullsized":"2024-04-22T01:18:52.341044+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:10.326477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.302507+0000","last_change":"2024-04-22T01:18:51.313239+0000","last_active":"2024-04-22T01:18:52.302507+0000","last_peered":"2024-04-22T01:18:52.302507+0000","last_clean":"2024-04-22T01:18:52.302507+0000","last_became_active":"2024-04-22T01:18:51.312748+0000","last_became_peered":"2024-04-22T01:18:51.312748+0000","last_unstale":"2024-04-22T01:18:52.302507+0000","last_undegraded":"2024-04-22T01:18:52.302507+0000","last_fullsized":"2024-04-22T01:18:52.302507+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:22:16.460663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318847+0000","last_change":"2024-04-22T01:18:52.330108+0000","last_active":"2024-04-22T01:18:53.318847+0000","last_peered":"2024-04-22T01:18:53.318847+0000","last_clean":"2024-04-22T01:18:53.318847+0000","last_became_active":"2024-04-22T01:18:52.329686+0000","last_became_peered":"2024-04-22T01:18:52.329686+0000","last_unstale":"2024-04-22T01:18:53.318847+0000","last_undegraded":"2024-04-22T01:18:53.318847+0000","last_fullsized":"2024-04-22T01:18:53.318847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:22.225277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"102'8","reported_seq":22,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321567+0000","last_change":"2024-04-22T01:18:51.295650+0000","last_active":"2024-04-22T01:18:53.321567+0000","last_peered":"2024-04-22T01:18:53.321567+0000","last_clean":"2024-04-22T01:18:53.321567+0000","last_became_active":"2024-04-22T01:18:51.294914+0000","last_became_peered":"2024-04-22T01:18:51.294914+0000","last_unstale":"2024-04-22T01:18:53.321567+0000","last_undegraded":"2024-04-22T01:18:53.321567+0000","last_fullsized":"2024-04-22T01:18:53.321567+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:37.686456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.311149+0000","last_change":"2024-04-22T01:18:52.310177+0000","last_active":"2024-04-22T01:18:52.311149+0000","last_peered":"2024-04-22T01:18:52.311149+0000","last_clean":"2024-04-22T01:18:52.311149+0000","last_became_active":"2024-04-22T01:18:52.309882+0000","last_became_peered":"2024-04-22T01:18:52.309882+0000","last_unstale":"2024-04-22T01:18:52.311149+0000","last_undegraded":"2024-04-22T01:18:52.311149+0000","last_fullsized":"2024-04-22T01:18:52.311149+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:26.880594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.349934+0000","last_change":"2024-04-22T01:18:51.312797+0000","last_active":"2024-04-22T01:18:52.349934+0000","last_peered":"2024-04-22T01:18:52.349934+0000","last_clean":"2024-04-22T01:18:52.349934+0000","last_became_active":"2024-04-22T01:18:51.311969+0000","last_became_peered":"2024-04-22T01:18:51.311969+0000","last_unstale":"2024-04-22T01:18:52.349934+0000","last_undegraded":"2024-04-22T01:18:52.349934+0000","last_fullsized":"2024-04-22T01:18:52.349934+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:04.332870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361962+0000","last_change":"2024-04-22T01:18:52.356757+0000","last_active":"2024-04-22T01:18:52.361962+0000","last_peered":"2024-04-22T01:18:52.361962+0000","last_clean":"2024-04-22T01:18:52.361962+0000","last_became_active":"2024-04-22T01:18:52.348856+0000","last_became_peered":"2024-04-22T01:18:52.348856+0000","last_unstale":"2024-04-22T01:18:52.361962+0000","last_undegraded":"2024-04-22T01:18:52.361962+0000","last_fullsized":"2024-04-22T01:18:52.361962+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591804+0000","last_change":"2024-04-22T01:18:51.301777+0000","last_active":"2024-04-22T01:18:53.591804+0000","last_peered":"2024-04-22T01:18:53.591804+0000","last_clean":"2024-04-22T01:18:53.591804+0000","last_became_active":"2024-04-22T01:18:51.301148+0000","last_became_peered":"2024-04-22T01:18:51.301148+0000","last_unstale":"2024-04-22T01:18:53.591804+0000","last_undegraded":"2024-04-22T01:18:53.591804+0000","last_fullsized":"2024-04-22T01:18:53.591804+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:09.635098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337600+0000","last_change":"2024-04-22T01:18:52.335551+0000","last_active":"2024-04-22T01:18:52.337600+0000","last_peered":"2024-04-22T01:18:52.337600+0000","last_clean":"2024-04-22T01:18:52.337600+0000","last_became_active":"2024-04-22T01:18:52.335219+0000","last_became_peered":"2024-04-22T01:18:52.335219+0000","last_unstale":"2024-04-22T01:18:52.337600+0000","last_undegraded":"2024-04-22T01:18:52.337600+0000","last_fullsized":"2024-04-22T01:18:52.337600+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323015+0000","last_change":"2024-04-22T01:18:51.298240+0000","last_active":"2024-04-22T01:18:53.323015+0000","last_peered":"2024-04-22T01:18:53.323015+0000","last_clean":"2024-04-22T01:18:53.323015+0000","last_became_active":"2024-04-22T01:18:51.295827+0000","last_became_peered":"2024-04-22T01:18:51.295827+0000","last_unstale":"2024-04-22T01:18:53.323015+0000","last_undegraded":"2024-04-22T01:18:53.323015+0000","last_fullsized":"2024-04-22T01:18:53.323015+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:57.349352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318466+0000","last_change":"2024-04-22T01:18:52.311313+0000","last_active":"2024-04-22T01:18:53.318466+0000","last_peered":"2024-04-22T01:18:53.318466+0000","last_clean":"2024-04-22T01:18:53.318466+0000","last_became_active":"2024-04-22T01:18:52.311006+0000","last_became_peered":"2024-04-22T01:18:52.311006+0000","last_unstale":"2024-04-22T01:18:53.318466+0000","last_undegraded":"2024-04-22T01:18:53.318466+0000","last_fullsized":"2024-04-22T01:18:53.318466+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:24.725150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.314393+0000","last_change":"2024-04-22T01:18:51.298098+0000","last_active":"2024-04-22T01:18:52.314393+0000","last_peered":"2024-04-22T01:18:52.314393+0000","last_clean":"2024-04-22T01:18:52.314393+0000","last_became_active":"2024-04-22T01:18:51.297653+0000","last_became_peered":"2024-04-22T01:18:51.297653+0000","last_unstale":"2024-04-22T01:18:52.314393+0000","last_undegraded":"2024-04-22T01:18:52.314393+0000","last_fullsized":"2024-04-22T01:18:52.314393+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:48.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318714+0000","last_change":"2024-04-22T01:18:52.331792+0000","last_active":"2024-04-22T01:18:53.318714+0000","last_peered":"2024-04-22T01:18:53.318714+0000","last_clean":"2024-04-22T01:18:53.318714+0000","last_became_active":"2024-04-22T01:18:52.331169+0000","last_became_peered":"2024-04-22T01:18:52.331169+0000","last_unstale":"2024-04-22T01:18:53.318714+0000","last_undegraded":"2024-04-22T01:18:53.318714+0000","last_fullsized":"2024-04-22T01:18:53.318714+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:16.183115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.301745+0000","last_change":"2024-04-22T01:18:51.302622+0000","last_active":"2024-04-22T01:18:52.301745+0000","last_peered":"2024-04-22T01:18:52.301745+0000","last_clean":"2024-04-22T01:18:52.301745+0000","last_became_active":"2024-04-22T01:18:51.302208+0000","last_became_peered":"2024-04-22T01:18:51.302208+0000","last_unstale":"2024-04-22T01:18:52.301745+0000","last_undegraded":"2024-04-22T01:18:52.301745+0000","last_fullsized":"2024-04-22T01:18:52.301745+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:10.091431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.360276+0000","last_change":"2024-04-22T01:18:52.359536+0000","last_active":"2024-04-22T01:18:52.360276+0000","last_peered":"2024-04-22T01:18:52.360276+0000","last_clean":"2024-04-22T01:18:52.360276+0000","last_became_active":"2024-04-22T01:18:52.359209+0000","last_became_peered":"2024-04-22T01:18:52.359209+0000","last_unstale":"2024-04-22T01:18:52.360276+0000","last_undegraded":"2024-04-22T01:18:52.360276+0000","last_fullsized":"2024-04-22T01:18:52.360276+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:12.739140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.312035+0000","last_change":"2024-04-22T01:18:51.298037+0000","last_active":"2024-04-22T01:18:52.312035+0000","last_peered":"2024-04-22T01:18:52.312035+0000","last_clean":"2024-04-22T01:18:52.312035+0000","last_became_active":"2024-04-22T01:18:51.297351+0000","last_became_peered":"2024-04-22T01:18:51.297351+0000","last_unstale":"2024-04-22T01:18:52.312035+0000","last_undegraded":"2024-04-22T01:18:52.312035+0000","last_fullsized":"2024-04-22T01:18:52.312035+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:03:01.467024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337131+0000","last_change":"2024-04-22T01:18:52.335372+0000","last_active":"2024-04-22T01:18:52.337131+0000","last_peered":"2024-04-22T01:18:52.337131+0000","last_clean":"2024-04-22T01:18:52.337131+0000","last_became_active":"2024-04-22T01:18:52.334585+0000","last_became_peered":"2024-04-22T01:18:52.334585+0000","last_unstale":"2024-04-22T01:18:52.337131+0000","last_undegraded":"2024-04-22T01:18:52.337131+0000","last_fullsized":"2024-04-22T01:18:52.337131+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:22:14.804238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.301911+0000","last_change":"2024-04-22T01:18:51.303004+0000","last_active":"2024-04-22T01:18:52.301911+0000","last_peered":"2024-04-22T01:18:52.301911+0000","last_clean":"2024-04-22T01:18:52.301911+0000","last_became_active":"2024-04-22T01:18:51.302647+0000","last_became_peered":"2024-04-22T01:18:51.302647+0000","last_unstale":"2024-04-22T01:18:52.301911+0000","last_undegraded":"2024-04-22T01:18:52.301911+0000","last_fullsized":"2024-04-22T01:18:52.301911+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:38.124357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318234+0000","last_change":"2024-04-22T01:18:52.311250+0000","last_active":"2024-04-22T01:18:53.318234+0000","last_peered":"2024-04-22T01:18:53.318234+0000","last_clean":"2024-04-22T01:18:53.318234+0000","last_became_active":"2024-04-22T01:18:52.310892+0000","last_became_peered":"2024-04-22T01:18:52.310892+0000","last_unstale":"2024-04-22T01:18:53.318234+0000","last_undegraded":"2024-04-22T01:18:53.318234+0000","last_fullsized":"2024-04-22T01:18:53.318234+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:00:25.064406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337837+0000","last_change":"2024-04-22T01:18:51.299160+0000","last_active":"2024-04-22T01:18:52.337837+0000","last_peered":"2024-04-22T01:18:52.337837+0000","last_clean":"2024-04-22T01:18:52.337837+0000","last_became_active":"2024-04-22T01:18:51.298775+0000","last_became_peered":"2024-04-22T01:18:51.298775+0000","last_unstale":"2024-04-22T01:18:52.337837+0000","last_undegraded":"2024-04-22T01:18:52.337837+0000","last_fullsized":"2024-04-22T01:18:52.337837+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:00.219961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.320617+0000","last_change":"2024-04-22T01:18:52.314406+0000","last_active":"2024-04-22T01:18:53.320617+0000","last_peered":"2024-04-22T01:18:53.320617+0000","last_clean":"2024-04-22T01:18:53.320617+0000","last_became_active":"2024-04-22T01:18:52.313689+0000","last_became_peered":"2024-04-22T01:18:52.313689+0000","last_unstale":"2024-04-22T01:18:53.320617+0000","last_undegraded":"2024-04-22T01:18:53.320617+0000","last_fullsized":"2024-04-22T01:18:53.320617+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:02.573459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318692+0000","last_change":"2024-04-22T01:18:51.284358+0000","last_active":"2024-04-22T01:18:53.318692+0000","last_peered":"2024-04-22T01:18:53.318692+0000","last_clean":"2024-04-22T01:18:53.318692+0000","last_became_active":"2024-04-22T01:18:51.281948+0000","last_became_peered":"2024-04-22T01:18:51.281948+0000","last_unstale":"2024-04-22T01:18:53.318692+0000","last_undegraded":"2024-04-22T01:18:53.318692+0000","last_fullsized":"2024-04-22T01:18:53.318692+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:46:22.310137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.358243+0000","last_change":"2024-04-22T01:18:52.357795+0000","last_active":"2024-04-22T01:18:52.358243+0000","last_peered":"2024-04-22T01:18:52.358243+0000","last_clean":"2024-04-22T01:18:52.358243+0000","last_became_active":"2024-04-22T01:18:52.357376+0000","last_became_peered":"2024-04-22T01:18:52.357376+0000","last_unstale":"2024-04-22T01:18:52.358243+0000","last_undegraded":"2024-04-22T01:18:52.358243+0000","last_fullsized":"2024-04-22T01:18:52.358243+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.325886+0000","last_change":"2024-04-22T01:18:51.306618+0000","last_active":"2024-04-22T01:18:52.325886+0000","last_peered":"2024-04-22T01:18:52.325886+0000","last_clean":"2024-04-22T01:18:52.325886+0000","last_became_active":"2024-04-22T01:18:51.305474+0000","last_became_peered":"2024-04-22T01:18:51.305474+0000","last_unstale":"2024-04-22T01:18:52.325886+0000","last_undegraded":"2024-04-22T01:18:52.325886+0000","last_fullsized":"2024-04-22T01:18:52.325886+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:09:14.958959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323009+0000","last_change":"2024-04-22T01:18:52.332479+0000","last_active":"2024-04-22T01:18:53.323009+0000","last_peered":"2024-04-22T01:18:53.323009+0000","last_clean":"2024-04-22T01:18:53.323009+0000","last_became_active":"2024-04-22T01:18:52.329098+0000","last_became_peered":"2024-04-22T01:18:52.329098+0000","last_unstale":"2024-04-22T01:18:53.323009+0000","last_undegraded":"2024-04-22T01:18:53.323009+0000","last_fullsized":"2024-04-22T01:18:53.323009+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:08.054090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.326547+0000","last_change":"2024-04-22T01:18:51.295451+0000","last_active":"2024-04-22T01:18:52.326547+0000","last_peered":"2024-04-22T01:18:52.326547+0000","last_clean":"2024-04-22T01:18:52.326547+0000","last_became_active":"2024-04-22T01:18:51.295068+0000","last_became_peered":"2024-04-22T01:18:51.295068+0000","last_unstale":"2024-04-22T01:18:52.326547+0000","last_undegraded":"2024-04-22T01:18:52.326547+0000","last_fullsized":"2024-04-22T01:18:52.326547+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:01.570250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337532+0000","last_change":"2024-04-22T01:18:52.336315+0000","last_active":"2024-04-22T01:18:52.337532+0000","last_peered":"2024-04-22T01:18:52.337532+0000","last_clean":"2024-04-22T01:18:52.337532+0000","last_became_active":"2024-04-22T01:18:52.335962+0000","last_became_peered":"2024-04-22T01:18:52.335962+0000","last_unstale":"2024-04-22T01:18:52.337532+0000","last_undegraded":"2024-04-22T01:18:52.337532+0000","last_fullsized":"2024-04-22T01:18:52.337532+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:02.384790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321139+0000","last_change":"2024-04-22T01:18:51.316135+0000","last_active":"2024-04-22T01:18:53.321139+0000","last_peered":"2024-04-22T01:18:53.321139+0000","last_clean":"2024-04-22T01:18:53.321139+0000","last_became_active":"2024-04-22T01:18:51.315520+0000","last_became_peered":"2024-04-22T01:18:51.315520+0000","last_unstale":"2024-04-22T01:18:53.321139+0000","last_undegraded":"2024-04-22T01:18:53.321139+0000","last_fullsized":"2024-04-22T01:18:53.321139+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:59.636525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337140+0000","last_change":"2024-04-22T01:18:52.335124+0000","last_active":"2024-04-22T01:18:52.337140+0000","last_peered":"2024-04-22T01:18:52.337140+0000","last_clean":"2024-04-22T01:18:52.337140+0000","last_became_active":"2024-04-22T01:18:52.334828+0000","last_became_peered":"2024-04-22T01:18:52.334828+0000","last_unstale":"2024-04-22T01:18:52.337140+0000","last_undegraded":"2024-04-22T01:18:52.337140+0000","last_fullsized":"2024-04-22T01:18:52.337140+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:35.664356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.302689+0000","last_change":"2024-04-22T01:18:51.312864+0000","last_active":"2024-04-22T01:18:52.302689+0000","last_peered":"2024-04-22T01:18:52.302689+0000","last_clean":"2024-04-22T01:18:52.302689+0000","last_became_active":"2024-04-22T01:18:51.312664+0000","last_became_peered":"2024-04-22T01:18:51.312664+0000","last_unstale":"2024-04-22T01:18:52.302689+0000","last_undegraded":"2024-04-22T01:18:52.302689+0000","last_fullsized":"2024-04-22T01:18:52.302689+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:01.499168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.592090+0000","last_change":"2024-04-22T01:18:52.355090+0000","last_active":"2024-04-22T01:18:53.592090+0000","last_peered":"2024-04-22T01:18:53.592090+0000","last_clean":"2024-04-22T01:18:53.592090+0000","last_became_active":"2024-04-22T01:18:52.353987+0000","last_became_peered":"2024-04-22T01:18:52.353987+0000","last_unstale":"2024-04-22T01:18:53.592090+0000","last_undegraded":"2024-04-22T01:18:53.592090+0000","last_fullsized":"2024-04-22T01:18:53.592090+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:06.681876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.312435+0000","last_change":"2024-04-22T01:18:51.305955+0000","last_active":"2024-04-22T01:18:52.312435+0000","last_peered":"2024-04-22T01:18:52.312435+0000","last_clean":"2024-04-22T01:18:52.312435+0000","last_became_active":"2024-04-22T01:18:51.305106+0000","last_became_peered":"2024-04-22T01:18:51.305106+0000","last_unstale":"2024-04-22T01:18:52.312435+0000","last_undegraded":"2024-04-22T01:18:52.312435+0000","last_fullsized":"2024-04-22T01:18:52.312435+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:13.306518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.324115+0000","last_change":"2024-04-22T01:18:52.341025+0000","last_active":"2024-04-22T01:18:53.324115+0000","last_peered":"2024-04-22T01:18:53.324115+0000","last_clean":"2024-04-22T01:18:53.324115+0000","last_became_active":"2024-04-22T01:18:52.340112+0000","last_became_peered":"2024-04-22T01:18:52.340112+0000","last_unstale":"2024-04-22T01:18:53.324115+0000","last_undegraded":"2024-04-22T01:18:53.324115+0000","last_fullsized":"2024-04-22T01:18:53.324115+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:22.805839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.304486+0000","last_change":"2024-04-22T01:18:51.311664+0000","last_active":"2024-04-22T01:18:52.304486+0000","last_peered":"2024-04-22T01:18:52.304486+0000","last_clean":"2024-04-22T01:18:52.304486+0000","last_became_active":"2024-04-22T01:18:51.311362+0000","last_became_peered":"2024-04-22T01:18:51.311362+0000","last_unstale":"2024-04-22T01:18:52.304486+0000","last_undegraded":"2024-04-22T01:18:52.304486+0000","last_fullsized":"2024-04-22T01:18:52.304486+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:52.087842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.322101+0000","last_change":"2024-04-22T01:18:52.330939+0000","last_active":"2024-04-22T01:18:53.322101+0000","last_peered":"2024-04-22T01:18:53.322101+0000","last_clean":"2024-04-22T01:18:53.322101+0000","last_became_active":"2024-04-22T01:18:52.327407+0000","last_became_peered":"2024-04-22T01:18:52.327407+0000","last_unstale":"2024-04-22T01:18:53.322101+0000","last_undegraded":"2024-04-22T01:18:53.322101+0000","last_fullsized":"2024-04-22T01:18:53.322101+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.322839+0000","last_change":"2024-04-22T01:18:51.297303+0000","last_active":"2024-04-22T01:18:52.322839+0000","last_peered":"2024-04-22T01:18:52.322839+0000","last_clean":"2024-04-22T01:18:52.322839+0000","last_became_active":"2024-04-22T01:18:51.291301+0000","last_became_peered":"2024-04-22T01:18:51.291301+0000","last_unstale":"2024-04-22T01:18:52.322839+0000","last_undegraded":"2024-04-22T01:18:52.322839+0000","last_fullsized":"2024-04-22T01:18:52.322839+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:18.938085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":318,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591219+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:18:53.591219+0000","last_peered":"2024-04-22T01:18:53.591219+0000","last_clean":"2024-04-22T01:18:53.591219+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:18:53.591219+0000","last_undegraded":"2024-04-22T01:18:53.591219+0000","last_fullsized":"2024-04-22T01:18:53.591219+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1867776,"data_stored":1173705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":585393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1265664,"data_stored":583177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32428,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339412,"statfs":{"total":96636764160,"available":96603557888,"internally_reserved":0,"allocated":1273856,"data_stored":583269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1269760,"data_stored":583759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1269760,"data_stored":583211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1269760,"data_stored":583211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1888256,"data_stored":1181939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:18:55.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:18:55.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.475+0000 7f607b9c5640 1 Processor -- start 2024-04-22T01:18:55.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.475+0000 7f607b9c5640 1 -- start start 2024-04-22T01:18:55.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f607b9c5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60741455b0 0x7f60740cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f607b9c5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60740d0250 con 0x7f6074072230 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f607b9c5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60740cd7b0 con 0x7f60740cc150 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f607b9c5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f60740cc710 con 0x7f60741455b0 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079761640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60741455b0 0x7f60740cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079f62640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6074072230 0x7f607406e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49942/0 (socket says 172.21.15.70:49942) 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079f62640 1 -- 172.21.15.70:0/2508011584 learned_addr learned my addr 172.21.15.70:0/2508011584 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079761640 1 -- 172.21.15.70:0/2508011584 >> v1:172.21.15.98:6789/0 conn(0x7f60740cc150 legacy=0x7f6074071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079761640 1 -- 172.21.15.70:0/2508011584 >> v1:172.21.15.70:6789/0 conn(0x7f6074072230 legacy=0x7f607406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079761640 1 -- 172.21.15.70:0/2508011584 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60740cc9f0 con 0x7f60741455b0 2024-04-22T01:18:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f606bfff640 1 -- 172.21.15.70:0/2508011584 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2917038393 0 0) 0x7f60740d0250 con 0x7f6074072230 2024-04-22T01:18:55.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f6079761640 1 --2- 172.21.15.70:0/2508011584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60741455b0 0x7f60740cfd10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f60700097c0 tx=0x7f607001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c918da27d8d4a4ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f606bfff640 1 -- 172.21.15.70:0/2508011584 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700b0070 con 0x7f60741455b0 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f606bfff640 1 -- 172.21.15.70:0/2508011584 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60700ab600 con 0x7f60741455b0 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.479+0000 7f606bfff640 1 -- 172.21.15.70:0/2508011584 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700ab9d0 con 0x7f60741455b0 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/2508011584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60741455b0 msgr2=0x7f60740cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 --2- 172.21.15.70:0/2508011584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60741455b0 0x7f60740cfd10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f60700097c0 tx=0x7f607001c340 comp rx=0 tx=0).stop 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/2508011584 shutdown_connections 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 --2- 172.21.15.70:0/2508011584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60741455b0 0x7f60740cfd10 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/2508011584 >> 172.21.15.70:0/2508011584 conn(0x7f607406b1f0 msgr2=0x7f607406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/2508011584 shutdown_connections 2024-04-22T01:18:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/2508011584 wait complete. 2024-04-22T01:18:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 Processor -- start 2024-04-22T01:18:55.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- start start 2024-04-22T01:18:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 0x7f607413a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60740cc150 0x7f607415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60741455b0 0x7f607415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6074169ee0 con 0x7f6074072230 2024-04-22T01:18:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6074169d60 con 0x7f60740cc150 2024-04-22T01:18:55.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f607416a060 con 0x7f60741455b0 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 0x7f607413a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 0x7f607413a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60418/0 (socket says 172.21.15.70:60418) 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 -- 172.21.15.70:0/3715962057 learned_addr learned my addr 172.21.15.70:0/3715962057 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 -- 172.21.15.70:0/3715962057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60741455b0 msgr2=0x7f607415b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60741455b0 0x7f607415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 -- 172.21.15.70:0/3715962057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60740cc150 msgr2=0x7f607415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60740cc150 0x7f607415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 -- 172.21.15.70:0/3715962057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6074140280 con 0x7f6074072230 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6079761640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 0x7f607413a470 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f60700ae040 tx=0x7f607001ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700bb070 con 0x7f6074072230 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6070004020 con 0x7f6074072230 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607413f1e0 con 0x7f6074072230 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700b0040 con 0x7f6074072230 2024-04-22T01:18:55.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.483+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607413f7a0 con 0x7f6074072230 2024-04-22T01:18:55.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.487+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6074068210 con 0x7f6074072230 2024-04-22T01:18:55.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.487+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6070003620 con 0x7f6074072230 2024-04-22T01:18:55.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.487+0000 7f6069ffb640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60580678c0 0x7f6058069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.487+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f6070120540 con 0x7f6074072230 2024-04-22T01:18:55.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.487+0000 7f6078f60640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60580678c0 0x7f6058069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.487+0000 7f6078f60640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60580678c0 0x7f6058069d80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6074067c60 tx=0x7f6064006bd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.491+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6074140280 con 0x7f6074072230 2024-04-22T01:18:55.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.647+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f607415f510 con 0x7f60580678c0 2024-04-22T01:18:55.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.651+0000 7f6069ffb640 1 -- 172.21.15.70:0/3715962057 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252454 (secure 0 0 0) 0x7f607415f510 con 0x7f60580678c0 2024-04-22T01:18:55.656 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:55.659 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:18:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60580678c0 msgr2=0x7f6058069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60580678c0 0x7f6058069d80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6074067c60 tx=0x7f6064006bd0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 msgr2=0x7f607413a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 0x7f607413a470 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f60700ae040 tx=0x7f607001ab80 comp rx=0 tx=0).stop 2024-04-22T01:18:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 shutdown_connections 2024-04-22T01:18:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f60580678c0 0x7f6058069d80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60741455b0 0x7f607415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60740cc150 0x7f607415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 --2- 172.21.15.70:0/3715962057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6074072230 0x7f607413a470 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 >> 172.21.15.70:0/3715962057 conn(0x7f607406b1f0 msgr2=0x7f60740b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 shutdown_connections 2024-04-22T01:18:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.655+0000 7f607b9c5640 1 -- 172.21.15.70:0/3715962057 wait complete. 2024-04-22T01:18:55.673 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-04-22T01:18:54.659127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277252,"kb_used_data":11124,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754697468,"statfs":{"total":773094113280,"available":772810207232,"internally_reserved":0,"allocated":11390976,"data_stored":5857664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.397436"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318647+0000","last_change":"2024-04-22T01:18:52.314149+0000","last_active":"2024-04-22T01:18:53.318647+0000","last_peered":"2024-04-22T01:18:53.318647+0000","last_clean":"2024-04-22T01:18:53.318647+0000","last_became_active":"2024-04-22T01:18:52.313904+0000","last_became_peered":"2024-04-22T01:18:52.313904+0000","last_unstale":"2024-04-22T01:18:53.318647+0000","last_undegraded":"2024-04-22T01:18:53.318647+0000","last_fullsized":"2024-04-22T01:18:53.318647+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:01.926744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361953+0000","last_change":"2024-04-22T01:18:52.356375+0000","last_active":"2024-04-22T01:18:52.361953+0000","last_peered":"2024-04-22T01:18:52.361953+0000","last_clean":"2024-04-22T01:18:52.361953+0000","last_became_active":"2024-04-22T01:18:52.348920+0000","last_became_peered":"2024-04-22T01:18:52.348920+0000","last_unstale":"2024-04-22T01:18:52.361953+0000","last_undegraded":"2024-04-22T01:18:52.361953+0000","last_fullsized":"2024-04-22T01:18:52.361953+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318481+0000","last_change":"2024-04-22T01:18:52.307572+0000","last_active":"2024-04-22T01:18:53.318481+0000","last_peered":"2024-04-22T01:18:53.318481+0000","last_clean":"2024-04-22T01:18:53.318481+0000","last_became_active":"2024-04-22T01:18:52.307257+0000","last_became_peered":"2024-04-22T01:18:52.307257+0000","last_unstale":"2024-04-22T01:18:53.318481+0000","last_undegraded":"2024-04-22T01:18:53.318481+0000","last_fullsized":"2024-04-22T01:18:53.318481+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:28:21.608094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.347742+0000","last_change":"2024-04-22T01:18:52.347162+0000","last_active":"2024-04-22T01:18:52.347742+0000","last_peered":"2024-04-22T01:18:52.347742+0000","last_clean":"2024-04-22T01:18:52.347742+0000","last_became_active":"2024-04-22T01:18:52.346337+0000","last_became_peered":"2024-04-22T01:18:52.346337+0000","last_unstale":"2024-04-22T01:18:52.347742+0000","last_undegraded":"2024-04-22T01:18:52.347742+0000","last_fullsized":"2024-04-22T01:18:52.347742+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:01.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.347284+0000","last_change":"2024-04-22T01:18:52.346935+0000","last_active":"2024-04-22T01:18:52.347284+0000","last_peered":"2024-04-22T01:18:52.347284+0000","last_clean":"2024-04-22T01:18:52.347284+0000","last_became_active":"2024-04-22T01:18:52.346680+0000","last_became_peered":"2024-04-22T01:18:52.346680+0000","last_unstale":"2024-04-22T01:18:52.347284+0000","last_undegraded":"2024-04-22T01:18:52.347284+0000","last_fullsized":"2024-04-22T01:18:52.347284+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361925+0000","last_change":"2024-04-22T01:18:52.356098+0000","last_active":"2024-04-22T01:18:52.361925+0000","last_peered":"2024-04-22T01:18:52.361925+0000","last_clean":"2024-04-22T01:18:52.361925+0000","last_became_active":"2024-04-22T01:18:52.340731+0000","last_became_peered":"2024-04-22T01:18:52.340731+0000","last_unstale":"2024-04-22T01:18:52.361925+0000","last_undegraded":"2024-04-22T01:18:52.361925+0000","last_fullsized":"2024-04-22T01:18:52.361925+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:52.479137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361992+0000","last_change":"2024-04-22T01:18:52.356791+0000","last_active":"2024-04-22T01:18:52.361992+0000","last_peered":"2024-04-22T01:18:52.361992+0000","last_clean":"2024-04-22T01:18:52.361992+0000","last_became_active":"2024-04-22T01:18:52.355533+0000","last_became_peered":"2024-04-22T01:18:52.355533+0000","last_unstale":"2024-04-22T01:18:52.361992+0000","last_undegraded":"2024-04-22T01:18:52.361992+0000","last_fullsized":"2024-04-22T01:18:52.361992+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:01.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.329204+0000","last_change":"2024-04-22T01:18:52.327508+0000","last_active":"2024-04-22T01:18:52.329204+0000","last_peered":"2024-04-22T01:18:52.329204+0000","last_clean":"2024-04-22T01:18:52.329204+0000","last_became_active":"2024-04-22T01:18:52.327030+0000","last_became_peered":"2024-04-22T01:18:52.327030+0000","last_unstale":"2024-04-22T01:18:52.329204+0000","last_undegraded":"2024-04-22T01:18:52.329204+0000","last_fullsized":"2024-04-22T01:18:52.329204+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:35.303367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.357415+0000","last_change":"2024-04-22T01:18:52.350875+0000","last_active":"2024-04-22T01:18:52.357415+0000","last_peered":"2024-04-22T01:18:52.357415+0000","last_clean":"2024-04-22T01:18:52.357415+0000","last_became_active":"2024-04-22T01:18:52.349998+0000","last_became_peered":"2024-04-22T01:18:52.349998+0000","last_unstale":"2024-04-22T01:18:52.357415+0000","last_undegraded":"2024-04-22T01:18:52.357415+0000","last_fullsized":"2024-04-22T01:18:52.357415+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:56:41.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.359900+0000","last_change":"2024-04-22T01:18:52.359516+0000","last_active":"2024-04-22T01:18:52.359900+0000","last_peered":"2024-04-22T01:18:52.359900+0000","last_clean":"2024-04-22T01:18:52.359900+0000","last_became_active":"2024-04-22T01:18:52.359240+0000","last_became_peered":"2024-04-22T01:18:52.359240+0000","last_unstale":"2024-04-22T01:18:52.359900+0000","last_undegraded":"2024-04-22T01:18:52.359900+0000","last_fullsized":"2024-04-22T01:18:52.359900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:12.739140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323538+0000","last_change":"2024-04-22T01:18:52.358815+0000","last_active":"2024-04-22T01:18:53.323538+0000","last_peered":"2024-04-22T01:18:53.323538+0000","last_clean":"2024-04-22T01:18:53.323538+0000","last_became_active":"2024-04-22T01:18:52.358253+0000","last_became_peered":"2024-04-22T01:18:52.358253+0000","last_unstale":"2024-04-22T01:18:53.323538+0000","last_undegraded":"2024-04-22T01:18:53.323538+0000","last_fullsized":"2024-04-22T01:18:53.323538+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:10.638308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321503+0000","last_change":"2024-04-22T01:18:52.317096+0000","last_active":"2024-04-22T01:18:53.321503+0000","last_peered":"2024-04-22T01:18:53.321503+0000","last_clean":"2024-04-22T01:18:53.321503+0000","last_became_active":"2024-04-22T01:18:52.316306+0000","last_became_peered":"2024-04-22T01:18:52.316306+0000","last_unstale":"2024-04-22T01:18:53.321503+0000","last_undegraded":"2024-04-22T01:18:53.321503+0000","last_fullsized":"2024-04-22T01:18:53.321503+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:01.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.354862+0000","last_change":"2024-04-22T01:18:52.354479+0000","last_active":"2024-04-22T01:18:52.354862+0000","last_peered":"2024-04-22T01:18:52.354862+0000","last_clean":"2024-04-22T01:18:52.354862+0000","last_became_active":"2024-04-22T01:18:52.354195+0000","last_became_peered":"2024-04-22T01:18:52.354195+0000","last_unstale":"2024-04-22T01:18:52.354862+0000","last_undegraded":"2024-04-22T01:18:52.354862+0000","last_fullsized":"2024-04-22T01:18:52.354862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591761+0000","last_change":"2024-04-22T01:18:52.332159+0000","last_active":"2024-04-22T01:18:53.591761+0000","last_peered":"2024-04-22T01:18:53.591761+0000","last_clean":"2024-04-22T01:18:53.591761+0000","last_became_active":"2024-04-22T01:18:52.331379+0000","last_became_peered":"2024-04-22T01:18:52.331379+0000","last_unstale":"2024-04-22T01:18:53.591761+0000","last_undegraded":"2024-04-22T01:18:53.591761+0000","last_fullsized":"2024-04-22T01:18:53.591761+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:09:57.302694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.592091+0000","last_change":"2024-04-22T01:18:52.354786+0000","last_active":"2024-04-22T01:18:53.592091+0000","last_peered":"2024-04-22T01:18:53.592091+0000","last_clean":"2024-04-22T01:18:53.592091+0000","last_became_active":"2024-04-22T01:18:52.353998+0000","last_became_peered":"2024-04-22T01:18:52.353998+0000","last_unstale":"2024-04-22T01:18:53.592091+0000","last_undegraded":"2024-04-22T01:18:53.592091+0000","last_fullsized":"2024-04-22T01:18:53.592091+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:00:55.019846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.592054+0000","last_change":"2024-04-22T01:18:52.334389+0000","last_active":"2024-04-22T01:18:53.592054+0000","last_peered":"2024-04-22T01:18:53.592054+0000","last_clean":"2024-04-22T01:18:53.592054+0000","last_became_active":"2024-04-22T01:18:52.333487+0000","last_became_peered":"2024-04-22T01:18:52.333487+0000","last_unstale":"2024-04-22T01:18:53.592054+0000","last_undegraded":"2024-04-22T01:18:53.592054+0000","last_fullsized":"2024-04-22T01:18:53.592054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:09:12.835038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318727+0000","last_change":"2024-04-22T01:18:52.329841+0000","last_active":"2024-04-22T01:18:53.318727+0000","last_peered":"2024-04-22T01:18:53.318727+0000","last_clean":"2024-04-22T01:18:53.318727+0000","last_became_active":"2024-04-22T01:18:52.329550+0000","last_became_peered":"2024-04-22T01:18:52.329550+0000","last_unstale":"2024-04-22T01:18:53.318727+0000","last_undegraded":"2024-04-22T01:18:53.318727+0000","last_fullsized":"2024-04-22T01:18:53.318727+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:57.240873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361311+0000","last_change":"2024-04-22T01:18:52.355588+0000","last_active":"2024-04-22T01:18:52.361311+0000","last_peered":"2024-04-22T01:18:52.361311+0000","last_clean":"2024-04-22T01:18:52.361311+0000","last_became_active":"2024-04-22T01:18:52.340740+0000","last_became_peered":"2024-04-22T01:18:52.340740+0000","last_unstale":"2024-04-22T01:18:52.361311+0000","last_undegraded":"2024-04-22T01:18:52.361311+0000","last_fullsized":"2024-04-22T01:18:52.361311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:04:42.756386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318630+0000","last_change":"2024-04-22T01:18:52.358493+0000","last_active":"2024-04-22T01:18:53.318630+0000","last_peered":"2024-04-22T01:18:53.318630+0000","last_clean":"2024-04-22T01:18:53.318630+0000","last_became_active":"2024-04-22T01:18:52.358164+0000","last_became_peered":"2024-04-22T01:18:52.358164+0000","last_unstale":"2024-04-22T01:18:53.318630+0000","last_undegraded":"2024-04-22T01:18:53.318630+0000","last_fullsized":"2024-04-22T01:18:53.318630+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:54:43.782657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321327+0000","last_change":"2024-04-22T01:18:52.335923+0000","last_active":"2024-04-22T01:18:53.321327+0000","last_peered":"2024-04-22T01:18:53.321327+0000","last_clean":"2024-04-22T01:18:53.321327+0000","last_became_active":"2024-04-22T01:18:52.334164+0000","last_became_peered":"2024-04-22T01:18:52.334164+0000","last_unstale":"2024-04-22T01:18:53.321327+0000","last_undegraded":"2024-04-22T01:18:53.321327+0000","last_fullsized":"2024-04-22T01:18:53.321327+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.323451+0000","last_change":"2024-04-22T01:18:52.322252+0000","last_active":"2024-04-22T01:18:52.323451+0000","last_peered":"2024-04-22T01:18:52.323451+0000","last_clean":"2024-04-22T01:18:52.323451+0000","last_became_active":"2024-04-22T01:18:52.317433+0000","last_became_peered":"2024-04-22T01:18:52.317433+0000","last_unstale":"2024-04-22T01:18:52.323451+0000","last_undegraded":"2024-04-22T01:18:52.323451+0000","last_fullsized":"2024-04-22T01:18:52.323451+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:04.752486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333992+0000","last_change":"2024-04-22T01:18:52.327197+0000","last_active":"2024-04-22T01:18:52.333992+0000","last_peered":"2024-04-22T01:18:52.333992+0000","last_clean":"2024-04-22T01:18:52.333992+0000","last_became_active":"2024-04-22T01:18:52.326654+0000","last_became_peered":"2024-04-22T01:18:52.326654+0000","last_unstale":"2024-04-22T01:18:52.333992+0000","last_undegraded":"2024-04-22T01:18:52.333992+0000","last_fullsized":"2024-04-22T01:18:52.333992+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318262+0000","last_change":"2024-04-22T01:18:52.340870+0000","last_active":"2024-04-22T01:18:53.318262+0000","last_peered":"2024-04-22T01:18:53.318262+0000","last_clean":"2024-04-22T01:18:53.318262+0000","last_became_active":"2024-04-22T01:18:52.340582+0000","last_became_peered":"2024-04-22T01:18:52.340582+0000","last_unstale":"2024-04-22T01:18:53.318262+0000","last_undegraded":"2024-04-22T01:18:53.318262+0000","last_fullsized":"2024-04-22T01:18:53.318262+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:24:05.347394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333912+0000","last_change":"2024-04-22T01:18:52.326186+0000","last_active":"2024-04-22T01:18:52.333912+0000","last_peered":"2024-04-22T01:18:52.333912+0000","last_clean":"2024-04-22T01:18:52.333912+0000","last_became_active":"2024-04-22T01:18:52.325885+0000","last_became_peered":"2024-04-22T01:18:52.325885+0000","last_unstale":"2024-04-22T01:18:52.333912+0000","last_undegraded":"2024-04-22T01:18:52.333912+0000","last_fullsized":"2024-04-22T01:18:52.333912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:56:41.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321720+0000","last_change":"2024-04-22T01:18:52.347405+0000","last_active":"2024-04-22T01:18:53.321720+0000","last_peered":"2024-04-22T01:18:53.321720+0000","last_clean":"2024-04-22T01:18:53.321720+0000","last_became_active":"2024-04-22T01:18:52.346612+0000","last_became_peered":"2024-04-22T01:18:52.346612+0000","last_unstale":"2024-04-22T01:18:53.321720+0000","last_undegraded":"2024-04-22T01:18:53.321720+0000","last_fullsized":"2024-04-22T01:18:53.321720+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321618+0000","last_change":"2024-04-22T01:18:52.351374+0000","last_active":"2024-04-22T01:18:53.321618+0000","last_peered":"2024-04-22T01:18:53.321618+0000","last_clean":"2024-04-22T01:18:53.321618+0000","last_became_active":"2024-04-22T01:18:52.350733+0000","last_became_peered":"2024-04-22T01:18:52.350733+0000","last_unstale":"2024-04-22T01:18:53.321618+0000","last_undegraded":"2024-04-22T01:18:53.321618+0000","last_fullsized":"2024-04-22T01:18:53.321618+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323751+0000","last_change":"2024-04-22T01:18:52.342467+0000","last_active":"2024-04-22T01:18:53.323751+0000","last_peered":"2024-04-22T01:18:53.323751+0000","last_clean":"2024-04-22T01:18:53.323751+0000","last_became_active":"2024-04-22T01:18:52.341578+0000","last_became_peered":"2024-04-22T01:18:52.341578+0000","last_unstale":"2024-04-22T01:18:53.323751+0000","last_undegraded":"2024-04-22T01:18:53.323751+0000","last_fullsized":"2024-04-22T01:18:53.323751+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:22.235466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.356638+0000","last_change":"2024-04-22T01:18:52.349240+0000","last_active":"2024-04-22T01:18:52.356638+0000","last_peered":"2024-04-22T01:18:52.356638+0000","last_clean":"2024-04-22T01:18:52.356638+0000","last_became_active":"2024-04-22T01:18:52.340896+0000","last_became_peered":"2024-04-22T01:18:52.340896+0000","last_unstale":"2024-04-22T01:18:52.356638+0000","last_undegraded":"2024-04-22T01:18:52.356638+0000","last_fullsized":"2024-04-22T01:18:52.356638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:39.029774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333747+0000","last_change":"2024-04-22T01:18:52.333354+0000","last_active":"2024-04-22T01:18:52.333747+0000","last_peered":"2024-04-22T01:18:52.333747+0000","last_clean":"2024-04-22T01:18:52.333747+0000","last_became_active":"2024-04-22T01:18:52.333069+0000","last_became_peered":"2024-04-22T01:18:52.333069+0000","last_unstale":"2024-04-22T01:18:52.333747+0000","last_undegraded":"2024-04-22T01:18:52.333747+0000","last_fullsized":"2024-04-22T01:18:52.333747+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:16.653262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.320705+0000","last_change":"2024-04-22T01:18:52.346605+0000","last_active":"2024-04-22T01:18:53.320705+0000","last_peered":"2024-04-22T01:18:53.320705+0000","last_clean":"2024-04-22T01:18:53.320705+0000","last_became_active":"2024-04-22T01:18:52.346007+0000","last_became_peered":"2024-04-22T01:18:52.346007+0000","last_unstale":"2024-04-22T01:18:53.320705+0000","last_undegraded":"2024-04-22T01:18:53.320705+0000","last_fullsized":"2024-04-22T01:18:53.320705+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:52.479137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.360688+0000","last_change":"2024-04-22T01:18:52.355633+0000","last_active":"2024-04-22T01:18:52.360688+0000","last_peered":"2024-04-22T01:18:52.360688+0000","last_clean":"2024-04-22T01:18:52.360688+0000","last_became_active":"2024-04-22T01:18:52.350967+0000","last_became_peered":"2024-04-22T01:18:52.350967+0000","last_unstale":"2024-04-22T01:18:52.360688+0000","last_undegraded":"2024-04-22T01:18:52.360688+0000","last_fullsized":"2024-04-22T01:18:52.360688+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:02.573459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.340972+0000","last_change":"2024-04-22T01:18:52.340194+0000","last_active":"2024-04-22T01:18:52.340972+0000","last_peered":"2024-04-22T01:18:52.340972+0000","last_clean":"2024-04-22T01:18:52.340972+0000","last_became_active":"2024-04-22T01:18:52.339904+0000","last_became_peered":"2024-04-22T01:18:52.339904+0000","last_unstale":"2024-04-22T01:18:52.340972+0000","last_undegraded":"2024-04-22T01:18:52.340972+0000","last_fullsized":"2024-04-22T01:18:52.340972+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:51:15.255767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318396+0000","last_change":"2024-04-22T01:18:52.341326+0000","last_active":"2024-04-22T01:18:53.318396+0000","last_peered":"2024-04-22T01:18:53.318396+0000","last_clean":"2024-04-22T01:18:53.318396+0000","last_became_active":"2024-04-22T01:18:52.341102+0000","last_became_peered":"2024-04-22T01:18:52.341102+0000","last_unstale":"2024-04-22T01:18:53.318396+0000","last_undegraded":"2024-04-22T01:18:53.318396+0000","last_fullsized":"2024-04-22T01:18:53.318396+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:12.237330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"102'4","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.358352+0000","last_change":"2024-04-22T01:18:51.315283+0000","last_active":"2024-04-22T01:18:52.358352+0000","last_peered":"2024-04-22T01:18:52.358352+0000","last_clean":"2024-04-22T01:18:52.358352+0000","last_became_active":"2024-04-22T01:18:51.315068+0000","last_became_peered":"2024-04-22T01:18:51.315068+0000","last_unstale":"2024-04-22T01:18:52.358352+0000","last_undegraded":"2024-04-22T01:18:52.358352+0000","last_fullsized":"2024-04-22T01:18:52.358352+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:04:41.753177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.318274+0000","last_change":"2024-04-22T01:18:52.317724+0000","last_active":"2024-04-22T01:18:52.318274+0000","last_peered":"2024-04-22T01:18:52.318274+0000","last_clean":"2024-04-22T01:18:52.318274+0000","last_became_active":"2024-04-22T01:18:52.317404+0000","last_became_peered":"2024-04-22T01:18:52.317404+0000","last_unstale":"2024-04-22T01:18:52.318274+0000","last_undegraded":"2024-04-22T01:18:52.318274+0000","last_fullsized":"2024-04-22T01:18:52.318274+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:30:47.755545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.332994+0000","last_change":"2024-04-22T01:18:51.308359+0000","last_active":"2024-04-22T01:18:52.332994+0000","last_peered":"2024-04-22T01:18:52.332994+0000","last_clean":"2024-04-22T01:18:52.332994+0000","last_became_active":"2024-04-22T01:18:51.307681+0000","last_became_peered":"2024-04-22T01:18:51.307681+0000","last_unstale":"2024-04-22T01:18:52.332994+0000","last_undegraded":"2024-04-22T01:18:52.332994+0000","last_fullsized":"2024-04-22T01:18:52.332994+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:29.570206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321104+0000","last_change":"2024-04-22T01:18:52.327103+0000","last_active":"2024-04-22T01:18:53.321104+0000","last_peered":"2024-04-22T01:18:53.321104+0000","last_clean":"2024-04-22T01:18:53.321104+0000","last_became_active":"2024-04-22T01:18:52.326404+0000","last_became_peered":"2024-04-22T01:18:52.326404+0000","last_unstale":"2024-04-22T01:18:53.321104+0000","last_undegraded":"2024-04-22T01:18:53.321104+0000","last_fullsized":"2024-04-22T01:18:53.321104+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321206+0000","last_change":"2024-04-22T01:18:51.311531+0000","last_active":"2024-04-22T01:18:53.321206+0000","last_peered":"2024-04-22T01:18:53.321206+0000","last_clean":"2024-04-22T01:18:53.321206+0000","last_became_active":"2024-04-22T01:18:51.310780+0000","last_became_peered":"2024-04-22T01:18:51.310780+0000","last_unstale":"2024-04-22T01:18:53.321206+0000","last_undegraded":"2024-04-22T01:18:53.321206+0000","last_fullsized":"2024-04-22T01:18:53.321206+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:07.837857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323053+0000","last_change":"2024-04-22T01:18:52.347947+0000","last_active":"2024-04-22T01:18:53.323053+0000","last_peered":"2024-04-22T01:18:53.323053+0000","last_clean":"2024-04-22T01:18:53.323053+0000","last_became_active":"2024-04-22T01:18:52.347277+0000","last_became_peered":"2024-04-22T01:18:52.347277+0000","last_unstale":"2024-04-22T01:18:53.323053+0000","last_undegraded":"2024-04-22T01:18:53.323053+0000","last_fullsized":"2024-04-22T01:18:53.323053+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:38.433456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.346807+0000","last_change":"2024-04-22T01:18:51.302794+0000","last_active":"2024-04-22T01:18:52.346807+0000","last_peered":"2024-04-22T01:18:52.346807+0000","last_clean":"2024-04-22T01:18:52.346807+0000","last_became_active":"2024-04-22T01:18:51.302436+0000","last_became_peered":"2024-04-22T01:18:51.302436+0000","last_unstale":"2024-04-22T01:18:52.346807+0000","last_undegraded":"2024-04-22T01:18:52.346807+0000","last_fullsized":"2024-04-22T01:18:52.346807+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:04:41.753177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.336488+0000","last_change":"2024-04-22T01:18:52.335978+0000","last_active":"2024-04-22T01:18:52.336488+0000","last_peered":"2024-04-22T01:18:52.336488+0000","last_clean":"2024-04-22T01:18:52.336488+0000","last_became_active":"2024-04-22T01:18:52.335651+0000","last_became_peered":"2024-04-22T01:18:52.335651+0000","last_unstale":"2024-04-22T01:18:52.336488+0000","last_undegraded":"2024-04-22T01:18:52.336488+0000","last_fullsized":"2024-04-22T01:18:52.336488+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591760+0000","last_change":"2024-04-22T01:18:51.298870+0000","last_active":"2024-04-22T01:18:53.591760+0000","last_peered":"2024-04-22T01:18:53.591760+0000","last_clean":"2024-04-22T01:18:53.591760+0000","last_became_active":"2024-04-22T01:18:51.298047+0000","last_became_peered":"2024-04-22T01:18:51.298047+0000","last_unstale":"2024-04-22T01:18:53.591760+0000","last_undegraded":"2024-04-22T01:18:53.591760+0000","last_fullsized":"2024-04-22T01:18:53.591760+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:21.232256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318757+0000","last_change":"2024-04-22T01:18:52.358235+0000","last_active":"2024-04-22T01:18:53.318757+0000","last_peered":"2024-04-22T01:18:53.318757+0000","last_clean":"2024-04-22T01:18:53.318757+0000","last_became_active":"2024-04-22T01:18:52.358007+0000","last_became_peered":"2024-04-22T01:18:52.358007+0000","last_unstale":"2024-04-22T01:18:53.318757+0000","last_undegraded":"2024-04-22T01:18:53.318757+0000","last_fullsized":"2024-04-22T01:18:53.318757+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:30:16.786285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.322609+0000","last_change":"2024-04-22T01:18:51.303578+0000","last_active":"2024-04-22T01:18:53.322609+0000","last_peered":"2024-04-22T01:18:53.322609+0000","last_clean":"2024-04-22T01:18:53.322609+0000","last_became_active":"2024-04-22T01:18:51.302994+0000","last_became_peered":"2024-04-22T01:18:51.302994+0000","last_unstale":"2024-04-22T01:18:53.322609+0000","last_undegraded":"2024-04-22T01:18:53.322609+0000","last_fullsized":"2024-04-22T01:18:53.322609+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:38.026564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.362081+0000","last_change":"2024-04-22T01:18:52.357882+0000","last_active":"2024-04-22T01:18:52.362081+0000","last_peered":"2024-04-22T01:18:52.362081+0000","last_clean":"2024-04-22T01:18:52.362081+0000","last_became_active":"2024-04-22T01:18:52.348758+0000","last_became_peered":"2024-04-22T01:18:52.348758+0000","last_unstale":"2024-04-22T01:18:52.362081+0000","last_undegraded":"2024-04-22T01:18:52.362081+0000","last_fullsized":"2024-04-22T01:18:52.362081+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.320420+0000","last_change":"2024-04-22T01:18:51.309550+0000","last_active":"2024-04-22T01:18:53.320420+0000","last_peered":"2024-04-22T01:18:53.320420+0000","last_clean":"2024-04-22T01:18:53.320420+0000","last_became_active":"2024-04-22T01:18:51.303367+0000","last_became_peered":"2024-04-22T01:18:51.303367+0000","last_unstale":"2024-04-22T01:18:53.320420+0000","last_undegraded":"2024-04-22T01:18:53.320420+0000","last_fullsized":"2024-04-22T01:18:53.320420+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:40:39.157115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.320782+0000","last_change":"2024-04-22T01:18:52.313086+0000","last_active":"2024-04-22T01:18:52.320782+0000","last_peered":"2024-04-22T01:18:52.320782+0000","last_clean":"2024-04-22T01:18:52.320782+0000","last_became_active":"2024-04-22T01:18:52.312403+0000","last_became_peered":"2024-04-22T01:18:52.312403+0000","last_unstale":"2024-04-22T01:18:52.320782+0000","last_undegraded":"2024-04-22T01:18:52.320782+0000","last_fullsized":"2024-04-22T01:18:52.320782+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.304978+0000","last_change":"2024-04-22T01:18:51.282774+0000","last_active":"2024-04-22T01:18:52.304978+0000","last_peered":"2024-04-22T01:18:52.304978+0000","last_clean":"2024-04-22T01:18:52.304978+0000","last_became_active":"2024-04-22T01:18:51.282219+0000","last_became_peered":"2024-04-22T01:18:51.282219+0000","last_unstale":"2024-04-22T01:18:52.304978+0000","last_undegraded":"2024-04-22T01:18:52.304978+0000","last_fullsized":"2024-04-22T01:18:52.304978+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:01.570250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318296+0000","last_change":"2024-04-22T01:18:52.341648+0000","last_active":"2024-04-22T01:18:53.318296+0000","last_peered":"2024-04-22T01:18:53.318296+0000","last_clean":"2024-04-22T01:18:53.318296+0000","last_became_active":"2024-04-22T01:18:52.341296+0000","last_became_peered":"2024-04-22T01:18:52.341296+0000","last_unstale":"2024-04-22T01:18:53.318296+0000","last_undegraded":"2024-04-22T01:18:53.318296+0000","last_fullsized":"2024-04-22T01:18:53.318296+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:59:50.633160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321953+0000","last_change":"2024-04-22T01:18:51.311920+0000","last_active":"2024-04-22T01:18:53.321953+0000","last_peered":"2024-04-22T01:18:53.321953+0000","last_clean":"2024-04-22T01:18:53.321953+0000","last_became_active":"2024-04-22T01:18:51.310648+0000","last_became_peered":"2024-04-22T01:18:51.310648+0000","last_unstale":"2024-04-22T01:18:53.321953+0000","last_undegraded":"2024-04-22T01:18:53.321953+0000","last_fullsized":"2024-04-22T01:18:53.321953+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:03:01.467024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.334549+0000","last_change":"2024-04-22T01:18:52.323570+0000","last_active":"2024-04-22T01:18:52.334549+0000","last_peered":"2024-04-22T01:18:52.334549+0000","last_clean":"2024-04-22T01:18:52.334549+0000","last_became_active":"2024-04-22T01:18:52.322934+0000","last_became_peered":"2024-04-22T01:18:52.322934+0000","last_unstale":"2024-04-22T01:18:52.334549+0000","last_undegraded":"2024-04-22T01:18:52.334549+0000","last_fullsized":"2024-04-22T01:18:52.334549+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:39.029774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318595+0000","last_change":"2024-04-22T01:18:51.303127+0000","last_active":"2024-04-22T01:18:53.318595+0000","last_peered":"2024-04-22T01:18:53.318595+0000","last_clean":"2024-04-22T01:18:53.318595+0000","last_became_active":"2024-04-22T01:18:51.302773+0000","last_became_peered":"2024-04-22T01:18:51.302773+0000","last_unstale":"2024-04-22T01:18:53.318595+0000","last_undegraded":"2024-04-22T01:18:53.318595+0000","last_fullsized":"2024-04-22T01:18:53.318595+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:01:16.888369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.329450+0000","last_change":"2024-04-22T01:18:52.327804+0000","last_active":"2024-04-22T01:18:52.329450+0000","last_peered":"2024-04-22T01:18:52.329450+0000","last_clean":"2024-04-22T01:18:52.329450+0000","last_became_active":"2024-04-22T01:18:52.327100+0000","last_became_peered":"2024-04-22T01:18:52.327100+0000","last_unstale":"2024-04-22T01:18:52.329450+0000","last_undegraded":"2024-04-22T01:18:52.329450+0000","last_fullsized":"2024-04-22T01:18:52.329450+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:09:24.954206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"102'3","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591842+0000","last_change":"2024-04-22T01:18:51.289102+0000","last_active":"2024-04-22T01:18:53.591842+0000","last_peered":"2024-04-22T01:18:53.591842+0000","last_clean":"2024-04-22T01:18:53.591842+0000","last_became_active":"2024-04-22T01:18:51.288334+0000","last_became_peered":"2024-04-22T01:18:51.288334+0000","last_unstale":"2024-04-22T01:18:53.591842+0000","last_undegraded":"2024-04-22T01:18:53.591842+0000","last_fullsized":"2024-04-22T01:18:53.591842+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:37.430246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.333950+0000","last_change":"2024-04-22T01:18:52.326573+0000","last_active":"2024-04-22T01:18:52.333950+0000","last_peered":"2024-04-22T01:18:52.333950+0000","last_clean":"2024-04-22T01:18:52.333950+0000","last_became_active":"2024-04-22T01:18:52.325971+0000","last_became_peered":"2024-04-22T01:18:52.325971+0000","last_unstale":"2024-04-22T01:18:52.333950+0000","last_undegraded":"2024-04-22T01:18:52.333950+0000","last_fullsized":"2024-04-22T01:18:52.333950+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:01.822312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"102'6","reported_seq":18,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.362747+0000","last_change":"2024-04-22T01:18:51.311610+0000","last_active":"2024-04-22T01:18:52.362747+0000","last_peered":"2024-04-22T01:18:52.362747+0000","last_clean":"2024-04-22T01:18:52.362747+0000","last_became_active":"2024-04-22T01:18:51.305910+0000","last_became_peered":"2024-04-22T01:18:51.305910+0000","last_unstale":"2024-04-22T01:18:52.362747+0000","last_undegraded":"2024-04-22T01:18:52.362747+0000","last_fullsized":"2024-04-22T01:18:52.362747+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:57.349352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.322994+0000","last_change":"2024-04-22T01:18:52.330283+0000","last_active":"2024-04-22T01:18:53.322994+0000","last_peered":"2024-04-22T01:18:53.322994+0000","last_clean":"2024-04-22T01:18:53.322994+0000","last_became_active":"2024-04-22T01:18:52.328732+0000","last_became_peered":"2024-04-22T01:18:52.328732+0000","last_unstale":"2024-04-22T01:18:53.322994+0000","last_undegraded":"2024-04-22T01:18:53.322994+0000","last_fullsized":"2024-04-22T01:18:53.322994+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:12.739140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337539+0000","last_change":"2024-04-22T01:18:51.299682+0000","last_active":"2024-04-22T01:18:52.337539+0000","last_peered":"2024-04-22T01:18:52.337539+0000","last_clean":"2024-04-22T01:18:52.337539+0000","last_became_active":"2024-04-22T01:18:51.299324+0000","last_became_peered":"2024-04-22T01:18:51.299324+0000","last_unstale":"2024-04-22T01:18:52.337539+0000","last_undegraded":"2024-04-22T01:18:52.337539+0000","last_fullsized":"2024-04-22T01:18:52.337539+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:51.475927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.362033+0000","last_change":"2024-04-22T01:18:52.357031+0000","last_active":"2024-04-22T01:18:52.362033+0000","last_peered":"2024-04-22T01:18:52.362033+0000","last_clean":"2024-04-22T01:18:52.362033+0000","last_became_active":"2024-04-22T01:18:52.349376+0000","last_became_peered":"2024-04-22T01:18:52.349376+0000","last_unstale":"2024-04-22T01:18:52.362033+0000","last_undegraded":"2024-04-22T01:18:52.362033+0000","last_fullsized":"2024-04-22T01:18:52.362033+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.306549+0000","last_change":"2024-04-22T01:18:51.282341+0000","last_active":"2024-04-22T01:18:52.306549+0000","last_peered":"2024-04-22T01:18:52.306549+0000","last_clean":"2024-04-22T01:18:52.306549+0000","last_became_active":"2024-04-22T01:18:51.281442+0000","last_became_peered":"2024-04-22T01:18:51.281442+0000","last_unstale":"2024-04-22T01:18:52.306549+0000","last_undegraded":"2024-04-22T01:18:52.306549+0000","last_fullsized":"2024-04-22T01:18:52.306549+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:50:51.475927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.341044+0000","last_change":"2024-04-22T01:18:52.340116+0000","last_active":"2024-04-22T01:18:52.341044+0000","last_peered":"2024-04-22T01:18:52.341044+0000","last_clean":"2024-04-22T01:18:52.341044+0000","last_became_active":"2024-04-22T01:18:52.339855+0000","last_became_peered":"2024-04-22T01:18:52.339855+0000","last_unstale":"2024-04-22T01:18:52.341044+0000","last_undegraded":"2024-04-22T01:18:52.341044+0000","last_fullsized":"2024-04-22T01:18:52.341044+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:10.326477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.302507+0000","last_change":"2024-04-22T01:18:51.313239+0000","last_active":"2024-04-22T01:18:52.302507+0000","last_peered":"2024-04-22T01:18:52.302507+0000","last_clean":"2024-04-22T01:18:52.302507+0000","last_became_active":"2024-04-22T01:18:51.312748+0000","last_became_peered":"2024-04-22T01:18:51.312748+0000","last_unstale":"2024-04-22T01:18:52.302507+0000","last_undegraded":"2024-04-22T01:18:52.302507+0000","last_fullsized":"2024-04-22T01:18:52.302507+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:22:16.460663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318847+0000","last_change":"2024-04-22T01:18:52.330108+0000","last_active":"2024-04-22T01:18:53.318847+0000","last_peered":"2024-04-22T01:18:53.318847+0000","last_clean":"2024-04-22T01:18:53.318847+0000","last_became_active":"2024-04-22T01:18:52.329686+0000","last_became_peered":"2024-04-22T01:18:52.329686+0000","last_unstale":"2024-04-22T01:18:53.318847+0000","last_undegraded":"2024-04-22T01:18:53.318847+0000","last_fullsized":"2024-04-22T01:18:53.318847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:22.225277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"102'8","reported_seq":22,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321567+0000","last_change":"2024-04-22T01:18:51.295650+0000","last_active":"2024-04-22T01:18:53.321567+0000","last_peered":"2024-04-22T01:18:53.321567+0000","last_clean":"2024-04-22T01:18:53.321567+0000","last_became_active":"2024-04-22T01:18:51.294914+0000","last_became_peered":"2024-04-22T01:18:51.294914+0000","last_unstale":"2024-04-22T01:18:53.321567+0000","last_undegraded":"2024-04-22T01:18:53.321567+0000","last_fullsized":"2024-04-22T01:18:53.321567+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:01:37.686456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.311149+0000","last_change":"2024-04-22T01:18:52.310177+0000","last_active":"2024-04-22T01:18:52.311149+0000","last_peered":"2024-04-22T01:18:52.311149+0000","last_clean":"2024-04-22T01:18:52.311149+0000","last_became_active":"2024-04-22T01:18:52.309882+0000","last_became_peered":"2024-04-22T01:18:52.309882+0000","last_unstale":"2024-04-22T01:18:52.311149+0000","last_undegraded":"2024-04-22T01:18:52.311149+0000","last_fullsized":"2024-04-22T01:18:52.311149+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:26.880594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.349934+0000","last_change":"2024-04-22T01:18:51.312797+0000","last_active":"2024-04-22T01:18:52.349934+0000","last_peered":"2024-04-22T01:18:52.349934+0000","last_clean":"2024-04-22T01:18:52.349934+0000","last_became_active":"2024-04-22T01:18:51.311969+0000","last_became_peered":"2024-04-22T01:18:51.311969+0000","last_unstale":"2024-04-22T01:18:52.349934+0000","last_undegraded":"2024-04-22T01:18:52.349934+0000","last_fullsized":"2024-04-22T01:18:52.349934+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:04.332870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.361962+0000","last_change":"2024-04-22T01:18:52.356757+0000","last_active":"2024-04-22T01:18:52.361962+0000","last_peered":"2024-04-22T01:18:52.361962+0000","last_clean":"2024-04-22T01:18:52.361962+0000","last_became_active":"2024-04-22T01:18:52.348856+0000","last_became_peered":"2024-04-22T01:18:52.348856+0000","last_unstale":"2024-04-22T01:18:52.361962+0000","last_undegraded":"2024-04-22T01:18:52.361962+0000","last_fullsized":"2024-04-22T01:18:52.361962+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:26.568695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591804+0000","last_change":"2024-04-22T01:18:51.301777+0000","last_active":"2024-04-22T01:18:53.591804+0000","last_peered":"2024-04-22T01:18:53.591804+0000","last_clean":"2024-04-22T01:18:53.591804+0000","last_became_active":"2024-04-22T01:18:51.301148+0000","last_became_peered":"2024-04-22T01:18:51.301148+0000","last_unstale":"2024-04-22T01:18:53.591804+0000","last_undegraded":"2024-04-22T01:18:53.591804+0000","last_fullsized":"2024-04-22T01:18:53.591804+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:23:09.635098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337600+0000","last_change":"2024-04-22T01:18:52.335551+0000","last_active":"2024-04-22T01:18:52.337600+0000","last_peered":"2024-04-22T01:18:52.337600+0000","last_clean":"2024-04-22T01:18:52.337600+0000","last_became_active":"2024-04-22T01:18:52.335219+0000","last_became_peered":"2024-04-22T01:18:52.335219+0000","last_unstale":"2024-04-22T01:18:52.337600+0000","last_undegraded":"2024-04-22T01:18:52.337600+0000","last_fullsized":"2024-04-22T01:18:52.337600+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:31:52.637163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323015+0000","last_change":"2024-04-22T01:18:51.298240+0000","last_active":"2024-04-22T01:18:53.323015+0000","last_peered":"2024-04-22T01:18:53.323015+0000","last_clean":"2024-04-22T01:18:53.323015+0000","last_became_active":"2024-04-22T01:18:51.295827+0000","last_became_peered":"2024-04-22T01:18:51.295827+0000","last_unstale":"2024-04-22T01:18:53.323015+0000","last_undegraded":"2024-04-22T01:18:53.323015+0000","last_fullsized":"2024-04-22T01:18:53.323015+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:57.349352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318466+0000","last_change":"2024-04-22T01:18:52.311313+0000","last_active":"2024-04-22T01:18:53.318466+0000","last_peered":"2024-04-22T01:18:53.318466+0000","last_clean":"2024-04-22T01:18:53.318466+0000","last_became_active":"2024-04-22T01:18:52.311006+0000","last_became_peered":"2024-04-22T01:18:52.311006+0000","last_unstale":"2024-04-22T01:18:53.318466+0000","last_undegraded":"2024-04-22T01:18:53.318466+0000","last_fullsized":"2024-04-22T01:18:53.318466+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:24.725150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.314393+0000","last_change":"2024-04-22T01:18:51.298098+0000","last_active":"2024-04-22T01:18:52.314393+0000","last_peered":"2024-04-22T01:18:52.314393+0000","last_clean":"2024-04-22T01:18:52.314393+0000","last_became_active":"2024-04-22T01:18:51.297653+0000","last_became_peered":"2024-04-22T01:18:51.297653+0000","last_unstale":"2024-04-22T01:18:52.314393+0000","last_undegraded":"2024-04-22T01:18:52.314393+0000","last_fullsized":"2024-04-22T01:18:52.314393+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:48.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318714+0000","last_change":"2024-04-22T01:18:52.331792+0000","last_active":"2024-04-22T01:18:53.318714+0000","last_peered":"2024-04-22T01:18:53.318714+0000","last_clean":"2024-04-22T01:18:53.318714+0000","last_became_active":"2024-04-22T01:18:52.331169+0000","last_became_peered":"2024-04-22T01:18:52.331169+0000","last_unstale":"2024-04-22T01:18:53.318714+0000","last_undegraded":"2024-04-22T01:18:53.318714+0000","last_fullsized":"2024-04-22T01:18:53.318714+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:16.183115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.301745+0000","last_change":"2024-04-22T01:18:51.302622+0000","last_active":"2024-04-22T01:18:52.301745+0000","last_peered":"2024-04-22T01:18:52.301745+0000","last_clean":"2024-04-22T01:18:52.301745+0000","last_became_active":"2024-04-22T01:18:51.302208+0000","last_became_peered":"2024-04-22T01:18:51.302208+0000","last_unstale":"2024-04-22T01:18:52.301745+0000","last_undegraded":"2024-04-22T01:18:52.301745+0000","last_fullsized":"2024-04-22T01:18:52.301745+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:10.091431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.360276+0000","last_change":"2024-04-22T01:18:52.359536+0000","last_active":"2024-04-22T01:18:52.360276+0000","last_peered":"2024-04-22T01:18:52.360276+0000","last_clean":"2024-04-22T01:18:52.360276+0000","last_became_active":"2024-04-22T01:18:52.359209+0000","last_became_peered":"2024-04-22T01:18:52.359209+0000","last_unstale":"2024-04-22T01:18:52.360276+0000","last_undegraded":"2024-04-22T01:18:52.360276+0000","last_fullsized":"2024-04-22T01:18:52.360276+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:12.739140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.312035+0000","last_change":"2024-04-22T01:18:51.298037+0000","last_active":"2024-04-22T01:18:52.312035+0000","last_peered":"2024-04-22T01:18:52.312035+0000","last_clean":"2024-04-22T01:18:52.312035+0000","last_became_active":"2024-04-22T01:18:51.297351+0000","last_became_peered":"2024-04-22T01:18:51.297351+0000","last_unstale":"2024-04-22T01:18:52.312035+0000","last_undegraded":"2024-04-22T01:18:52.312035+0000","last_fullsized":"2024-04-22T01:18:52.312035+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:03:01.467024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337131+0000","last_change":"2024-04-22T01:18:52.335372+0000","last_active":"2024-04-22T01:18:52.337131+0000","last_peered":"2024-04-22T01:18:52.337131+0000","last_clean":"2024-04-22T01:18:52.337131+0000","last_became_active":"2024-04-22T01:18:52.334585+0000","last_became_peered":"2024-04-22T01:18:52.334585+0000","last_unstale":"2024-04-22T01:18:52.337131+0000","last_undegraded":"2024-04-22T01:18:52.337131+0000","last_fullsized":"2024-04-22T01:18:52.337131+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:22:14.804238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.301911+0000","last_change":"2024-04-22T01:18:51.303004+0000","last_active":"2024-04-22T01:18:52.301911+0000","last_peered":"2024-04-22T01:18:52.301911+0000","last_clean":"2024-04-22T01:18:52.301911+0000","last_became_active":"2024-04-22T01:18:51.302647+0000","last_became_peered":"2024-04-22T01:18:51.302647+0000","last_unstale":"2024-04-22T01:18:52.301911+0000","last_undegraded":"2024-04-22T01:18:52.301911+0000","last_fullsized":"2024-04-22T01:18:52.301911+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:38.124357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318234+0000","last_change":"2024-04-22T01:18:52.311250+0000","last_active":"2024-04-22T01:18:53.318234+0000","last_peered":"2024-04-22T01:18:53.318234+0000","last_clean":"2024-04-22T01:18:53.318234+0000","last_became_active":"2024-04-22T01:18:52.310892+0000","last_became_peered":"2024-04-22T01:18:52.310892+0000","last_unstale":"2024-04-22T01:18:53.318234+0000","last_undegraded":"2024-04-22T01:18:53.318234+0000","last_fullsized":"2024-04-22T01:18:53.318234+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:00:25.064406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337837+0000","last_change":"2024-04-22T01:18:51.299160+0000","last_active":"2024-04-22T01:18:52.337837+0000","last_peered":"2024-04-22T01:18:52.337837+0000","last_clean":"2024-04-22T01:18:52.337837+0000","last_became_active":"2024-04-22T01:18:51.298775+0000","last_became_peered":"2024-04-22T01:18:51.298775+0000","last_unstale":"2024-04-22T01:18:52.337837+0000","last_undegraded":"2024-04-22T01:18:52.337837+0000","last_fullsized":"2024-04-22T01:18:52.337837+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:14:00.219961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.320617+0000","last_change":"2024-04-22T01:18:52.314406+0000","last_active":"2024-04-22T01:18:53.320617+0000","last_peered":"2024-04-22T01:18:53.320617+0000","last_clean":"2024-04-22T01:18:53.320617+0000","last_became_active":"2024-04-22T01:18:52.313689+0000","last_became_peered":"2024-04-22T01:18:52.313689+0000","last_unstale":"2024-04-22T01:18:53.320617+0000","last_undegraded":"2024-04-22T01:18:53.320617+0000","last_fullsized":"2024-04-22T01:18:53.320617+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:02.573459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.318692+0000","last_change":"2024-04-22T01:18:51.284358+0000","last_active":"2024-04-22T01:18:53.318692+0000","last_peered":"2024-04-22T01:18:53.318692+0000","last_clean":"2024-04-22T01:18:53.318692+0000","last_became_active":"2024-04-22T01:18:51.281948+0000","last_became_peered":"2024-04-22T01:18:51.281948+0000","last_unstale":"2024-04-22T01:18:53.318692+0000","last_undegraded":"2024-04-22T01:18:53.318692+0000","last_fullsized":"2024-04-22T01:18:53.318692+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:46:22.310137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.358243+0000","last_change":"2024-04-22T01:18:52.357795+0000","last_active":"2024-04-22T01:18:52.358243+0000","last_peered":"2024-04-22T01:18:52.358243+0000","last_clean":"2024-04-22T01:18:52.358243+0000","last_became_active":"2024-04-22T01:18:52.357376+0000","last_became_peered":"2024-04-22T01:18:52.357376+0000","last_unstale":"2024-04-22T01:18:52.358243+0000","last_undegraded":"2024-04-22T01:18:52.358243+0000","last_fullsized":"2024-04-22T01:18:52.358243+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.325886+0000","last_change":"2024-04-22T01:18:51.306618+0000","last_active":"2024-04-22T01:18:52.325886+0000","last_peered":"2024-04-22T01:18:52.325886+0000","last_clean":"2024-04-22T01:18:52.325886+0000","last_became_active":"2024-04-22T01:18:51.305474+0000","last_became_peered":"2024-04-22T01:18:51.305474+0000","last_unstale":"2024-04-22T01:18:52.325886+0000","last_undegraded":"2024-04-22T01:18:52.325886+0000","last_fullsized":"2024-04-22T01:18:52.325886+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:09:14.958959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.323009+0000","last_change":"2024-04-22T01:18:52.332479+0000","last_active":"2024-04-22T01:18:53.323009+0000","last_peered":"2024-04-22T01:18:53.323009+0000","last_clean":"2024-04-22T01:18:53.323009+0000","last_became_active":"2024-04-22T01:18:52.329098+0000","last_became_peered":"2024-04-22T01:18:52.329098+0000","last_unstale":"2024-04-22T01:18:53.323009+0000","last_undegraded":"2024-04-22T01:18:53.323009+0000","last_fullsized":"2024-04-22T01:18:53.323009+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:08.054090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.326547+0000","last_change":"2024-04-22T01:18:51.295451+0000","last_active":"2024-04-22T01:18:52.326547+0000","last_peered":"2024-04-22T01:18:52.326547+0000","last_clean":"2024-04-22T01:18:52.326547+0000","last_became_active":"2024-04-22T01:18:51.295068+0000","last_became_peered":"2024-04-22T01:18:51.295068+0000","last_unstale":"2024-04-22T01:18:52.326547+0000","last_undegraded":"2024-04-22T01:18:52.326547+0000","last_fullsized":"2024-04-22T01:18:52.326547+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:18:01.570250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337532+0000","last_change":"2024-04-22T01:18:52.336315+0000","last_active":"2024-04-22T01:18:52.337532+0000","last_peered":"2024-04-22T01:18:52.337532+0000","last_clean":"2024-04-22T01:18:52.337532+0000","last_became_active":"2024-04-22T01:18:52.335962+0000","last_became_peered":"2024-04-22T01:18:52.335962+0000","last_unstale":"2024-04-22T01:18:52.337532+0000","last_undegraded":"2024-04-22T01:18:52.337532+0000","last_fullsized":"2024-04-22T01:18:52.337532+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:02.384790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.321139+0000","last_change":"2024-04-22T01:18:51.316135+0000","last_active":"2024-04-22T01:18:53.321139+0000","last_peered":"2024-04-22T01:18:53.321139+0000","last_clean":"2024-04-22T01:18:53.321139+0000","last_became_active":"2024-04-22T01:18:51.315520+0000","last_became_peered":"2024-04-22T01:18:51.315520+0000","last_unstale":"2024-04-22T01:18:53.321139+0000","last_undegraded":"2024-04-22T01:18:53.321139+0000","last_fullsized":"2024-04-22T01:18:53.321139+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:59.636525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.337140+0000","last_change":"2024-04-22T01:18:52.335124+0000","last_active":"2024-04-22T01:18:52.337140+0000","last_peered":"2024-04-22T01:18:52.337140+0000","last_clean":"2024-04-22T01:18:52.337140+0000","last_became_active":"2024-04-22T01:18:52.334828+0000","last_became_peered":"2024-04-22T01:18:52.334828+0000","last_unstale":"2024-04-22T01:18:52.337140+0000","last_undegraded":"2024-04-22T01:18:52.337140+0000","last_fullsized":"2024-04-22T01:18:52.337140+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:28:35.664356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.302689+0000","last_change":"2024-04-22T01:18:51.312864+0000","last_active":"2024-04-22T01:18:52.302689+0000","last_peered":"2024-04-22T01:18:52.302689+0000","last_clean":"2024-04-22T01:18:52.302689+0000","last_became_active":"2024-04-22T01:18:51.312664+0000","last_became_peered":"2024-04-22T01:18:51.312664+0000","last_unstale":"2024-04-22T01:18:52.302689+0000","last_undegraded":"2024-04-22T01:18:52.302689+0000","last_fullsized":"2024-04-22T01:18:52.302689+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:01.499168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.592090+0000","last_change":"2024-04-22T01:18:52.355090+0000","last_active":"2024-04-22T01:18:53.592090+0000","last_peered":"2024-04-22T01:18:53.592090+0000","last_clean":"2024-04-22T01:18:53.592090+0000","last_became_active":"2024-04-22T01:18:52.353987+0000","last_became_peered":"2024-04-22T01:18:52.353987+0000","last_unstale":"2024-04-22T01:18:53.592090+0000","last_undegraded":"2024-04-22T01:18:53.592090+0000","last_fullsized":"2024-04-22T01:18:53.592090+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:06.681876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.312435+0000","last_change":"2024-04-22T01:18:51.305955+0000","last_active":"2024-04-22T01:18:52.312435+0000","last_peered":"2024-04-22T01:18:52.312435+0000","last_clean":"2024-04-22T01:18:52.312435+0000","last_became_active":"2024-04-22T01:18:51.305106+0000","last_became_peered":"2024-04-22T01:18:51.305106+0000","last_unstale":"2024-04-22T01:18:52.312435+0000","last_undegraded":"2024-04-22T01:18:52.312435+0000","last_fullsized":"2024-04-22T01:18:52.312435+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:13.306518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.324115+0000","last_change":"2024-04-22T01:18:52.341025+0000","last_active":"2024-04-22T01:18:53.324115+0000","last_peered":"2024-04-22T01:18:53.324115+0000","last_clean":"2024-04-22T01:18:53.324115+0000","last_became_active":"2024-04-22T01:18:52.340112+0000","last_became_peered":"2024-04-22T01:18:52.340112+0000","last_unstale":"2024-04-22T01:18:53.324115+0000","last_undegraded":"2024-04-22T01:18:53.324115+0000","last_fullsized":"2024-04-22T01:18:53.324115+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:22.805839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.304486+0000","last_change":"2024-04-22T01:18:51.311664+0000","last_active":"2024-04-22T01:18:52.304486+0000","last_peered":"2024-04-22T01:18:52.304486+0000","last_clean":"2024-04-22T01:18:52.304486+0000","last_became_active":"2024-04-22T01:18:51.311362+0000","last_became_peered":"2024-04-22T01:18:51.311362+0000","last_unstale":"2024-04-22T01:18:52.304486+0000","last_undegraded":"2024-04-22T01:18:52.304486+0000","last_fullsized":"2024-04-22T01:18:52.304486+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:52.087842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.322101+0000","last_change":"2024-04-22T01:18:52.330939+0000","last_active":"2024-04-22T01:18:53.322101+0000","last_peered":"2024-04-22T01:18:53.322101+0000","last_clean":"2024-04-22T01:18:53.322101+0000","last_became_active":"2024-04-22T01:18:52.327407+0000","last_became_peered":"2024-04-22T01:18:52.327407+0000","last_unstale":"2024-04-22T01:18:53.322101+0000","last_undegraded":"2024-04-22T01:18:53.322101+0000","last_fullsized":"2024-04-22T01:18:53.322101+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:51.260532+0000","last_clean_scrub_stamp":"2024-04-22T01:18:51.260532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:15.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-22T01:18:52.322839+0000","last_change":"2024-04-22T01:18:51.297303+0000","last_active":"2024-04-22T01:18:52.322839+0000","last_peered":"2024-04-22T01:18:52.322839+0000","last_clean":"2024-04-22T01:18:52.322839+0000","last_became_active":"2024-04-22T01:18:51.291301+0000","last_became_peered":"2024-04-22T01:18:51.291301+0000","last_unstale":"2024-04-22T01:18:52.322839+0000","last_undegraded":"2024-04-22T01:18:52.322839+0000","last_fullsized":"2024-04-22T01:18:52.322839+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:18:50.257322+0000","last_clean_scrub_stamp":"2024-04-22T01:18:50.257322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:18.938085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":318,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-22T01:18:53.591219+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:18:53.591219+0000","last_peered":"2024-04-22T01:18:53.591219+0000","last_clean":"2024-04-22T01:18:53.591219+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:18:53.591219+0000","last_undegraded":"2024-04-22T01:18:53.591219+0000","last_fullsized":"2024-04-22T01:18:53.591219+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1867776,"data_stored":1173705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":585393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1265664,"data_stored":583177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32428,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339412,"statfs":{"total":96636764160,"available":96603557888,"internally_reserved":0,"allocated":1273856,"data_stored":583269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1269760,"data_stored":583759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1269760,"data_stored":583211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36588,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335252,"statfs":{"total":96636764160,"available":96599298048,"internally_reserved":0,"allocated":1269760,"data_stored":583211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1888256,"data_stored":1181939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:18:55.676 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:18:55.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-22T01:18:55.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 Processor -- start 2024-04-22T01:18:55.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 -- start start 2024-04-22T01:18:55.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e1e0 0x7fb8b80d3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8b80b30e0 0x7fb8b80b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b806d740 con 0x7fb8b80b5da0 2024-04-22T01:18:55.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b806d2b0 con 0x7fb8b80b30e0 2024-04-22T01:18:55.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8bdc78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b806c980 con 0x7fb8b815e1e0 2024-04-22T01:18:55.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8b6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8b80b30e0 0x7fb8b80b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8b77fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e1e0 0x7fb8b80d3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8b6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8b80b30e0 0x7fb8b80b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.70:60128/0 (socket says 172.21.15.70:60128) 2024-04-22T01:18:55.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b6ffd640 1 -- 172.21.15.70:0/3095263289 learned_addr learned my addr 172.21.15.70:0/3095263289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:55.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.851+0000 7fb8b77fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e1e0 0x7fb8b80d3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:58502/0 (socket says 172.21.15.70:58502) 2024-04-22T01:18:55.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b6ffd640 1 -- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e1e0 msgr2=0x7fb8b80d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:55.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b6ffd640 1 --2- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e1e0 0x7fb8b80d3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b6ffd640 1 -- 172.21.15.70:0/3095263289 >> v1:172.21.15.98:6789/0 conn(0x7fb8b80b5da0 legacy=0x7fb8b80d2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b6ffd640 1 -- 172.21.15.70:0/3095263289 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b806cc60 con 0x7fb8b80b30e0 2024-04-22T01:18:55.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b6ffd640 1 --2- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80b5860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009870 tx=0x7fb8ac01c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faf2593c0dbb52e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b67fc640 1 -- 172.21.15.70:0/3095263289 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0b2070 con 0x7fb8b80b30e0 2024-04-22T01:18:55.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b67fc640 1 -- 172.21.15.70:0/3095263289 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ac0a9070 con 0x7fb8b80b30e0 2024-04-22T01:18:55.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8b67fc640 1 -- 172.21.15.70:0/3095263289 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0ad880 con 0x7fb8b80b30e0 2024-04-22T01:18:55.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 -- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80b30e0 msgr2=0x7fb8b80b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80b5860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009870 tx=0x7fb8ac01c560 comp rx=0 tx=0).stop 2024-04-22T01:18:55.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 -- 172.21.15.70:0/3095263289 shutdown_connections 2024-04-22T01:18:55.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80b5860 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/3095263289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b815e1e0 0x7fb8b80d3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 -- 172.21.15.70:0/3095263289 >> 172.21.15.70:0/3095263289 conn(0x7fb8b8073110 msgr2=0x7fb8b80d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.855+0000 7fb8bdc78640 1 -- 172.21.15.70:0/3095263289 shutdown_connections 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- 172.21.15.70:0/3095263289 wait complete. 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 Processor -- start 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- start start 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b80b5da0 0x7fb8b8154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b815e1e0 0x7fb8b8141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b816af30 con 0x7fb8b80b30e0 2024-04-22T01:18:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b816adb0 con 0x7fb8b815e1e0 2024-04-22T01:18:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b816b0b0 con 0x7fb8b80b5da0 2024-04-22T01:18:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60424/0 (socket says 172.21.15.70:60424) 2024-04-22T01:18:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 -- 172.21.15.70:0/396916690 learned_addr learned my addr 172.21.15.70:0/396916690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b7fff640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b815e1e0 0x7fb8b8141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b6ffd640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b80b5da0 0x7fb8b8154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 -- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b80b5da0 msgr2=0x7fb8b8154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b80b5da0 0x7fb8b8154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 -- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b815e1e0 msgr2=0x7fb8b8141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b815e1e0 0x7fb8b8141fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 -- 172.21.15.70:0/396916690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b8154b00 con 0x7fb8b80b30e0 2024-04-22T01:18:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b6ffd640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b80b5da0 0x7fb8b8154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8b77fe640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80cb110 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800e930 tx=0x7fb8a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8017070 con 0x7fb8b80b30e0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b8154de0 con 0x7fb8b80b30e0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a800ca00 con 0x7fb8b80b30e0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a800cdb0 con 0x7fb8b80b30e0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.859+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b815b3a0 con 0x7fb8b80b30e0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.863+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b8068210 con 0x7fb8b80b30e0 2024-04-22T01:18:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.863+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb8a801a070 con 0x7fb8b80b30e0 2024-04-22T01:18:55.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.863+0000 7fb89ffff640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:55.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.863+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb8a8086220 con 0x7fb8b80b30e0 2024-04-22T01:18:55.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.863+0000 7fb8b6ffd640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:55.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.863+0000 7fb8b6ffd640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0af000 tx=0x7fb8ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:55.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:55.867+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb8a8056ca0 con 0x7fb8b80b30e0 2024-04-22T01:18:56.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.087+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb8b80d2330 con 0x7fb8b80b30e0 2024-04-22T01:18:56.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb89ffff640 1 -- 172.21.15.70:0/396916690 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) ==== 190+0+147 (secure 0 0 0) 0x7fb8a80565a0 con 0x7fb8b80b30e0 2024-04-22T01:18:56.090 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-22T01:18:56.090 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDIuSVmPAopNxAAbRe4o1u7UtrDSV0Xzy85vw== 2024-04-22T01:18:56.091 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-22T01:18:56.091 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-22T01:18:56.091 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-22T01:18:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-22T01:18:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 msgr2=0x7fb8a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0af000 tx=0x7fb8ac008040 comp rx=0 tx=0).stop 2024-04-22T01:18:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 msgr2=0x7fb8b80cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80cb110 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800e930 tx=0x7fb8a800edf0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 shutdown_connections 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8a40678c0 0x7fb8a4069d80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8b815e1e0 0x7fb8b8141fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b80b5da0 0x7fb8b8154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 --2- 172.21.15.70:0/396916690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b30e0 0x7fb8b80cb110 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 >> 172.21.15.70:0/396916690 conn(0x7fb8b8073110 msgr2=0x7fb8b80ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 shutdown_connections 2024-04-22T01:18:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.091+0000 7fb8bdc78640 1 -- 172.21.15.70:0/396916690 wait complete. 2024-04-22T01:18:56.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:18:56.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 Processor -- start 2024-04-22T01:18:56.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 -- start start 2024-04-22T01:18:56.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7b406d790 0x7fd7b409a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b40d03a0 con 0x7fd7b40cfe50 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b40d0fb0 con 0x7fd7b406d790 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b8af9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b40d0680 con 0x7fd7b415dd90 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7abfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7b406d790 0x7fd7b409a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b2d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd7b40cfe50 0x7fd7b4161b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43264/0 (socket says 172.21.15.70:43264) 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7abfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7b406d790 0x7fd7b409a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45674/0 (socket says 172.21.15.70:45674) 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b2d9d640 1 -- 172.21.15.70:0/1501762826 learned_addr learned my addr 172.21.15.70:0/1501762826 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b259c640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7abfff640 1 -- 172.21.15.70:0/1501762826 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 msgr2=0x7fd7b4099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7abfff640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4099cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7abfff640 1 -- 172.21.15.70:0/1501762826 >> v1:172.21.15.70:6789/0 conn(0x7fd7b40cfe50 legacy=0x7fd7b4161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7abfff640 1 -- 172.21.15.70:0/1501762826 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b40d0960 con 0x7fd7b406d790 2024-04-22T01:18:56.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b1d9b640 1 -- 172.21.15.70:0/1501762826 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287960993 0 0) 0x7fd7b40d03a0 con 0x7fd7b40cfe50 2024-04-22T01:18:56.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.231+0000 7fd7b259c640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4099cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:56.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7abfff640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b406d790 0x7fd7b409a200 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097f0 tx=0x7fd7a401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab77f6d5f41a446 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:56.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b1d9b640 1 -- 172.21.15.70:0/1501762826 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a40aa5f0 con 0x7fd7b406d790 2024-04-22T01:18:56.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b1d9b640 1 -- 172.21.15.70:0/1501762826 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a40af070 con 0x7fd7b406d790 2024-04-22T01:18:56.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b1d9b640 1 -- 172.21.15.70:0/1501762826 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a40aa9b0 con 0x7fd7b406d790 2024-04-22T01:18:56.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1501762826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b406d790 msgr2=0x7fd7b409a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b406d790 0x7fd7b409a200 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097f0 tx=0x7fd7a401c470 comp rx=0 tx=0).stop 2024-04-22T01:18:56.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1501762826 shutdown_connections 2024-04-22T01:18:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b406d790 0x7fd7b409a200 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1501762826 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4099cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1501762826 >> 172.21.15.70:0/1501762826 conn(0x7fd7b40b35d0 msgr2=0x7fd7b40b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1501762826 shutdown_connections 2024-04-22T01:18:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1501762826 wait complete. 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 Processor -- start 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- start start 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b406d790 0x7fd7b415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 0x7fd7b4158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b416b020 con 0x7fd7b40cfe50 2024-04-22T01:18:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b416aea0 con 0x7fd7b406d790 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b8af9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b416b1a0 con 0x7fd7b415dd90 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 0x7fd7b4158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 0x7fd7b4158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45678/0 (socket says 172.21.15.70:45678) 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7abfff640 1 -- 172.21.15.70:0/1564742061 learned_addr learned my addr 172.21.15.70:0/1564742061 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b2d9d640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.235+0000 7fd7b259c640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b406d790 0x7fd7b415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7abfff640 1 -- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 msgr2=0x7fd7b4158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7abfff640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7abfff640 1 -- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b406d790 msgr2=0x7fd7b415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7abfff640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b406d790 0x7fd7b415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7abfff640 1 -- 172.21.15.70:0/1564742061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b40b4fb0 con 0x7fd7b40cfe50 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7abfff640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 0x7fd7b4158a50 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097f0 tx=0x7fd7a401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4004220 con 0x7fd7b40cfe50 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a40043c0 con 0x7fd7b40cfe50 2024-04-22T01:18:56.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b40b5290 con 0x7fd7b40cfe50 2024-04-22T01:18:56.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4003820 con 0x7fd7b40cfe50 2024-04-22T01:18:56.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b41595f0 con 0x7fd7b40cfe50 2024-04-22T01:18:56.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b40b5950 con 0x7fd7b40cfe50 2024-04-22T01:18:56.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd7a40039c0 con 0x7fd7b40cfe50 2024-04-22T01:18:56.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7aaffd640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7980678c0 0x7fd798069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fd7a4120490 con 0x7fd7b40cfe50 2024-04-22T01:18:56.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7b259c640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7980678c0 0x7fd798069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.239+0000 7fd7b259c640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7980678c0 0x7fd798069d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd7b40ca840 tx=0x7fd79c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:56.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.243+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd7b40b5950 con 0x7fd7b40cfe50 2024-04-22T01:18:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.439+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7b4068920 con 0x7fd7b40cfe50 2024-04-22T01:18:56.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7aaffd640 1 -- 172.21.15.70:0/1564742061 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7fd7a4016090 con 0x7fd7b40cfe50 2024-04-22T01:18:56.442 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:18:56.442 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:51.626962+0000","modified":"2024-04-22T01:18:53.648845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7629":{"gid":7629,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3578750329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3578750329},{"type":"v1","addr":"172.21.15.70:6837","nonce":3578750329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7629,"qdb_cluster":[7629]},"id":7}]} 2024-04-22T01:18:56.442 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 50 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7980678c0 msgr2=0x7fd798069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7980678c0 0x7fd798069d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd7b40ca840 tx=0x7fd79c008040 comp rx=0 tx=0).stop 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 msgr2=0x7fd7b4158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 0x7fd7b4158a50 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097f0 tx=0x7fd7a401acb0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 shutdown_connections 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd7980678c0 0x7fd798069d80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b415dd90 0x7fd7b4158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7b40cfe50 0x7fd7b4158a50 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 --2- 172.21.15.70:0/1564742061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b406d790 0x7fd7b415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 >> 172.21.15.70:0/1564742061 conn(0x7fd7b40b35d0 msgr2=0x7fd7b41607f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 shutdown_connections 2024-04-22T01:18:56.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.443+0000 7fd7b8af9640 1 -- 172.21.15.70:0/1564742061 wait complete. 2024-04-22T01:18:56.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:18:51.626962+0000', 'modified': '2024-04-22T01:18:53.648845+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7629}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7629': {'gid': 7629, 'name': 'a', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/3578750329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3578750329}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3578750329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7629, 'qdb_cluster': [7629]} 2024-04-22T01:18:56.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:18:56.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-22T01:18:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 Processor -- start 2024-04-22T01:18:56.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 -- start start 2024-04-22T01:18:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8fe00b02e0 0x7f8fe00675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe00ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe015e4a0 con 0x7f8fe015e0c0 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe006c9b0 con 0x7f8fe00b02e0 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe636d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe006bda0 con 0x7f8fe0099a60 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8fe00b02e0 0x7f8fe00675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8fe00b02e0 0x7f8fe00675b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45682/0 (socket says 172.21.15.70:45682) 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 -- 172.21.15.70:0/2433434037 learned_addr learned my addr 172.21.15.70:0/2433434037 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fe490a640 1 --2- 172.21.15.70:0/2433434037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe00ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 -- 172.21.15.70:0/2433434037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 msgr2=0x7f8fe00ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/2433434037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe00ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 -- 172.21.15.70:0/2433434037 >> v1:172.21.15.98:6789/0 conn(0x7f8fe015e0c0 legacy=0x7f8fe00afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:56.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:56.999+0000 7f8fd7fff640 1 -- 172.21.15.70:0/2433434037 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe006c080 con 0x7f8fe00b02e0 2024-04-22T01:18:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/2433434037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe00b02e0 0x7f8fe00675b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474f6a89407a3658 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fdf7fe640 1 -- 172.21.15.70:0/2433434037 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80b0070 con 0x7f8fe00b02e0 2024-04-22T01:18:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fdf7fe640 1 -- 172.21.15.70:0/2433434037 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc80ab600 con 0x7f8fe00b02e0 2024-04-22T01:18:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fdf7fe640 1 -- 172.21.15.70:0/2433434037 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80ab9d0 con 0x7f8fe00b02e0 2024-04-22T01:18:57.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- 172.21.15.70:0/2433434037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe00b02e0 msgr2=0x7f8fe00675b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 --2- 172.21.15.70:0/2433434037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe00b02e0 0x7f8fe00675b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc801c4d0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- 172.21.15.70:0/2433434037 shutdown_connections 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 --2- 172.21.15.70:0/2433434037 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe00ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 --2- 172.21.15.70:0/2433434037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe00b02e0 0x7f8fe00675b0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- 172.21.15.70:0/2433434037 >> 172.21.15.70:0/2433434037 conn(0x7f8fe00b3c80 msgr2=0x7f8fe00b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- 172.21.15.70:0/2433434037 shutdown_connections 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- 172.21.15.70:0/2433434037 wait complete. 2024-04-22T01:18:57.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 Processor -- start 2024-04-22T01:18:57.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- start start 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe0152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe00b02e0 0x7f8fe015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 0x7f8fe015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe00ae2f0 con 0x7f8fe015e0c0 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe00ae170 con 0x7f8fe00b02e0 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe636d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe00ae470 con 0x7f8fe0099a60 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 0x7f8fe015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 0x7f8fe015b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45684/0 (socket says 172.21.15.70:45684) 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe00b02e0 0x7f8fe015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 -- 172.21.15.70:0/4271926735 learned_addr learned my addr 172.21.15.70:0/4271926735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fdffff640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe0152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 -- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 msgr2=0x7f8fe0152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe0152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 -- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe00b02e0 msgr2=0x7f8fe015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe00b02e0 0x7f8fe015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.003+0000 7f8fe490a640 1 -- 172.21.15.70:0/4271926735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe015f9f0 con 0x7f8fe015e0c0 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fe490a640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 0x7f8fe015b6a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8007e30 tx=0x7f8fd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdd7fa640 1 -- 172.21.15.70:0/4271926735 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8014620 con 0x7f8fe015e0c0 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdd7fa640 1 -- 172.21.15.70:0/4271926735 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd8019070 con 0x7f8fe015e0c0 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe015fcd0 con 0x7f8fe015e0c0 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdd7fa640 1 -- 172.21.15.70:0/4271926735 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8014a00 con 0x7f8fe015e0c0 2024-04-22T01:18:57.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe0160290 con 0x7f8fe015e0c0 2024-04-22T01:18:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdd7fa640 1 -- 172.21.15.70:0/4271926735 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8fd8014ba0 con 0x7f8fe015e0c0 2024-04-22T01:18:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdd7fa640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8fc00678c0 0x7f8fc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdffff640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8fc00678c0 0x7f8fc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdd7fa640 1 -- 172.21.15.70:0/4271926735 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8fd8086740 con 0x7f8fe015e0c0 2024-04-22T01:18:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fdffff640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8fc00678c0 0x7f8fc0069d80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0001770 tx=0x7f8fd0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fe94e6000 1 Processor -- start 2024-04-22T01:18:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.007+0000 7f8fe94e6000 1 -- start start 2024-04-22T01:18:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e309ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636e2eed5e0 con 0x5636e30e8ef0 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5636e2e74c50 con 0x5636e30e9810 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5636e2ee95a0 con 0x5636e309fbc0 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30e92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45698/0 (socket says 172.21.15.70:45698) 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 -- 172.21.15.70:0/1183760268 learned_addr learned my addr 172.21.15.70:0/1183760268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe490a640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e309ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 -- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 msgr2=0x5636e309ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e309ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 -- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 msgr2=0x5636e309f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 -- 172.21.15.70:0/1183760268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e2e72d10 con 0x5636e30e8ef0 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe490a640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e309ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30e92d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000fae0 tx=0x7f8fd0000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd64687a60a16b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd5ffb640 1 -- 172.21.15.70:0/1183760268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0025070 con 0x5636e30e8ef0 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd5ffb640 1 -- 172.21.15.70:0/1183760268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd0015bf0 con 0x5636e30e8ef0 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd5ffb640 1 -- 172.21.15.70:0/1183760268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0020400 con 0x5636e30e8ef0 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- 172.21.15.70:0/1183760268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 msgr2=0x5636e30e92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30e92d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000fae0 tx=0x7f8fd0000ee0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- 172.21.15.70:0/1183760268 shutdown_connections 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e309ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/1183760268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30e92d0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- 172.21.15.70:0/1183760268 >> 172.21.15.70:0/1183760268 conn(0x5636e309bb70 msgr2=0x5636e309df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- 172.21.15.70:0/1183760268 shutdown_connections 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- 172.21.15.70:0/1183760268 wait complete. 2024-04-22T01:18:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- start start 2024-04-22T01:18:57.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 Processor -- start 2024-04-22T01:18:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e30cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636e2ea7520 con 0x5636e30e8ef0 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5636e2e93810 con 0x5636e30e9810 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5636e309b9f0 con 0x5636e309fbc0 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 10 client.0 _ll_register_callbacks cb 0x5636e308a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45706/0 (socket says 172.21.15.70:45706) 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 -- 172.21.15.70:0/938456615 learned_addr learned my addr 172.21.15.70:0/938456615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe490a640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e30cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 -- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 msgr2=0x5636e30cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e30cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 -- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 msgr2=0x5636e30ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 -- 172.21.15.70:0/938456615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e2e857b0 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fdffff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e30cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30ce460 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb4c7f7ce971b61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fbeffd640 1 -- 172.21.15.70:0/938456615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80bb070 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fbeffd640 1 -- 172.21.15.70:0/938456615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8fc80abc80 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.011+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd0015bf0 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 1 -- 172.21.15.70:0/938456615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f8fc80c6050 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 10 client.7773 fetch_fsmap learned FSMap version 50 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x5636e2e74ea0 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 1 -- 172.21.15.70:0/938456615 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd0015bf0 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 1 -- 172.21.15.70:0/938456615 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f8fc80c9020 con 0x5636e30e8ef0 2024-04-22T01:18:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 10 client.7773 fetch_fsmap finished waiting for FSMap version 50 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 10 client.7773 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 20 client.7773 populate_metadata read hostname 'smithi070' 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30f8f90 0x5636e30fb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 4 client.7773 mds_command: new command op to 7629 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 --> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636e2e88ab0 con 0x5636e30f8f90 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fdffff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30f8f90 0x5636e30fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 10 client.7773 ms_handle_connect on v2:172.21.15.70:6836/3578750329 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fdffff640 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30f8f90 0x5636e30fb380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a906c9a3553e811 server_cookie=52ace0eedd14ad7d in_seq=0 out_seq=0 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 1 -- 172.21.15.70:0/938456615 <== mds.0 v2:172.21.15.70:6836/3578750329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636e2e88ab0 con 0x5636e30f8f90 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 10 client.7773 handle_command_reply: tid=0 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 client.7773 shutdown 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 msgr2=0x5636e30ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30ce460 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc801ad10 comp rx=0 tx=0).stop 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 shutdown_connections 2024-04-22T01:18:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30f8f90 0x5636e30fb380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fbeffd640 0 client.7773 ms_handle_reset on v2:172.21.15.70:6836/3578750329 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e30ce460 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/938456615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e30cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 >> 172.21.15.70:0/938456615 conn(0x5636e309bb70 msgr2=0x5636e309e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 shutdown_connections 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 1 -- 172.21.15.70:0/938456615 wait complete. 2024-04-22T01:18:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.015+0000 7f8fe94e6000 20 client.7773 trim_cache size 0 max 16384 2024-04-22T01:18:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.031+0000 7f8fe94e6000 1 Processor -- start 2024-04-22T01:18:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.031+0000 7f8fe94e6000 1 -- start start 2024-04-22T01:18:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 0x5636e309df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e30e8ef0 0x5636e309e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe02038f0 con 0x5636e309fbc0 2024-04-22T01:18:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0202940 con 0x5636e30e9810 2024-04-22T01:18:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe00ca840 con 0x5636e30e8ef0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 0x5636e309df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 0x5636e309df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45712/0 (socket says 172.21.15.70:45712) 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 -- 172.21.15.70:0/2446394508 learned_addr learned my addr 172.21.15.70:0/2446394508 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe490a640 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 -- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e30e8ef0 msgr2=0x5636e309e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e30e8ef0 0x5636e309e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e30e8ef0 0x5636e309e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 -- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 msgr2=0x5636e309eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 -- 172.21.15.70:0/2446394508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e2e75e80 con 0x5636e309fbc0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe490a640 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 0x5636e309df60 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0023040 tx=0x7f8fd000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe526be8310f01b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd57fa640 1 -- 172.21.15.70:0/2446394508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0036070 con 0x5636e309fbc0 2024-04-22T01:18:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd57fa640 1 -- 172.21.15.70:0/2446394508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd0030e50 con 0x5636e309fbc0 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd57fa640 1 -- 172.21.15.70:0/2446394508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0030030 con 0x5636e309fbc0 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- 172.21.15.70:0/2446394508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 msgr2=0x5636e309df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 0x5636e309df60 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0023040 tx=0x7f8fd000a000 comp rx=0 tx=0).stop 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- 172.21.15.70:0/2446394508 shutdown_connections 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e309eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e30e8ef0 0x5636e309e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/2446394508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e309fbc0 0x5636e309df60 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- 172.21.15.70:0/2446394508 >> 172.21.15.70:0/2446394508 conn(0x5636e309bb70 msgr2=0x5636e30f64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- 172.21.15.70:0/2446394508 shutdown_connections 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- 172.21.15.70:0/2446394508 wait complete. 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- start start 2024-04-22T01:18:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 Processor -- start 2024-04-22T01:18:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e3158d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e3159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30fb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636e2f9fe80 con 0x5636e30e8ef0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0202370 con 0x5636e30e9810 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x5636e2ea1c40 con 0x5636e309fbc0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe94e6000 10 client.0 _ll_register_callbacks cb 0x7f8fc80096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e3159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e3159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45726/0 (socket says 172.21.15.70:45726) 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 -- 172.21.15.70:0/4031785823 learned_addr learned my addr 172.21.15.70:0/4031785823 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fdffff640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e3158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fe490a640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30fb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 -- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 msgr2=0x5636e3158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e3158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 -- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 msgr2=0x5636e30fb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30fb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.035+0000 7f8fd7fff640 1 -- 172.21.15.70:0/4031785823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e2eba760 con 0x5636e30e8ef0 2024-04-22T01:18:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fd7fff640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e3159260 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8fc801cd70 tx=0x7f8fc80a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbafa1f520a945c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 1 -- 172.21.15.70:0/4031785823 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80bb040 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 1 -- 172.21.15.70:0/4031785823 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8fc8004450 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd0030e50 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 1 -- 172.21.15.70:0/4031785823 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f8fc80b0040 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 10 client.7779 fetch_fsmap learned FSMap version 50 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x5636e2f469e0 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 1 -- 172.21.15.70:0/4031785823 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd0030e50 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 1 -- 172.21.15.70:0/4031785823 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f8fc80c9070 con 0x5636e30e8ef0 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 10 client.7779 fetch_fsmap finished waiting for FSMap version 50 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 10 client.7779 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 20 client.7779 populate_metadata read hostname 'smithi070' 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30fb810 0x5636e30fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 4 client.7779 mds_command: new command op to 7629 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 --> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] -- command(tid 0: {"prefix": "status"}) -- 0x5636e2efab20 con 0x5636e30fb810 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fdffff640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30fb810 0x5636e30fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fdffff640 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30fb810 0x5636e30fbbf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bae26f31e474187 server_cookie=a66be3c750c1646b in_seq=0 out_seq=0 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 10 client.7779 ms_handle_connect on v2:172.21.15.70:6836/3578750329 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 1 -- 172.21.15.70:0/4031785823 <== mds.0 v2:172.21.15.70:6836/3578750329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x5636e2efab20 con 0x5636e30fb810 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 10 client.7779 handle_command_reply: tid=0 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 client.7779 shutdown 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 msgr2=0x5636e3159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e3159260 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8fc801cd70 tx=0x7f8fc80a8000 comp rx=0 tx=0).stop 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 shutdown_connections 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:6836/3578750329,v1:172.21.15.70:6837/3578750329] conn(0x5636e30fb810 0x5636e30fbbf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5636e30e9810 0x5636e30fb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636e30e8ef0 0x5636e3159260 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 --2- 172.21.15.70:0/4031785823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5636e309fbc0 0x5636e3158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 >> 172.21.15.70:0/4031785823 conn(0x5636e309bb70 msgr2=0x5636e30e77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fbeffd640 0 client.7779 ms_handle_reset on v2:172.21.15.70:6836/3578750329 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 shutdown_connections 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 1 -- 172.21.15.70:0/4031785823 wait complete. 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe94e6000 20 client.7779 trim_cache size 0 max 16384 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "b126cadd-d35d-4fad-9fb6-c51f861f8653", 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 7629, 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-22T01:18:57.045 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 4.7480830000000003, 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 50, 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 103, 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 103, 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 10.113186862999999 2024-04-22T01:18:57.046 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8fc00678c0 msgr2=0x7f8fc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8fc00678c0 0x7f8fc0069d80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0001770 tx=0x7f8fd0009040 comp rx=0 tx=0).stop 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 msgr2=0x7f8fe015b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 0x7f8fe015b6a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8007e30 tx=0x7f8fd800e4b0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 shutdown_connections 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8fc00678c0 0x7f8fc0069d80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fe015e0c0 0x7f8fe015b6a0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe00b02e0 0x7f8fe015f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 --2- 172.21.15.70:0/4271926735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8fe0099a60 0x7f8fe0152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 >> 172.21.15.70:0/4271926735 conn(0x7f8fe00b3c80 msgr2=0x7f8fe006ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 shutdown_connections 2024-04-22T01:18:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.039+0000 7f8fe636d640 1 -- 172.21.15.70:0/4271926735 wait complete. 2024-04-22T01:18:57.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:18:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 Processor -- start 2024-04-22T01:18:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 -- start start 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5cf8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5cf8072060 0x7f5cf806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf80cd7b0 con 0x7f5cf81455b0 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf80cc710 con 0x7f5cf8072060 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cfcc89640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf80cc9f0 con 0x7f5cf80cc150 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5cf81455b0 0x7f5cf80cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43270/0 (socket says 172.21.15.70:43270) 2024-04-22T01:18:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf659c640 1 -- 172.21.15.70:0/597565436 learned_addr learned my addr 172.21.15.70:0/597565436 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:18:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf6d9d640 1 --2- 172.21.15.70:0/597565436 >> v2:172.21.15.98:3300/0 conn(0x7f5cf8072060 0x7f5cf806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf5d9b640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5cf8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494723275 0 0) 0x7f5cf80cd7b0 con 0x7f5cf81455b0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce0003530 con 0x7f5cf81455b0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 659898387 0 0) 0x7f5ce0003530 con 0x7f5cf81455b0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 msgr2=0x7f5cf8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5cf8071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 >> v2:172.21.15.98:3300/0 conn(0x7f5cf8072060 msgr2=0x7f5cf806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 --2- 172.21.15.70:0/597565436 >> v2:172.21.15.98:3300/0 conn(0x7f5cf8072060 0x7f5cf806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf80cccd0 con 0x7f5cf81455b0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f5ce8001680 con 0x7f5cf81455b0 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 >> v1:172.21.15.70:6789/0 conn(0x7f5cf81455b0 legacy=0x7f5cf80cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce00039c0 0x7f5ce0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5ce0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce0004d50 0x7f5ce00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf80cc710 con 0x7f5ce00039c0 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf80cc9f0 con 0x7f5ce0004d50 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce000ba80 con 0x7f5cf80cc150 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.167+0000 7f5cf659c640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce00039c0 0x7f5ce0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf5d9b640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5ce0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf6d9d640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce0004d50 0x7f5ce00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf659c640 1 -- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 msgr2=0x7f5ce0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf659c640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5ce0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf659c640 1 -- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce0004d50 msgr2=0x7f5ce00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf659c640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce0004d50 0x7f5ce00051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf659c640 1 -- 172.21.15.70:0/597565436 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce0003530 con 0x7f5ce00039c0 2024-04-22T01:18:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf5d9b640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf80cc150 0x7f5ce0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf659c640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce00039c0 0x7f5ce0003de0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c240 tx=0x7f5ce801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e29b3074eab59bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8007d60 con 0x7f5ce00039c0 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce80a9020 con 0x7f5ce00039c0 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf559a640 1 -- 172.21.15.70:0/597565436 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8005b00 con 0x7f5ce00039c0 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cfcc89640 1 -- 172.21.15.70:0/597565436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce00039c0 msgr2=0x7f5ce0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/597565436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce00039c0 0x7f5ce0003de0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c240 tx=0x7f5ce801ccc0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cf6d9d640 1 -- 172.21.15.70:0/597565436 reap_dead start 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cfcc89640 1 -- 172.21.15.70:0/597565436 shutdown_connections 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cfcc89640 1 -- 172.21.15.70:0/597565436 >> 172.21.15.70:0/597565436 conn(0x7f5cf806b1f0 msgr2=0x7f5cf806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cfcc89640 1 -- 172.21.15.70:0/597565436 shutdown_connections 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.171+0000 7f5cfcc89640 1 -- 172.21.15.70:0/597565436 wait complete. 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 Processor -- start 2024-04-22T01:18:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- start start 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce00039c0 0x7f5cf815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf8143680 0x7f5cf8143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 0x7f5cf81e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf806f2e0 con 0x7f5cf81e9970 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf806f160 con 0x7f5ce00039c0 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf8072470 con 0x7f5cf8143680 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 0x7f5cf81e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 0x7f5cf81e9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45742/0 (socket says 172.21.15.70:45742) 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 -- 172.21.15.70:0/2101369275 learned_addr learned my addr 172.21.15.70:0/2101369275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf659c640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce00039c0 0x7f5cf815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf5d9b640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf8143680 0x7f5cf8143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 -- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf8143680 msgr2=0x7f5cf8143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf8143680 0x7f5cf8143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 -- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce00039c0 msgr2=0x7f5cf815ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce00039c0 0x7f5cf815ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 -- 172.21.15.70:0/2101369275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf81f8620 con 0x7f5cf81e9970 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf6d9d640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 0x7f5cf81e9df0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8070030 tx=0x7f5cf000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf001c070 con 0x7f5cf81e9970 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0017400 con 0x7f5cf81e9970 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf81f8900 con 0x7f5cf81e9970 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00177f0 con 0x7f5cf81e9970 2024-04-22T01:18:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf81f8ec0 con 0x7f5cf81e9970 2024-04-22T01:18:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf8068210 con 0x7f5cf81e9970 2024-04-22T01:18:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5cf001f070 con 0x7f5cf81e9970 2024-04-22T01:18:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5ce6ffd640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5cd80678f0 0x7f5cd8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f5cf00861c0 con 0x7f5cf81e9970 2024-04-22T01:18:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.175+0000 7f5cf659c640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5cd80678f0 0x7f5cd8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.179+0000 7f5cf659c640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5cd80678f0 0x7f5cd8069db0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80aa000 tx=0x7f5ce80b0040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.183+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5cf0012070 con 0x7f5cf81e9970 2024-04-22T01:18:57.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.375+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5cf815f510 con 0x7f5cf81e9970 2024-04-22T01:18:57.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.375+0000 7f5ce6ffd640 1 -- 172.21.15.70:0/2101369275 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7f5cf0056ad0 con 0x7f5cf81e9970 2024-04-22T01:18:57.375 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.375+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5cd80678f0 msgr2=0x7f5cd8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.375+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5cd80678f0 0x7f5cd8069db0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80aa000 tx=0x7f5ce80b0040 comp rx=0 tx=0).stop 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.375+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 msgr2=0x7f5cf81e9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.375+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 0x7f5cf81e9df0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8070030 tx=0x7f5cf000f4a0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 shutdown_connections 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5cd80678f0 0x7f5cd8069db0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cf81e9970 0x7f5cf81e9df0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cf8143680 0x7f5cf8143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 --2- 172.21.15.70:0/2101369275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ce00039c0 0x7f5cf815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:57.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 >> 172.21.15.70:0/2101369275 conn(0x7f5cf806b1f0 msgr2=0x7f5cf80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:57.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 shutdown_connections 2024-04-22T01:18:57.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57.379+0000 7f5cfcc89640 1 -- 172.21.15.70:0/2101369275 wait complete. 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:18:57.392 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:18:57.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:57.392 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.398 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:18:57.399 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42284sec preferred_lft 42284sec 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 02:42:cb:f6:7d:bd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:57.400 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run.smithi070.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:18:57.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:57.401 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:18:57.449 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-22T01:18:57.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:57.451 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:57.502 INFO:teuthology.orchestra.run.smithi070.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:18:57.502 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:18:57.502 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:18:57.502 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-22T01:18:57.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:18:57.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:18:57.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:18:57.502 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-22T01:18:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:18:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:57.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:57.621 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:18:57.669 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:18:57.669 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:18:57.670 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:18:57.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:57.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-22T01:18:57.726 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T01:18:57.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:57.726 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:57.773 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:18:57.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:18:57.774 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:57.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:18:57.878 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:18:57.926 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:57.927 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:57.928 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:57.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:57.929 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:18:57.935 INFO:tasks.ceph.mds.a.smithi070.stderr:2024-04-22T01:18:57.935+0000 7fccfb4de640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:18:57.940 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:18:57.940 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T01:18:57.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:18:58.002 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:58.049 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:18:58.049 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:18:58.050 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:58.051 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:58.052 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:58.052 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:58.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:58.053 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:18:58.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-22T01:18:58.067+0000 7f22b708a580 -1 init, newargv = 0x556b5c052b50 newargc=15 2024-04-22T01:18:58.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[46738]: starting ceph client 2024-04-22T01:18:58.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[46738]: starting fuse 2024-04-22T01:18:59.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-22T01:18:59.186 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-22T01:18:59.236 INFO:teuthology.orchestra.run.smithi070.stdout:/run/lock 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/fs/nfsd 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/rpc_pipefs 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-22T01:18:59.237 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:18:59.238 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:59.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:59.239 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-22T01:18:59.286 INFO:teuthology.orchestra.run.smithi070.stdout:45 2024-04-22T01:18:59.286 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:18:59.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:18:59.286 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-22T01:18:59.286 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-22T01:18:59.286 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-22T01:18:59.286 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:18:59.287 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-22T01:18:59.288 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-22T01:18:59.288 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-22T01:18:59.288 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:18:59.288 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-22T01:18:59.288 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-22T01:18:59.288 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-22T01:18:59.410 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.46738.asok 2024-04-22T01:18:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:18:59.422 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46738.asok 2024-04-22T01:18:59.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:59.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46738.asok status 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T01:18:59.577 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "46738", 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 7788, 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 7788 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:18:59.578 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3010334419 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3010334419 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.7788 192.168.0.1:0/3010334419", 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3010334419", 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 50, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 103, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-22T01:18:59.579 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-22T01:18:59.580 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:18:59.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:59.591 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-22T01:18:59.591 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:59.597 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-22T01:18:59.597 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:59.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:18:59.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:18:59.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-22T01:18:59.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.819+0000 7fc648721640 1 Processor -- start 2024-04-22T01:18:59.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.819+0000 7fc648721640 1 -- start start 2024-04-22T01:18:59.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6401455b0 0x7fc6400cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:59.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc640072230 0x7fc64006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:59.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6400d0250 con 0x7fc6400cc150 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc6400cd7b0 con 0x7fc640072230 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc6400cc710 con 0x7fc6401455b0 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc640072230 0x7fc64006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc640072230 0x7fc64006e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45754/0 (socket says 172.21.15.70:45754) 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 -- 172.21.15.70:0/3046486693 learned_addr learned my addr 172.21.15.70:0/3046486693 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc645cbc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc6400cc150 0x7fc640071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43284/0 (socket says 172.21.15.70:43284) 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc6464bd640 1 --2- 172.21.15.70:0/3046486693 >> v2:172.21.15.98:3300/0 conn(0x7fc6401455b0 0x7fc6400cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 -- 172.21.15.70:0/3046486693 >> v2:172.21.15.98:3300/0 conn(0x7fc6401455b0 msgr2=0x7fc6400cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 --2- 172.21.15.70:0/3046486693 >> v2:172.21.15.98:3300/0 conn(0x7fc6401455b0 0x7fc6400cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 -- 172.21.15.70:0/3046486693 >> v1:172.21.15.70:6789/0 conn(0x7fc6400cc150 legacy=0x7fc640071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 -- 172.21.15.70:0/3046486693 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6400cc9f0 con 0x7fc640072230 2024-04-22T01:18:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc6454bb640 1 -- 172.21.15.70:0/3046486693 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1937056765 0 0) 0x7fc6400d0250 con 0x7fc6400cc150 2024-04-22T01:18:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc646cbe640 1 --2- 172.21.15.70:0/3046486693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64006e200 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc6400d0700 tx=0x7fc63801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b7cf68b645bbd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc6454bb640 1 -- 172.21.15.70:0/3046486693 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6380a5590 con 0x7fc640072230 2024-04-22T01:18:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc6454bb640 1 -- 172.21.15.70:0/3046486693 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6380a5730 con 0x7fc640072230 2024-04-22T01:18:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc6454bb640 1 -- 172.21.15.70:0/3046486693 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6380a5b00 con 0x7fc640072230 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- 172.21.15.70:0/3046486693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 msgr2=0x7fc64006e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 --2- 172.21.15.70:0/3046486693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64006e200 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc6400d0700 tx=0x7fc63801c8d0 comp rx=0 tx=0).stop 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- 172.21.15.70:0/3046486693 shutdown_connections 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 --2- 172.21.15.70:0/3046486693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64006e200 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 --2- 172.21.15.70:0/3046486693 >> v2:172.21.15.98:3300/0 conn(0x7fc6401455b0 0x7fc6400cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- 172.21.15.70:0/3046486693 >> 172.21.15.70:0/3046486693 conn(0x7fc64006b1f0 msgr2=0x7fc64006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- 172.21.15.70:0/3046486693 shutdown_connections 2024-04-22T01:18:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.823+0000 7fc648721640 1 -- 172.21.15.70:0/3046486693 wait complete. 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 Processor -- start 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- start start 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64015b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6400cc150 0x7fc64013e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6401455b0 0x7fc64013e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc64006f2e0 con 0x7fc640072230 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc64006f160 con 0x7fc6400cc150 2024-04-22T01:18:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc640168b40 con 0x7fc6401455b0 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64015b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64015b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45762/0 (socket says 172.21.15.70:45762) 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 -- 172.21.15.70:0/2246143059 learned_addr learned my addr 172.21.15.70:0/2246143059 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc645cbc640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6400cc150 0x7fc64013e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc646cbe640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6401455b0 0x7fc64013e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 -- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6401455b0 msgr2=0x7fc64013e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6401455b0 0x7fc64013e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 -- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6400cc150 msgr2=0x7fc64013e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6400cc150 0x7fc64013e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 -- 172.21.15.70:0/2246143059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc64015b8e0 con 0x7fc640072230 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc645cbc640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6400cc150 0x7fc64013e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:18:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc6464bd640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64015b3a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc63c00d800 tx=0x7fc63c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c00b660 con 0x7fc640072230 2024-04-22T01:18:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc63c019070 con 0x7fc640072230 2024-04-22T01:18:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc64013c680 con 0x7fc640072230 2024-04-22T01:18:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c00ba00 con 0x7fc640072230 2024-04-22T01:18:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc64013cc40 con 0x7fc640072230 2024-04-22T01:18:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.827+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc640068210 con 0x7fc640072230 2024-04-22T01:18:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.831+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc63c00bba0 con 0x7fc640072230 2024-04-22T01:18:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.831+0000 7fc636ffd640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6240678c0 0x7fc624069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:18:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.831+0000 7fc645cbc640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6240678c0 0x7fc624069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:18:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.831+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc63c086270 con 0x7fc640072230 2024-04-22T01:18:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.831+0000 7fc645cbc640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6240678c0 0x7fc624069d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc630007fd0 tx=0x7fc630009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:18:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:18:59.835+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc63c056c20 con 0x7fc640072230 2024-04-22T01:19:00.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc64015f510 con 0x7fc640072230 2024-04-22T01:19:00.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc636ffd640 1 -- 172.21.15.70:0/2246143059 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7fc63c056520 con 0x7fc640072230 2024-04-22T01:19:00.030 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6240678c0 msgr2=0x7fc624069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6240678c0 0x7fc624069d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc630007fd0 tx=0x7fc630009040 comp rx=0 tx=0).stop 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 msgr2=0x7fc64015b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64015b3a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc63c00d800 tx=0x7fc63c00dcc0 comp rx=0 tx=0).stop 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 shutdown_connections 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc6240678c0 0x7fc624069d80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6401455b0 0x7fc64013e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6400cc150 0x7fc64013e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 --2- 172.21.15.70:0/2246143059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc640072230 0x7fc64015b3a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 >> 172.21.15.70:0/2246143059 conn(0x7fc64006b1f0 msgr2=0x7fc640071b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 shutdown_connections 2024-04-22T01:19:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:00.031+0000 7fc648721640 1 -- 172.21.15.70:0/2246143059 wait complete. 2024-04-22T01:19:00.047 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:19:00.047 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:19:00.047 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T01:19:00.047 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T01:19:00.047 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:00.047 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:19:00.048 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:19:00.048 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:19:00.048 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:19:00.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.048 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T01:19:00.055 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:19:00.055 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:19:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42286sec preferred_lft 42286sec 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:e6:d6:eb:61 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.057 INFO:teuthology.orchestra.run.smithi098.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-22T01:19:00.059 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:00.059 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-22T01:19:00.059 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:19:00.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.059 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:19:00.108 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T01:19:00.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.110 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T01:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-22T01:19:00.162 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:19:00.163 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:19:00.163 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T01:19:00.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T01:19:00.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T01:19:00.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T01:19:00.163 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T01:19:00.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:19:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:19:00.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:19:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:19:00.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.277 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:19:00.324 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:19:00.325 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:19:00.326 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:19:00.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T01:19:00.381 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T01:19:00.381 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:19:00.382 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:00.429 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T01:19:00.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:19:00.429 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:00.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:19:00.535 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:19:00.588 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:19:00.589 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:00.590 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:00.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.591 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:19:00.637 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:19:00.637 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T01:19:00.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:19:00.696 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:19:00.742 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:19:00.743 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:19:00.744 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:00.745 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:00.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:00.746 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:19:00.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:2024-04-22T01:19:00.763+0000 7f9f42c52580 -1 init, newargv = 0x560713dd0770 newargc=15 2024-04-22T01:19:00.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34806]: starting ceph client 2024-04-22T01:19:00.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34806]: starting fuse 2024-04-22T01:19:01.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-22T01:19:01.884 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-22T01:19:01.931 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/run/lock 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/fs/nfsd 2024-04-22T01:19:01.932 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/run/rpc_pipefs 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-4 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-5 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-6 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-7 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-22T01:19:01.933 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:01.934 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:01.934 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:01.934 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:01.934 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:01.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:01.935 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-22T01:19:01.983 INFO:teuthology.orchestra.run.smithi098.stdout:45 2024-04-22T01:19:01.983 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-22T01:19:01.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:19:01.983 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-22T01:19:01.983 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-22T01:19:01.983 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:19:01.983 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-22T01:19:01.984 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-22T01:19:01.985 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-22T01:19:01.985 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:19:01.985 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:01.985 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.1")) 2024-04-22T01:19:01.985 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:19:02.107 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.1.34806.asok 2024-04-22T01:19:02.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:19:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:19:02.119 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34806.asok 2024-04-22T01:19:02.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34806.asok status 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "1", 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T01:19:02.274 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "34806", 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5663, 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5663 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:19:02.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 879295054 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 879295054 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.5663 192.168.0.1:0/879295054", 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/879295054", 2024-04-22T01:19:02.276 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-22T01:19:02.277 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 50, 2024-04-22T01:19:02.277 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 103, 2024-04-22T01:19:02.277 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:19:02.277 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-22T01:19:02.277 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-22T01:19:02.277 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T01:19:02.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.288 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-22T01:19:02.288 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:02.294 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-22T01:19:02.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:02.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:02.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.350 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir_files) 2024-04-22T01:19:02.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.365 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir_data) 2024-04-22T01:19:02.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.418 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 100 ./subdir_files) 2024-04-22T01:19:02.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.478 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 104857600 ./subdir_data) 2024-04-22T01:19:02.532 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:02.533 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:02.539 DEBUG:teuthology.orchestra.run.smithi098:> stdin-killer -- python3 -c ' 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> import uuid 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> n = 20 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> create_hard_links = False 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> if 0 > 0: 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> create_hard_links = True 2024-04-22T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi098:> path = "/home/ubuntu/cephtest/mnt.1/subdir_files/file" 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> dpath = os.path.dirname(path) 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> fnameprefix = os.path.basename(path) 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> os.makedirs(dpath, exist_ok=True) 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> try: 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> for i in range(n): 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> with open(fpath, '"'"'w'"'"') as f: 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> f.write(f"{i}") 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> f.flush() 2024-04-22T01:19:02.541 DEBUG:teuthology.orchestra.run.smithi098:> os.fsync(f.fileno()) 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> os.unlink(fpath) 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> os.fsync(dirfd) 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> if create_hard_links: 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> for j in range(0): 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> os.fsync(dirfd) 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> finally: 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> os.close(dirfd) 2024-04-22T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:19:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:19:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:19:02.887 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:02.888 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:02.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:02.935 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=subdir_data/file bs=1M conv=fdatasync count=20 seek=0) 2024-04-22T01:19:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:20+0 records in 2024-04-22T01:19:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:20+0 records out 2024-04-22T01:19:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:20971520 bytes (21 MB, 20 MiB) copied, 0.206561 s, 102 MB/s 2024-04-22T01:19:03.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:03.202 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 10 ./subdir_files) 2024-04-22T01:19:03.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:03.215 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1048576 ./subdir_data) 2024-04-22T01:19:03.268 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:03.269 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> stdin-killer -- python3 -c ' 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> import uuid 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> n = 40 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> create_hard_links = False 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> if 0 > 0: 2024-04-22T01:19:03.275 DEBUG:teuthology.orchestra.run.smithi098:> create_hard_links = True 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> path = "/home/ubuntu/cephtest/mnt.1/subdir_files/file" 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> dpath = os.path.dirname(path) 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> fnameprefix = os.path.basename(path) 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> os.makedirs(dpath, exist_ok=True) 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> try: 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> for i in range(n): 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> with open(fpath, '"'"'w'"'"') as f: 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> f.write(f"{i}") 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:03.276 DEBUG:teuthology.orchestra.run.smithi098:> f.flush() 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> os.fsync(f.fileno()) 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> os.unlink(fpath) 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> os.fsync(dirfd) 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> if create_hard_links: 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> for j in range(0): 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> if False: 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> os.fsync(dirfd) 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> finally: 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> os.close(dirfd) 2024-04-22T01:19:03.277 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-22T01:19:03.673 INFO:teuthology.orchestra.run.smithi098.stderr:Traceback (most recent call last): 2024-04-22T01:19:03.673 INFO:teuthology.orchestra.run.smithi098.stderr: File "", line 20, in 2024-04-22T01:19:03.673 INFO:teuthology.orchestra.run.smithi098.stderr:OSError: [Errno 122] Disk quota exceeded: '/home/ubuntu/cephtest/mnt.1/subdir_files/file_20' 2024-04-22T01:19:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T01:19:03 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-04-22T01:19:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:03.693 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:03.693 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:03.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:03.742 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=subdir_data/file bs=1M conv=fdatasync count=40 seek=0) 2024-04-22T01:19:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:dd: error writing 'subdir_data/file': Disk quota exceeded 2024-04-22T01:19:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:3+0 records in 2024-04-22T01:19:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2+0 records out 2024-04-22T01:19:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2490368 bytes (2.5 MB, 2.4 MiB) copied, 0.0179134 s, 139 MB/s 2024-04-22T01:19:03.825 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-22T01:19:03.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:19:03.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-22T01:19:03.915 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-22T01:19:03.915 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:19:03.915 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:19:03.915 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:03.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:19:04.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a9ad29640 1 Processor -- start 2024-04-22T01:19:04.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a9ad29640 1 -- start start 2024-04-22T01:19:04.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a9ad29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 0x7f9a94099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a9ad29640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a9415c220 con 0x7f9a94066f50 2024-04-22T01:19:04.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a9ad29640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a940674a0 con 0x7f9a940ad880 2024-04-22T01:19:04.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a9ad29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9415c570 con 0x7f9a9415be40 2024-04-22T01:19:04.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9a94066f50 0x7f9a940ad170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43296/0 (socket says 172.21.15.70:43296) 2024-04-22T01:19:04.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 -- 172.21.15.70:0/437088816 learned_addr learned my addr 172.21.15.70:0/437088816 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 0x7f9a94099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875893091 0 0) 0x7f9a9415c220 con 0x7f9a94066f50 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a7c003530 con 0x7f9a94066f50 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2265504247 0 0) 0x7f9a7c003530 con 0x7f9a94066f50 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 msgr2=0x7f9a94099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 0x7f9a94099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 >> v1:172.21.15.98:6789/0 conn(0x7f9a940ad880 legacy=0x7f9a9415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9406c4b0 con 0x7f9a94066f50 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f9a84001680 con 0x7f9a94066f50 2024-04-22T01:19:04.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 >> v1:172.21.15.70:6789/0 conn(0x7f9a94066f50 legacy=0x7f9a940ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7c0039f0 0x7f9a7c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a7c004810 0x7f9a7c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 0x7f9a7c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9415c570 con 0x7f9a7c004810 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c00ba80 con 0x7f9a7c0039f0 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a7c00bc80 con 0x7f9a9415be40 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a7c004810 0x7f9a7c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a992c6640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 0x7f9a7c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7c0039f0 0x7f9a7c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 -- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 msgr2=0x7f9a7c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a9415be40 0x7f9a7c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 -- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7c0039f0 msgr2=0x7f9a7c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7c0039f0 0x7f9a7c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a93fff640 1 -- 172.21.15.70:0/437088816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7c003530 con 0x7f9a7c004810 2024-04-22T01:19:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.091+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7c0039f0 0x7f9a7c003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:19:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a93fff640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a7c004810 0x7f9a7c004c90 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400c210 tx=0x7f9a8401f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980df5b818c896b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84007d20 con 0x7f9a7c004810 2024-04-22T01:19:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a84005ae0 con 0x7f9a7c004810 2024-04-22T01:19:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a937fe640 1 -- 172.21.15.70:0/437088816 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84007070 con 0x7f9a7c004810 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- 172.21.15.70:0/437088816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a7c004810 msgr2=0x7f9a7c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/437088816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a7c004810 0x7f9a7c004c90 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400c210 tx=0x7f9a8401f390 comp rx=0 tx=0).stop 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 -- 172.21.15.70:0/437088816 reap_dead start 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- 172.21.15.70:0/437088816 shutdown_connections 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- 172.21.15.70:0/437088816 >> 172.21.15.70:0/437088816 conn(0x7f9a940b3d90 msgr2=0x7f9a940b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- 172.21.15.70:0/437088816 shutdown_connections 2024-04-22T01:19:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- 172.21.15.70:0/437088816 wait complete. 2024-04-22T01:19:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 Processor -- start 2024-04-22T01:19:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- start start 2024-04-22T01:19:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a940d0270 0x7f9a9415ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a940d0650 0x7f9a94158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 0x7f9a94155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a940d0fa0 con 0x7f9a94158d60 2024-04-22T01:19:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9406ca70 con 0x7f9a940d0270 2024-04-22T01:19:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a9ad29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a94099b80 con 0x7f9a940d0650 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 0x7f9a94155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 0x7f9a94155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45772/0 (socket says 172.21.15.70:45772) 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 -- 172.21.15.70:0/1806200601 learned_addr learned my addr 172.21.15.70:0/1806200601 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a93fff640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a940d0650 0x7f9a94158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a940d0270 0x7f9a9415ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 -- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a940d0650 msgr2=0x7f9a94158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a940d0650 0x7f9a94158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 -- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a940d0270 msgr2=0x7f9a9415ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a940d0270 0x7f9a9415ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 -- 172.21.15.70:0/1806200601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a94159d00 con 0x7f9a94158d60 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a93fff640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a940d0650 0x7f9a94158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a940d0270 0x7f9a9415ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:04.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.099+0000 7f9a992c6640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 0x7f9a94155c70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9a940add50 tx=0x7f9a8c00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c003b90 con 0x7f9a94158d60 2024-04-22T01:19:04.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8c018070 con 0x7f9a94158d60 2024-04-22T01:19:04.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a94159fe0 con 0x7f9a94158d60 2024-04-22T01:19:04.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0050d0 con 0x7f9a94158d60 2024-04-22T01:19:04.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9415a5a0 con 0x7f9a94158d60 2024-04-22T01:19:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a94068920 con 0x7f9a94158d60 2024-04-22T01:19:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9a8c005270 con 0x7f9a94158d60 2024-04-22T01:19:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a917fa640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a800678f0 0x7f9a80069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9a8c085cb0 con 0x7f9a94158d60 2024-04-22T01:19:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a800678f0 0x7f9a80069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.103+0000 7f9a98ac5640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a800678f0 0x7f9a80069db0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009a20 tx=0x7f9a88008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.107+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f9a8c056660 con 0x7f9a94158d60 2024-04-22T01:19:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.267+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a941e3640 con 0x7f9a94158d60 2024-04-22T01:19:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.271+0000 7f9a917fa640 1 -- 172.21.15.70:0/1806200601 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7f9a8c055f60 con 0x7f9a94158d60 2024-04-22T01:19:04.271 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:04.271 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":103,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:53.308712+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a800678f0 msgr2=0x7f9a80069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a800678f0 0x7f9a80069db0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009a20 tx=0x7f9a88008040 comp rx=0 tx=0).stop 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 msgr2=0x7f9a94155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 0x7f9a94155c70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9a940add50 tx=0x7f9a8c00fd40 comp rx=0 tx=0).stop 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 shutdown_connections 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f9a800678f0 0x7f9a80069db0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a94158d60 0x7f9a94155c70 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a940d0650 0x7f9a94158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 --2- 172.21.15.70:0/1806200601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a940d0270 0x7f9a9415ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 >> 172.21.15.70:0/1806200601 conn(0x7f9a940b3d90 msgr2=0x7f9a940ad170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 shutdown_connections 2024-04-22T01:19:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.275+0000 7f9a9ad29640 1 -- 172.21.15.70:0/1806200601 wait complete. 2024-04-22T01:19:04.290 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:18:49.869911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:18:50.621566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:19:04.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:19:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50f2080640 1 Processor -- start 2024-04-22T01:19:04.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50f2080640 1 -- start start 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50f2080640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50ec166b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50f2080640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec15c290 con 0x7f50ec0b4450 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50f2080640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec15c570 con 0x7f50ec072280 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50f2080640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50ec0b3a50 con 0x7f50ec0cd220 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f50ec0b4450 0x7f50ec071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43312/0 (socket says 172.21.15.70:43312) 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eb7fe640 1 -- 172.21.15.70:0/3157223175 learned_addr learned my addr 172.21.15.70:0/3157223175 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ebfff640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50ec166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564529836 0 0) 0x7f50ec15c290 con 0x7f50ec0b4450 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d4003530 con 0x7f50ec0b4450 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662044565 0 0) 0x7f50ec15c570 con 0x7f50ec072280 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50ec15c290 con 0x7f50ec072280 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3593341847 0 0) 0x7f50d4003530 con 0x7f50ec0b4450 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 msgr2=0x7f50ec166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50ec166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 >> v1:172.21.15.98:6789/0 conn(0x7f50ec072280 legacy=0x7f50ec0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec15c850 con 0x7f50ec0b4450 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f50e0001680 con 0x7f50ec0b4450 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ebfff640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50ec166b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 >> v1:172.21.15.70:6789/0 conn(0x7f50ec0b4450 legacy=0x7f50ec071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50d4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50d4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d4004d50 0x7f50d40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec0b3a50 con 0x7f50d40040c0 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50d400ba80 con 0x7f50d4004d50 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50d400bc80 con 0x7f50ec0cd220 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eaffd640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50d4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eb7fe640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50d4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50ebfff640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d4004d50 0x7f50d40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eaffd640 1 -- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 msgr2=0x7f50d4003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eaffd640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec0cd220 0x7f50d4003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eaffd640 1 -- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d4004d50 msgr2=0x7f50d40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eaffd640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d4004d50 0x7f50d40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.411+0000 7f50eaffd640 1 -- 172.21.15.70:0/3157223175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d4003530 con 0x7f50d40040c0 2024-04-22T01:19:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50eaffd640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50d4004810 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0018d0 tx=0x7f50dc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536f9c53bb438877 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc001520 con 0x7f50d40040c0 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50dc0052f0 con 0x7f50d40040c0 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50ea7fc640 1 -- 172.21.15.70:0/3157223175 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0a8950 con 0x7f50d40040c0 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 -- 172.21.15.70:0/3157223175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 msgr2=0x7f50d4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 --2- 172.21.15.70:0/3157223175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50d4004810 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0018d0 tx=0x7f50dc01ecf0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50ebfff640 1 -- 172.21.15.70:0/3157223175 reap_dead start 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 -- 172.21.15.70:0/3157223175 shutdown_connections 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 -- 172.21.15.70:0/3157223175 >> 172.21.15.70:0/3157223175 conn(0x7f50ec0b1d40 msgr2=0x7f50ec0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 -- 172.21.15.70:0/3157223175 shutdown_connections 2024-04-22T01:19:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 -- 172.21.15.70:0/3157223175 wait complete. 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 Processor -- start 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.415+0000 7f50f2080640 1 -- start start 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50ec15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec13cfc0 0x7f50ec13d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50ec06ee70 0x7f50ec06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec0b4830 con 0x7f50d40040c0 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec15d050 con 0x7f50ec06ee70 2024-04-22T01:19:04.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50ec0b49b0 con 0x7f50ec13cfc0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50ec15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50ec15aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45786/0 (socket says 172.21.15.70:45786) 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 -- 172.21.15.70:0/1881818734 learned_addr learned my addr 172.21.15.70:0/1881818734 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eaffd640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec13cfc0 0x7f50ec13d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50ebfff640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50ec06ee70 0x7f50ec06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 -- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec13cfc0 msgr2=0x7f50ec13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec13cfc0 0x7f50ec13d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 -- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50ec06ee70 msgr2=0x7f50ec06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50ec06ee70 0x7f50ec06f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 -- 172.21.15.70:0/1881818734 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec150320 con 0x7f50d40040c0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eb7fe640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50ec15aeb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f50e0007340 tx=0x7f50e0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e000f9c0 con 0x7f50d40040c0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e0018070 con 0x7f50d40040c0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e000fdc0 con 0x7f50d40040c0 2024-04-22T01:19:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec1505c0 con 0x7f50d40040c0 2024-04-22T01:19:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec150b80 con 0x7f50d40040c0 2024-04-22T01:19:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ec068210 con 0x7f50d40040c0 2024-04-22T01:19:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50e001c410 con 0x7f50d40040c0 2024-04-22T01:19:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50d3fff640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678f0 0x7f50d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f50e0085920 con 0x7f50d40040c0 2024-04-22T01:19:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.419+0000 7f50eaffd640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678f0 0x7f50d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.423+0000 7f50eaffd640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678f0 0x7f50d8069db0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0018d0 tx=0x7f50dc0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.423+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f50e0056360 con 0x7f50d40040c0 2024-04-22T01:19:04.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.583+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f50ec0d01e0 con 0x7f50d40040c0 2024-04-22T01:19:04.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50d3fff640 1 -- 172.21.15.70:0/1881818734 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7f50e0055c60 con 0x7f50d40040c0 2024-04-22T01:19:04.585 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:19:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678f0 msgr2=0x7f50d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678f0 0x7f50d8069db0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0018d0 tx=0x7f50dc0aa070 comp rx=0 tx=0).stop 2024-04-22T01:19:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 msgr2=0x7f50ec15aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50ec15aeb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f50e0007340 tx=0x7f50e0007800 comp rx=0 tx=0).stop 2024-04-22T01:19:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 shutdown_connections 2024-04-22T01:19:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f50d80678f0 0x7f50d8069db0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50ec06ee70 0x7f50ec06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50ec13cfc0 0x7f50ec13d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 --2- 172.21.15.70:0/1881818734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d40040c0 0x7f50ec15aeb0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 >> 172.21.15.70:0/1881818734 conn(0x7f50ec0b1d40 msgr2=0x7f50ec071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 shutdown_connections 2024-04-22T01:19:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.587+0000 7f50f2080640 1 -- 172.21.15.70:0/1881818734 wait complete. 2024-04-22T01:19:04.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:19:04.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.727+0000 7f1ecc141640 1 Processor -- start 2024-04-22T01:19:04.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ecc141640 1 -- start start 2024-04-22T01:19:04.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ecc141640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1ec415b030 0x7f1ec415b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ecc141640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec415ba20 con 0x7f1ec415ac50 2024-04-22T01:19:04.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ecc141640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec415bc90 con 0x7f1ec40d08e0 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ecc141640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec415bf70 con 0x7f1ec415b030 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1ec415ac50 0x7f1ec415f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43324/0 (socket says 172.21.15.70:43324) 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 -- 172.21.15.70:0/1678968913 learned_addr learned my addr 172.21.15.70:0/1678968913 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec96dc640 1 --2- 172.21.15.70:0/1678968913 >> v2:172.21.15.98:3300/0 conn(0x7f1ec415b030 0x7f1ec415b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833810659 0 0) 0x7f1ec415ba20 con 0x7f1ec415ac50 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb0003530 con 0x7f1ec415ac50 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3174096886 0 0) 0x7f1eb0003530 con 0x7f1ec415ac50 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 >> v2:172.21.15.98:3300/0 conn(0x7f1ec415b030 msgr2=0x7f1ec415b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 --2- 172.21.15.70:0/1678968913 >> v2:172.21.15.98:3300/0 conn(0x7f1ec415b030 0x7f1ec415b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 >> v1:172.21.15.98:6789/0 conn(0x7f1ec40d08e0 legacy=0x7f1ec40d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec415e940 con 0x7f1ec415ac50 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f1ec0001680 con 0x7f1ec415ac50 2024-04-22T01:19:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 >> v1:172.21.15.70:6789/0 conn(0x7f1ec415ac50 legacy=0x7f1ec415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1eb0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb0004810 0x7f1eb0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb00051d0 0x7f1eb000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec415bf70 con 0x7f1eb00039c0 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb000e280 con 0x7f1eb0004810 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb000e480 con 0x7f1eb00051d0 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1eb0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 -- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb00051d0 msgr2=0x7f1eb000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1eca6de640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb00051d0 0x7f1eb000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb00051d0 0x7f1eb000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec96dc640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb0004810 0x7f1eb0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 -- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb0004810 msgr2=0x7f1eb0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb0004810 0x7f1eb0004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1eca6de640 1 -- 172.21.15.70:0/1678968913 reap_dead start 2024-04-22T01:19:04.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.731+0000 7f1ec9edd640 1 -- 172.21.15.70:0/1678968913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb0003530 con 0x7f1eb00039c0 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1eb0003de0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000c240 tx=0x7f1ec001cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8233982d89cb96c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0007d20 con 0x7f1eb00039c0 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec00a9070 con 0x7f1eb00039c0 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec8edb640 1 -- 172.21.15.70:0/1678968913 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00058e0 con 0x7f1eb00039c0 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- 172.21.15.70:0/1678968913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 msgr2=0x7f1eb0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1eb0003de0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000c240 tx=0x7f1ec001cc50 comp rx=0 tx=0).stop 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- 172.21.15.70:0/1678968913 shutdown_connections 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 --2- 172.21.15.70:0/1678968913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1eb0003de0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- 172.21.15.70:0/1678968913 >> 172.21.15.70:0/1678968913 conn(0x7f1ec4099a60 msgr2=0x7f1ec4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- 172.21.15.70:0/1678968913 shutdown_connections 2024-04-22T01:19:04.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- 172.21.15.70:0/1678968913 wait complete. 2024-04-22T01:19:04.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 Processor -- start 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- start start 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1ec4149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec40ad8d0 0x7f1ec40add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec40c8550 0x7f1ec40c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec40cbc00 con 0x7f1eb00039c0 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec415b850 con 0x7f1ec40c8550 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ecc141640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec40cbd80 con 0x7f1ec40ad8d0 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1ec4149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1ec4149f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45804/0 (socket says 172.21.15.70:45804) 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 -- 172.21.15.70:0/4191027093 learned_addr learned my addr 172.21.15.70:0/4191027093 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1eca6de640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec40c8550 0x7f1ec40c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 -- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec40ad8d0 msgr2=0x7f1ec40add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec96dc640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec40ad8d0 0x7f1ec40add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec40ad8d0 0x7f1ec40add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 -- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec40c8550 msgr2=0x7f1ec40c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec40c8550 0x7f1ec40c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.735+0000 7f1ec9edd640 1 -- 172.21.15.70:0/4191027093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec40ae370 con 0x7f1eb00039c0 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eca6de640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec40c8550 0x7f1ec40c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ec96dc640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec40ad8d0 0x7f1ec40add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ec9edd640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1ec4149f70 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f1ec001ca80 tx=0x7f1eb0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0007d20 con 0x7f1eb00039c0 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec0000ee0 con 0x7f1eb00039c0 2024-04-22T01:19:04.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0004a10 con 0x7f1eb00039c0 2024-04-22T01:19:04.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec40ae650 con 0x7f1eb00039c0 2024-04-22T01:19:04.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec40c8030 con 0x7f1eb00039c0 2024-04-22T01:19:04.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec4068920 con 0x7f1eb00039c0 2024-04-22T01:19:04.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1ec0004bb0 con 0x7f1eb00039c0 2024-04-22T01:19:04.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eba7fc640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea80678c0 0x7f1ea8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:04.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f1ec011f370 con 0x7f1eb00039c0 2024-04-22T01:19:04.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ec96dc640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea80678c0 0x7f1ea8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:04.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.739+0000 7f1ec96dc640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea80678c0 0x7f1ea8069d80 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1ec41f6130 tx=0x7f1eb4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:04.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.743+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1ec0016990 con 0x7f1eb00039c0 2024-04-22T01:19:04.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ec415b9d0 con 0x7f1eb00039c0 2024-04-22T01:19:04.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1eba7fc640 1 -- 172.21.15.70:0/4191027093 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f1ec00efcc0 con 0x7f1eb00039c0 2024-04-22T01:19:04.902 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:19:04.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea80678c0 msgr2=0x7f1ea8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea80678c0 0x7f1ea8069d80 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1ec41f6130 tx=0x7f1eb4008040 comp rx=0 tx=0).stop 2024-04-22T01:19:04.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 msgr2=0x7f1ec4149f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:04.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1ec4149f70 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f1ec001ca80 tx=0x7f1eb0005840 comp rx=0 tx=0).stop 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 shutdown_connections 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1ea80678c0 0x7f1ea8069d80 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ec40c8550 0x7f1ec40c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ec40ad8d0 0x7f1ec40add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 --2- 172.21.15.70:0/4191027093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1eb00039c0 0x7f1ec4149f70 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 >> 172.21.15.70:0/4191027093 conn(0x7f1ec4099a60 msgr2=0x7f1ec415e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 shutdown_connections 2024-04-22T01:19:04.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:04.903+0000 7f1ecc141640 1 -- 172.21.15.70:0/4191027093 wait complete. 2024-04-22T01:19:04.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:19:05.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.039+0000 7faa43dd9640 1 Processor -- start 2024-04-22T01:19:05.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.039+0000 7faa43dd9640 1 -- start start 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa43dd9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa43dd9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa3c0cd220 0x7faa3c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa43dd9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3c15c290 con 0x7faa3c0720b0 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa43dd9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faa3c072510 con 0x7faa3c0cd220 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa43dd9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faa3c15c570 con 0x7faa3c0b4450 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa41b75640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa41b75640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.70:46820/0 (socket says 172.21.15.70:46820) 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa41b75640 1 -- 172.21.15.70:0/459542831 learned_addr learned my addr 172.21.15.70:0/459542831 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:05.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa42376640 1 --2- 172.21.15.70:0/459542831 >> v2:172.21.15.98:3300/0 conn(0x7faa3c0cd220 0x7faa3c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa42376640 1 -- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 msgr2=0x7faa3c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:05.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa42376640 1 --2- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa42376640 1 -- 172.21.15.70:0/459542831 >> v1:172.21.15.98:6789/0 conn(0x7faa3c0720b0 legacy=0x7faa3c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa42376640 1 -- 172.21.15.70:0/459542831 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c15c850 con 0x7faa3c0cd220 2024-04-22T01:19:05.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.043+0000 7faa41b75640 1 --2- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:19:05.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa42376640 1 --2- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0cd220 0x7faa3c0b2a40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faa34009d90 tx=0x7faa3401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265632b3b0dd2d02 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa40b73640 1 -- 172.21.15.70:0/459542831 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340b2070 con 0x7faa3c0cd220 2024-04-22T01:19:05.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa40b73640 1 -- 172.21.15.70:0/459542831 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa340a5990 con 0x7faa3c0cd220 2024-04-22T01:19:05.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa40b73640 1 -- 172.21.15.70:0/459542831 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340a5d60 con 0x7faa3c0cd220 2024-04-22T01:19:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 -- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0cd220 msgr2=0x7faa3c0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 --2- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0cd220 0x7faa3c0b2a40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faa34009d90 tx=0x7faa3401c940 comp rx=0 tx=0).stop 2024-04-22T01:19:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 -- 172.21.15.70:0/459542831 shutdown_connections 2024-04-22T01:19:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 --2- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0cd220 0x7faa3c0b2a40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 --2- 172.21.15.70:0/459542831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 -- 172.21.15.70:0/459542831 >> 172.21.15.70:0/459542831 conn(0x7faa3c0b1d40 msgr2=0x7faa3c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:05.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 -- 172.21.15.70:0/459542831 shutdown_connections 2024-04-22T01:19:05.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 -- 172.21.15.70:0/459542831 wait complete. 2024-04-22T01:19:05.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 Processor -- start 2024-04-22T01:19:05.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.047+0000 7faa43dd9640 1 -- start start 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0720b0 0x7faa3c14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 0x7faa3c152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c0cd980 con 0x7faa3c0cd220 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c0cd800 con 0x7faa3c0720b0 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faa3c0b3790 con 0x7faa3c0b4450 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 0x7faa3c152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 0x7faa3c152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45820/0 (socket says 172.21.15.70:45820) 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 -- 172.21.15.70:0/465499267 learned_addr learned my addr 172.21.15.70:0/465499267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa41b75640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0720b0 0x7faa3c14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa41374640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 -- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 msgr2=0x7faa3c145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c145320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 -- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0720b0 msgr2=0x7faa3c14c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0720b0 0x7faa3c14c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 -- 172.21.15.70:0/465499267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c145a90 con 0x7faa3c0cd220 2024-04-22T01:19:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa42376640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 0x7faa3c152ae0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7faa340b0040 tx=0x7faa340196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340b2050 con 0x7faa3c0cd220 2024-04-22T01:19:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa340be070 con 0x7faa3c0cd220 2024-04-22T01:19:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3c150850 con 0x7faa3c0cd220 2024-04-22T01:19:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340ada70 con 0x7faa3c0cd220 2024-04-22T01:19:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3c150e10 con 0x7faa3c0cd220 2024-04-22T01:19:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa3c068210 con 0x7faa3c0cd220 2024-04-22T01:19:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa340adc10 con 0x7faa3c0cd220 2024-04-22T01:19:05.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa327fc640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa200678c0 0x7faa20069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa41b75640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa200678c0 0x7faa20069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.051+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7faa34120dc0 con 0x7faa3c0cd220 2024-04-22T01:19:05.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.055+0000 7faa41b75640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa200678c0 0x7faa20069d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7faa38006cb0 tx=0x7faa38009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.059+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7faa34016560 con 0x7faa3c0cd220 2024-04-22T01:19:05.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.207+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa3c0d01e0 con 0x7faa3c0cd220 2024-04-22T01:19:05.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.207+0000 7faa327fc640 1 -- 172.21.15.70:0/465499267 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7faa340f12d0 con 0x7faa3c0cd220 2024-04-22T01:19:05.208 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa200678c0 msgr2=0x7faa20069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa200678c0 0x7faa20069d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7faa38006cb0 tx=0x7faa38009240 comp rx=0 tx=0).stop 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 msgr2=0x7faa3c152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 0x7faa3c152ae0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7faa340b0040 tx=0x7faa340196e0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 shutdown_connections 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7faa200678c0 0x7faa20069d80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa3c0cd220 0x7faa3c152ae0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa3c0b4450 0x7faa3c145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 --2- 172.21.15.70:0/465499267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa3c0720b0 0x7faa3c14c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 >> 172.21.15.70:0/465499267 conn(0x7faa3c0b1d40 msgr2=0x7faa3c0cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:05.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 shutdown_connections 2024-04-22T01:19:05.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.211+0000 7faa43dd9640 1 -- 172.21.15.70:0/465499267 wait complete. 2024-04-22T01:19:05.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:19:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 Processor -- start 2024-04-22T01:19:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 -- start start 2024-04-22T01:19:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3f6015bd00 0x7f3f6015cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3f600cdfe0 0x7f3f600b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6015c470 con 0x7f3f6006b1f0 2024-04-22T01:19:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3f600b1cd0 con 0x7f3f6015bd00 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f65002640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3f600b1f80 con 0x7f3f600cdfe0 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3f6015bd00 0x7f3f6015cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3f6015bd00 0x7f3f6015cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45836/0 (socket says 172.21.15.70:45836) 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 -- 172.21.15.70:0/3014507065 learned_addr learned my addr 172.21.15.70:0/3014507065 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5e59c640 1 --2- 172.21.15.70:0/3014507065 >> v2:172.21.15.98:3300/0 conn(0x7f3f600cdfe0 0x7f3f600b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 -- 172.21.15.70:0/3014507065 >> v2:172.21.15.98:3300/0 conn(0x7f3f600cdfe0 msgr2=0x7f3f600b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 --2- 172.21.15.70:0/3014507065 >> v2:172.21.15.98:3300/0 conn(0x7f3f600cdfe0 0x7f3f600b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 -- 172.21.15.70:0/3014507065 >> v1:172.21.15.70:6789/0 conn(0x7f3f6006b1f0 legacy=0x7f3f6015f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5ed9d640 1 -- 172.21.15.70:0/3014507065 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6015c710 con 0x7f3f6015bd00 2024-04-22T01:19:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.347+0000 7f3f5dd9b640 1 -- 172.21.15.70:0/3014507065 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212286012 0 0) 0x7f3f6015c470 con 0x7f3f6006b1f0 2024-04-22T01:19:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5ed9d640 1 --2- 172.21.15.70:0/3014507065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f6015bd00 0x7f3f6015cf10 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f3f600c4370 tx=0x7f3f5001cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110cba93f127fdec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5dd9b640 1 -- 172.21.15.70:0/3014507065 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500ab680 con 0x7f3f6015bd00 2024-04-22T01:19:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5dd9b640 1 -- 172.21.15.70:0/3014507065 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f500b0070 con 0x7f3f6015bd00 2024-04-22T01:19:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5dd9b640 1 -- 172.21.15.70:0/3014507065 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500aba40 con 0x7f3f6015bd00 2024-04-22T01:19:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- 172.21.15.70:0/3014507065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f6015bd00 msgr2=0x7f3f6015cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 --2- 172.21.15.70:0/3014507065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f6015bd00 0x7f3f6015cf10 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f3f600c4370 tx=0x7f3f5001cec0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- 172.21.15.70:0/3014507065 shutdown_connections 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 --2- 172.21.15.70:0/3014507065 >> v2:172.21.15.98:3300/0 conn(0x7f3f600cdfe0 0x7f3f600b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 --2- 172.21.15.70:0/3014507065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f6015bd00 0x7f3f6015cf10 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- 172.21.15.70:0/3014507065 >> 172.21.15.70:0/3014507065 conn(0x7f3f60099a60 msgr2=0x7f3f60099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- 172.21.15.70:0/3014507065 shutdown_connections 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- 172.21.15.70:0/3014507065 wait complete. 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 Processor -- start 2024-04-22T01:19:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- start start 2024-04-22T01:19:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f6006b1f0 0x7f3f60152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 0x7f3f600cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f6015bd00 0x7f3f600b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6006ba10 con 0x7f3f600cdfe0 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f60168cf0 con 0x7f3f6006b1f0 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f65002640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3f600adc90 con 0x7f3f6015bd00 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 0x7f3f600cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 0x7f3f600cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45850/0 (socket says 172.21.15.70:45850) 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.351+0000 7f3f5e59c640 1 -- 172.21.15.70:0/3858749866 learned_addr learned my addr 172.21.15.70:0/3858749866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5ed9d640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f6006b1f0 0x7f3f60152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5f59e640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f6015bd00 0x7f3f600b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5e59c640 1 -- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f6015bd00 msgr2=0x7f3f600b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5e59c640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f6015bd00 0x7f3f600b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5e59c640 1 -- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f6006b1f0 msgr2=0x7f3f60152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5e59c640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f6006b1f0 0x7f3f60152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5e59c640 1 -- 172.21.15.70:0/3858749866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f601534e0 con 0x7f3f600cdfe0 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5f59e640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f6015bd00 0x7f3f600b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5e59c640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 0x7f3f600cffc0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400d5d0 tx=0x7f3f5400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54004240 con 0x7f3f600cdfe0 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f540043e0 con 0x7f3f600cdfe0 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f600aff40 con 0x7f3f600cdfe0 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54003840 con 0x7f3f600cdfe0 2024-04-22T01:19:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f600b01e0 con 0x7f3f600cdfe0 2024-04-22T01:19:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f6004f510 con 0x7f3f600cdfe0 2024-04-22T01:19:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3f540039e0 con 0x7f3f600cdfe0 2024-04-22T01:19:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f477fe640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f4c0678c0 0x7f3f4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f3f54086080 con 0x7f3f600cdfe0 2024-04-22T01:19:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5ed9d640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f4c0678c0 0x7f3f4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.355+0000 7f3f5ed9d640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f4c0678c0 0x7f3f4c069d80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3f500af000 tx=0x7f3f50004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.359+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3f5401c360 con 0x7f3f600cdfe0 2024-04-22T01:19:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3f6006c060 con 0x7f3f600cdfe0 2024-04-22T01:19:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f477fe640 1 -- 172.21.15.70:0/3858749866 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v103) ==== 80+0+0 (secure 0 0 0) 0x7f3f54056a30 con 0x7f3f600cdfe0 2024-04-22T01:19:05.518 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f4c0678c0 msgr2=0x7f3f4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f4c0678c0 0x7f3f4c069d80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3f500af000 tx=0x7f3f50004740 comp rx=0 tx=0).stop 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 msgr2=0x7f3f600cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 0x7f3f600cffc0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400d5d0 tx=0x7f3f5400da90 comp rx=0 tx=0).stop 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 shutdown_connections 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f3f4c0678c0 0x7f3f4c069d80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f6015bd00 0x7f3f600b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f600cdfe0 0x7f3f600cffc0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 --2- 172.21.15.70:0/3858749866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f6006b1f0 0x7f3f60152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 >> 172.21.15.70:0/3858749866 conn(0x7f3f60099a60 msgr2=0x7f3f6015d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:05.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 shutdown_connections 2024-04-22T01:19:05.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.519+0000 7f3f65002640 1 -- 172.21.15.70:0/3858749866 wait complete. 2024-04-22T01:19:05.537 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-22T01:19:05.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:05.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T01:19:05.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[46738]: fuse finished with error 0 and tester_r 0 2024-04-22T01:19:05.547 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:05.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:05.547 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:19:05.670 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:19:05.670 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:05.670 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:05.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:19:05.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.799+0000 7fed4f607640 1 Processor -- start 2024-04-22T01:19:05.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.799+0000 7fed4f607640 1 -- start start 2024-04-22T01:19:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4f607640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed480cc150 0x7fed48071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4f607640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fed48072060 0x7fed4806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4f607640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed480cd7b0 con 0x7fed481455b0 2024-04-22T01:19:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4f607640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fed48072490 con 0x7fed480cc150 2024-04-22T01:19:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4f607640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fed480cc710 con 0x7fed48072060 2024-04-22T01:19:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed480cc150 0x7fed48071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed480cc150 0x7fed48071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45856/0 (socket says 172.21.15.70:45856) 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 -- 172.21.15.70:0/1336844774 learned_addr learned my addr 172.21.15.70:0/1336844774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4d3a3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fed481455b0 0x7fed480cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43350/0 (socket says 172.21.15.70:43350) 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4dba4640 1 --2- 172.21.15.70:0/1336844774 >> v2:172.21.15.98:3300/0 conn(0x7fed48072060 0x7fed4806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 -- 172.21.15.70:0/1336844774 >> v2:172.21.15.98:3300/0 conn(0x7fed48072060 msgr2=0x7fed4806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 --2- 172.21.15.70:0/1336844774 >> v2:172.21.15.98:3300/0 conn(0x7fed48072060 0x7fed4806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 -- 172.21.15.70:0/1336844774 >> v1:172.21.15.70:6789/0 conn(0x7fed481455b0 legacy=0x7fed480cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 -- 172.21.15.70:0/1336844774 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed480cc9f0 con 0x7fed480cc150 2024-04-22T01:19:05.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed3ffff640 1 -- 172.21.15.70:0/1336844774 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369714670 0 0) 0x7fed480cd7b0 con 0x7fed481455b0 2024-04-22T01:19:05.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed4cba2640 1 --2- 172.21.15.70:0/1336844774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48071b20 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fed4806ffb0 tx=0x7fed3801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153a4032f1b34e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed3ffff640 1 -- 172.21.15.70:0/1336844774 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380b0070 con 0x7fed480cc150 2024-04-22T01:19:05.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed3ffff640 1 -- 172.21.15.70:0/1336844774 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed380ab600 con 0x7fed480cc150 2024-04-22T01:19:05.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.803+0000 7fed3ffff640 1 -- 172.21.15.70:0/1336844774 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380ab9d0 con 0x7fed480cc150 2024-04-22T01:19:05.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1336844774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 msgr2=0x7fed48071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 --2- 172.21.15.70:0/1336844774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48071b20 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fed4806ffb0 tx=0x7fed3801c400 comp rx=0 tx=0).stop 2024-04-22T01:19:05.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1336844774 shutdown_connections 2024-04-22T01:19:05.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 --2- 172.21.15.70:0/1336844774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48071b20 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 --2- 172.21.15.70:0/1336844774 >> v2:172.21.15.98:3300/0 conn(0x7fed48072060 0x7fed4806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1336844774 >> 172.21.15.70:0/1336844774 conn(0x7fed4806b1f0 msgr2=0x7fed4806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:05.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1336844774 shutdown_connections 2024-04-22T01:19:05.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1336844774 wait complete. 2024-04-22T01:19:05.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 Processor -- start 2024-04-22T01:19:05.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- start start 2024-04-22T01:19:05.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed48072060 0x7fed4815e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed481455b0 0x7fed48138f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed4806f2e0 con 0x7fed480cc150 2024-04-22T01:19:05.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed4806f160 con 0x7fed48072060 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed48168b40 con 0x7fed481455b0 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48142240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45864/0 (socket says 172.21.15.70:45864) 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 -- 172.21.15.70:0/1346393133 learned_addr learned my addr 172.21.15.70:0/1346393133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4d3a3640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed48072060 0x7fed4815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 -- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed481455b0 msgr2=0x7fed48138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed481455b0 0x7fed48138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 -- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed48072060 msgr2=0x7fed4815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed48072060 0x7fed4815e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 -- 172.21.15.70:0/1346393133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4815e2f0 con 0x7fed480cc150 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4d3a3640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed48072060 0x7fed4815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4cba2640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48142240 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fed3801a030 tx=0x7fed3801ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380ba070 con 0x7fed480cc150 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed38004020 con 0x7fed480cc150 2024-04-22T01:19:05.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380b0040 con 0x7fed480cc150 2024-04-22T01:19:05.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4815c6f0 con 0x7fed480cc150 2024-04-22T01:19:05.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.807+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4815ccb0 con 0x7fed480cc150 2024-04-22T01:19:05.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.811+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed48068210 con 0x7fed480cc150 2024-04-22T01:19:05.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.811+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed38003620 con 0x7fed480cc150 2024-04-22T01:19:05.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.811+0000 7fed3dffb640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed2c0678f0 0x7fed2c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:05.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.811+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fed3811fe50 con 0x7fed480cc150 2024-04-22T01:19:05.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.811+0000 7fed4d3a3640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed2c0678f0 0x7fed2c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:05.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.811+0000 7fed4d3a3640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed2c0678f0 0x7fed2c069db0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fed44007fd0 tx=0x7fed44009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:05.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.815+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fed38016110 con 0x7fed480cc150 2024-04-22T01:19:05.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.975+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed4815f510 con 0x7fed480cc150 2024-04-22T01:19:05.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed3dffb640 1 -- 172.21.15.70:0/1346393133 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7fed380b5380 con 0x7fed480cc150 2024-04-22T01:19:05.978 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:05.978 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":103,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:53.308712+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:19:05.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed2c0678f0 msgr2=0x7fed2c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed2c0678f0 0x7fed2c069db0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fed44007fd0 tx=0x7fed44009040 comp rx=0 tx=0).stop 2024-04-22T01:19:05.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 msgr2=0x7fed48142240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:05.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48142240 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fed3801a030 tx=0x7fed3801ac40 comp rx=0 tx=0).stop 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 shutdown_connections 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fed2c0678f0 0x7fed2c069db0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed481455b0 0x7fed48138f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed480cc150 0x7fed48142240 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 --2- 172.21.15.70:0/1346393133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed48072060 0x7fed4815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 >> 172.21.15.70:0/1346393133 conn(0x7fed4806b1f0 msgr2=0x7fed4813bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 shutdown_connections 2024-04-22T01:19:05.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:05.979+0000 7fed4f607640 1 -- 172.21.15.70:0/1346393133 wait complete. 2024-04-22T01:19:05.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:18:49.869911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:18:50.621566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:19:05.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:19:06.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 Processor -- start 2024-04-22T01:19:06.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 -- start start 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbbac0b4450 0x7fbbac071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0720b0 0x7fbbac0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac15c290 con 0x7fbbac0cd050 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbbac072510 con 0x7fbbac0b4450 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb3877640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbbac0b3a50 con 0x7fbbac0720b0 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbbac0b4450 0x7fbbac071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbbac0b4450 0x7fbbac071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45876/0 (socket says 172.21.15.70:45876) 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 -- 172.21.15.70:0/713162975 learned_addr learned my addr 172.21.15.70:0/713162975 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb0e12640 1 --2- 172.21.15.70:0/713162975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0720b0 0x7fbbac0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 -- 172.21.15.70:0/713162975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0720b0 msgr2=0x7fbbac0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 --2- 172.21.15.70:0/713162975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0720b0 0x7fbbac0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 -- 172.21.15.70:0/713162975 >> v1:172.21.15.98:6789/0 conn(0x7fbbac0cd050 legacy=0x7fbbac15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:06.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.115+0000 7fbbb1613640 1 -- 172.21.15.70:0/713162975 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac15c570 con 0x7fbbac0b4450 2024-04-22T01:19:06.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb1613640 1 --2- 172.21.15.70:0/713162975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0b4450 0x7fbbac071b70 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009870 tx=0x7fbba801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e88f56409422a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbba3fff640 1 -- 172.21.15.70:0/713162975 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba80b1070 con 0x7fbbac0b4450 2024-04-22T01:19:06.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbba3fff640 1 -- 172.21.15.70:0/713162975 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba80ac630 con 0x7fbbac0b4450 2024-04-22T01:19:06.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbba3fff640 1 -- 172.21.15.70:0/713162975 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba80a8050 con 0x7fbbac0b4450 2024-04-22T01:19:06.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- 172.21.15.70:0/713162975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0b4450 msgr2=0x7fbbac071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 --2- 172.21.15.70:0/713162975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0b4450 0x7fbbac071b70 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009870 tx=0x7fbba801c4f0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- 172.21.15.70:0/713162975 shutdown_connections 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 --2- 172.21.15.70:0/713162975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0720b0 0x7fbbac0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 --2- 172.21.15.70:0/713162975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0b4450 0x7fbbac071b70 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- 172.21.15.70:0/713162975 >> 172.21.15.70:0/713162975 conn(0x7fbbac0b1d40 msgr2=0x7fbbac0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- 172.21.15.70:0/713162975 shutdown_connections 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- 172.21.15.70:0/713162975 wait complete. 2024-04-22T01:19:06.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 Processor -- start 2024-04-22T01:19:06.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- start start 2024-04-22T01:19:06.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbac0720b0 0x7fbbac0b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0b4450 0x7fbbac147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 0x7fbbac0af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac0b4830 con 0x7fbbac0cd050 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac0cd4c0 con 0x7fbbac0720b0 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb3877640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbbac0b49b0 con 0x7fbbac0b4450 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb1e14640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 0x7fbbac0af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb1e14640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 0x7fbbac0af6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45886/0 (socket says 172.21.15.70:45886) 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb1e14640 1 -- 172.21.15.70:0/361558073 learned_addr learned my addr 172.21.15.70:0/361558073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb0e12640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0b4450 0x7fbbac147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.119+0000 7fbbb1613640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbac0720b0 0x7fbbac0b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1e14640 1 -- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0b4450 msgr2=0x7fbbac147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1e14640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0b4450 0x7fbbac147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1e14640 1 -- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbac0720b0 msgr2=0x7fbbac0b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1e14640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbac0720b0 0x7fbbac0b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1e14640 1 -- 172.21.15.70:0/361558073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac148120 con 0x7fbbac0cd050 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1e14640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 0x7fbbac0af6b0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fbba400c7b0 tx=0x7fbba400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4004b70 con 0x7fbbac0cd050 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba4004d10 con 0x7fbbac0cd050 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbac148400 con 0x7fbbac0cd050 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba400aa90 con 0x7fbbac0cd050 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbac1489c0 con 0x7fbbac0cd050 2024-04-22T01:19:06.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbac068210 con 0x7fbbac0cd050 2024-04-22T01:19:06.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbba400ac30 con 0x7fbbac0cd050 2024-04-22T01:19:06.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbba1ffb640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb900678c0 0x7fbb90069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1613640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb900678c0 0x7fbb90069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fbba4086160 con 0x7fbbac0cd050 2024-04-22T01:19:06.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.123+0000 7fbbb1613640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb900678c0 0x7fbb90069d80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbba80ae000 tx=0x7fbba8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.127+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbba4020070 con 0x7fbbac0cd050 2024-04-22T01:19:06.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.287+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbbac0d01e0 con 0x7fbbac0cd050 2024-04-22T01:19:06.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.287+0000 7fbba1ffb640 1 -- 172.21.15.70:0/361558073 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7fbba4056a70 con 0x7fbbac0cd050 2024-04-22T01:19:06.287 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb900678c0 msgr2=0x7fbb90069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb900678c0 0x7fbb90069d80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbba80ae000 tx=0x7fbba8008040 comp rx=0 tx=0).stop 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 msgr2=0x7fbbac0af6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 0x7fbbac0af6b0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fbba400c7b0 tx=0x7fbba400cc70 comp rx=0 tx=0).stop 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 shutdown_connections 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbb900678c0 0x7fbb90069d80 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbac0cd050 0x7fbbac0af6b0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbac0b4450 0x7fbbac147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 --2- 172.21.15.70:0/361558073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbac0720b0 0x7fbbac0b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 >> 172.21.15.70:0/361558073 conn(0x7fbbac0b1d40 msgr2=0x7fbbac15ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:06.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 shutdown_connections 2024-04-22T01:19:06.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.291+0000 7fbbb3877640 1 -- 172.21.15.70:0/361558073 wait complete. 2024-04-22T01:19:06.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:19:06.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.415+0000 7f5db9d39640 1 Processor -- start 2024-04-22T01:19:06.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.415+0000 7f5db9d39640 1 -- start start 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db9d39640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5db415e050 0x7f5db40b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db9d39640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5db415e430 0x7f5db406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db9d39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db40af440 con 0x7f5db40670f0 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db9d39640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40af6e0 con 0x7f5db415e050 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db9d39640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40af9c0 con 0x7f5db415e430 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5db415e050 0x7f5db40b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5db415e050 0x7f5db40b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55296/0 (socket says 172.21.15.70:55296) 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 -- 172.21.15.70:0/2702742674 learned_addr learned my addr 172.21.15.70:0/2702742674 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db2ffd640 1 --2- 172.21.15.70:0/2702742674 >> v2:172.21.15.98:3300/0 conn(0x7f5db415e430 0x7f5db406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 -- 172.21.15.70:0/2702742674 >> v2:172.21.15.98:3300/0 conn(0x7f5db415e430 msgr2=0x7f5db406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:06.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 --2- 172.21.15.70:0/2702742674 >> v2:172.21.15.98:3300/0 conn(0x7f5db415e430 0x7f5db406e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 -- 172.21.15.70:0/2702742674 >> v1:172.21.15.98:6789/0 conn(0x7f5db40670f0 legacy=0x7f5db415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:06.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 -- 172.21.15.70:0/2702742674 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db40b6150 con 0x7f5db415e050 2024-04-22T01:19:06.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db37fe640 1 --2- 172.21.15.70:0/2702742674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db40b5a00 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f5db4073210 tx=0x7f5da401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f4a8688ae78679 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.419+0000 7f5db27fc640 1 -- 172.21.15.70:0/2702742674 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da40b0070 con 0x7f5db415e050 2024-04-22T01:19:06.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db27fc640 1 -- 172.21.15.70:0/2702742674 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da40ab600 con 0x7f5db415e050 2024-04-22T01:19:06.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db27fc640 1 -- 172.21.15.70:0/2702742674 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da40ab9d0 con 0x7f5db415e050 2024-04-22T01:19:06.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/2702742674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 msgr2=0x7f5db40b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 --2- 172.21.15.70:0/2702742674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db40b5a00 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f5db4073210 tx=0x7f5da401c340 comp rx=0 tx=0).stop 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/2702742674 shutdown_connections 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 --2- 172.21.15.70:0/2702742674 >> v2:172.21.15.98:3300/0 conn(0x7f5db415e430 0x7f5db406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 --2- 172.21.15.70:0/2702742674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db40b5a00 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/2702742674 >> 172.21.15.70:0/2702742674 conn(0x7f5db4099a60 msgr2=0x7f5db40b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/2702742674 shutdown_connections 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/2702742674 wait complete. 2024-04-22T01:19:06.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 Processor -- start 2024-04-22T01:19:06.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- start start 2024-04-22T01:19:06.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5db40670f0 0x7f5db414ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db41facd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db415e430 0x7f5db40ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4167fd0 con 0x7f5db415e050 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4167e50 con 0x7f5db415e430 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5db4168150 con 0x7f5db40670f0 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db41facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db41facd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55312/0 (socket says 172.21.15.70:55312) 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 -- 172.21.15.70:0/3048484351 learned_addr learned my addr 172.21.15.70:0/3048484351 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db37fe640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5db40670f0 0x7f5db414ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 -- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5db40670f0 msgr2=0x7f5db414ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5db40670f0 0x7f5db414ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 -- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db415e430 msgr2=0x7f5db40ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db3fff640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db415e430 0x7f5db40ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db415e430 0x7f5db40ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 -- 172.21.15.70:0/3048484351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db41fbb90 con 0x7f5db415e050 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db2ffd640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db41facd0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f5da800d800 tx=0x7f5da800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da800b6c0 con 0x7f5db415e050 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da8019070 con 0x7f5db415e050 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da800ba80 con 0x7f5db415e050 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db41fb5d0 con 0x7f5db415e050 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.423+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db40b0180 con 0x7f5db415e050 2024-04-22T01:19:06.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.427+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db4068920 con 0x7f5db415e050 2024-04-22T01:19:06.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.427+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5da800bc20 con 0x7f5db415e050 2024-04-22T01:19:06.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.427+0000 7f5d9bfff640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5da00678c0 0x7f5da0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.427+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f5da8086240 con 0x7f5db415e050 2024-04-22T01:19:06.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.427+0000 7f5db37fe640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5da00678c0 0x7f5da0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.427+0000 7f5db37fe640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5da00678c0 0x7f5da0069d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5da40ad000 tx=0x7f5da4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.431+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f5da8056bf0 con 0x7f5db415e050 2024-04-22T01:19:06.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.575+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5db4072f90 con 0x7f5db415e050 2024-04-22T01:19:06.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5d9bfff640 1 -- 172.21.15.70:0/3048484351 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f5da80564f0 con 0x7f5db415e050 2024-04-22T01:19:06.577 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5da00678c0 msgr2=0x7f5da0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5da00678c0 0x7f5da0069d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5da40ad000 tx=0x7f5da4008040 comp rx=0 tx=0).stop 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 msgr2=0x7f5db41facd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db41facd0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f5da800d800 tx=0x7f5da800dcc0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 shutdown_connections 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f5da00678c0 0x7f5da0069d80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db415e430 0x7f5db40ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5db415e050 0x7f5db41facd0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 --2- 172.21.15.70:0/3048484351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5db40670f0 0x7f5db414ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 >> 172.21.15.70:0/3048484351 conn(0x7f5db4099a60 msgr2=0x7f5db415d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 shutdown_connections 2024-04-22T01:19:06.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.579+0000 7f5db9d39640 1 -- 172.21.15.70:0/3048484351 wait complete. 2024-04-22T01:19:06.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:19:06.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.715+0000 7f15b55f7640 1 Processor -- start 2024-04-22T01:19:06.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.715+0000 7f15b55f7640 1 -- start start 2024-04-22T01:19:06.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15b55f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15b00b4450 0x7f15b0071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15b55f7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f15b00720b0 0x7f15b00ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15b55f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b015c290 con 0x7f15b00cd050 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15b55f7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f15b00b3a50 con 0x7f15b00720b0 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15b55f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f15b015c570 con 0x7f15b00b4450 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f15b00720b0 0x7f15b00ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f15b00720b0 0x7f15b00ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55318/0 (socket says 172.21.15.70:55318) 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 -- 172.21.15.70:0/2971827060 learned_addr learned my addr 172.21.15.70:0/2971827060 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15aeffd640 1 --2- 172.21.15.70:0/2971827060 >> v2:172.21.15.98:3300/0 conn(0x7f15b00b4450 0x7f15b0071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 -- 172.21.15.70:0/2971827060 >> v2:172.21.15.98:3300/0 conn(0x7f15b00b4450 msgr2=0x7f15b0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 --2- 172.21.15.70:0/2971827060 >> v2:172.21.15.98:3300/0 conn(0x7f15b00b4450 0x7f15b0071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 -- 172.21.15.70:0/2971827060 >> v1:172.21.15.98:6789/0 conn(0x7f15b00cd050 legacy=0x7f15b015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 -- 172.21.15.70:0/2971827060 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b015c850 con 0x7f15b00720b0 2024-04-22T01:19:06.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15ae7fc640 1 --2- 172.21.15.70:0/2971827060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b00ccb10 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f15b006fce0 tx=0x7f15a401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b0d75c2353a2ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15adffb640 1 -- 172.21.15.70:0/2971827060 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a40b0070 con 0x7f15b00720b0 2024-04-22T01:19:06.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15adffb640 1 -- 172.21.15.70:0/2971827060 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a40ab600 con 0x7f15b00720b0 2024-04-22T01:19:06.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.719+0000 7f15adffb640 1 -- 172.21.15.70:0/2971827060 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a40ab9d0 con 0x7f15b00720b0 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/2971827060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 msgr2=0x7f15b00ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 --2- 172.21.15.70:0/2971827060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b00ccb10 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f15b006fce0 tx=0x7f15a401c4e0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/2971827060 shutdown_connections 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 --2- 172.21.15.70:0/2971827060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b00ccb10 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 --2- 172.21.15.70:0/2971827060 >> v2:172.21.15.98:3300/0 conn(0x7f15b00b4450 0x7f15b0071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/2971827060 >> 172.21.15.70:0/2971827060 conn(0x7f15b00b1d40 msgr2=0x7f15b00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/2971827060 shutdown_connections 2024-04-22T01:19:06.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/2971827060 wait complete. 2024-04-22T01:19:06.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 Processor -- start 2024-04-22T01:19:06.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- start start 2024-04-22T01:19:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b0153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b00b4450 0x7f15b0144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15b00cd050 0x7f15b015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f15b00b49b0 con 0x7f15b00720b0 2024-04-22T01:19:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15b00b4830 con 0x7f15b00b4450 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f15b00cd800 con 0x7f15b00cd050 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b0153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b0153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55324/0 (socket says 172.21.15.70:55324) 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 -- 172.21.15.70:0/4106852832 learned_addr learned my addr 172.21.15.70:0/4106852832 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15ae7fc640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b00b4450 0x7f15b0144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15af7fe640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15b00cd050 0x7f15b015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 -- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15b00cd050 msgr2=0x7f15b015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15b00cd050 0x7f15b015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 -- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b00b4450 msgr2=0x7f15b0144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b00b4450 0x7f15b0144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 -- 172.21.15.70:0/4106852832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b0152210 con 0x7f15b00720b0 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15ae7fc640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b00b4450 0x7f15b0144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15af7fe640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15b00cd050 0x7f15b015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15aeffd640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b0153870 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f15a000d830 tx=0x7f15a000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a000b660 con 0x7f15b00720b0 2024-04-22T01:19:06.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a0019070 con 0x7f15b00720b0 2024-04-22T01:19:06.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b006f820 con 0x7f15b00720b0 2024-04-22T01:19:06.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a000ba00 con 0x7f15b00720b0 2024-04-22T01:19:06.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.723+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b006ed40 con 0x7f15b00720b0 2024-04-22T01:19:06.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.727+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b0068210 con 0x7f15b00720b0 2024-04-22T01:19:06.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.727+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f15a001e460 con 0x7f15b00720b0 2024-04-22T01:19:06.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.727+0000 7f15977fe640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f159c0678c0 0x7f159c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:06.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.727+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f15a0086ff0 con 0x7f15b00720b0 2024-04-22T01:19:06.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.727+0000 7f15ae7fc640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f159c0678c0 0x7f159c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:06.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.727+0000 7f15ae7fc640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f159c0678c0 0x7f159c069d80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ad000 tx=0x7f15a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:06.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.731+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f15a0057980 con 0x7f15b00720b0 2024-04-22T01:19:06.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.891+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f15b00d01e0 con 0x7f15b00720b0 2024-04-22T01:19:06.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.891+0000 7f15977fe640 1 -- 172.21.15.70:0/4106852832 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f15a0057280 con 0x7f15b00720b0 2024-04-22T01:19:06.891 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:19:06.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f159c0678c0 msgr2=0x7f159c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f159c0678c0 0x7f159c069d80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ad000 tx=0x7f15a4008040 comp rx=0 tx=0).stop 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 msgr2=0x7f15b0153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b0153870 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f15a000d830 tx=0x7f15a000dcf0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 shutdown_connections 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f159c0678c0 0x7f159c069d80 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15b00cd050 0x7f15b015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b00b4450 0x7f15b0144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 --2- 172.21.15.70:0/4106852832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15b00720b0 0x7f15b0153870 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 >> 172.21.15.70:0/4106852832 conn(0x7f15b00b1d40 msgr2=0x7f15b015dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 shutdown_connections 2024-04-22T01:19:06.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:06.895+0000 7f15b55f7640 1 -- 172.21.15.70:0/4106852832 wait complete. 2024-04-22T01:19:06.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:19:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdafabca640 1 Processor -- start 2024-04-22T01:19:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdafabca640 1 -- start start 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdafabca640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdaf4072380 0x7fdaf40727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdafabca640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaf40b3540 con 0x7fdaf413e780 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdafabca640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaf413dca0 con 0x7fdaf415a4f0 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdafabca640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdaf4072d20 con 0x7fdaf4072380 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdaf413e780 0x7fdaf413ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58544/0 (socket says 172.21.15.70:58544) 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 -- 172.21.15.70:0/2360439391 learned_addr learned my addr 172.21.15.70:0/2360439391 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 58173274 0 0) 0x7fdaf40b3540 con 0x7fdaf413e780 2024-04-22T01:19:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf3fff640 1 --2- 172.21.15.70:0/2360439391 >> v2:172.21.15.98:3300/0 conn(0x7fdaf4072380 0x7fdaf40727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdae4003530 con 0x7fdaf413e780 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 691829218 0 0) 0x7fdae4003530 con 0x7fdaf413e780 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 >> v2:172.21.15.98:3300/0 conn(0x7fdaf4072380 msgr2=0x7fdaf40727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 --2- 172.21.15.70:0/2360439391 >> v2:172.21.15.98:3300/0 conn(0x7fdaf4072380 0x7fdaf40727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 >> v1:172.21.15.98:6789/0 conn(0x7fdaf415a4f0 legacy=0x7fdaf40b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf413df80 con 0x7fdaf413e780 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fdaec001a00 con 0x7fdaf413e780 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 >> v1:172.21.15.70:6789/0 conn(0x7fdaf413e780 legacy=0x7fdaf413ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae4003c00 0x7fdae4004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdae40088b0 0x7fdae400ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdae400b540 0x7fdae400da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdae400e2b0 con 0x7fdae400b540 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdae400e460 con 0x7fdae40088b0 2024-04-22T01:19:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdae400e660 con 0x7fdae4003c00 2024-04-22T01:19:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdae400b540 0x7fdae400da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf8966640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae4003c00 0x7fdae4004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf3fff640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdae40088b0 0x7fdae400ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 -- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae4003c00 msgr2=0x7fdae4004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae4003c00 0x7fdae4004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 -- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdae40088b0 msgr2=0x7fdae400ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdae40088b0 0x7fdae400ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf9167640 1 -- 172.21.15.70:0/2360439391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae4003530 con 0x7fdae400b540 2024-04-22T01:19:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.019+0000 7fdaf3fff640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdae40088b0 0x7fdae400ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdaf9167640 1 -- 172.21.15.70:0/2360439391 reap_dead start 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdaf9167640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdae400b540 0x7fdae400da00 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00a860 tx=0x7fdaec01d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5185489ac76074 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00a480 con 0x7fdae400b540 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaec0ad630 con 0x7fdae400b540 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdaf37fe640 1 -- 172.21.15.70:0/2360439391 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0ada20 con 0x7fdae400b540 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- 172.21.15.70:0/2360439391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdae400b540 msgr2=0x7fdae400da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdae400b540 0x7fdae400da00 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00a860 tx=0x7fdaec01d380 comp rx=0 tx=0).stop 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- 172.21.15.70:0/2360439391 shutdown_connections 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 --2- 172.21.15.70:0/2360439391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdae400b540 0x7fdae400da00 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- 172.21.15.70:0/2360439391 >> 172.21.15.70:0/2360439391 conn(0x7fdaf40cf9e0 msgr2=0x7fdaf40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- 172.21.15.70:0/2360439391 shutdown_connections 2024-04-22T01:19:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- 172.21.15.70:0/2360439391 wait complete. 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 Processor -- start 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- start start 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae400b540 0x7fdaf4153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 0x7fdaf415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf406f140 0x7fdaf406f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf40b3820 con 0x7fdaf415c7a0 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf413e2f0 con 0x7fdaf406f140 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.023+0000 7fdafabca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdaf40727e0 con 0x7fdae400b540 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 0x7fdaf415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 0x7fdaf415cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55342/0 (socket says 172.21.15.70:55342) 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 -- 172.21.15.70:0/2725512715 learned_addr learned my addr 172.21.15.70:0/2725512715 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf8966640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae400b540 0x7fdaf4153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf9167640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf406f140 0x7fdaf406f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 -- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae400b540 msgr2=0x7fdaf4153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae400b540 0x7fdaf4153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 -- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf406f140 msgr2=0x7fdaf406f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf406f140 0x7fdaf406f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 -- 172.21.15.70:0/2725512715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf41289b0 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf9167640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf406f140 0x7fdaf406f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf3fff640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 0x7fdaf415cc20 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fdae80108f0 tx=0x7fdae8010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae800f040 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdae8004220 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf4153740 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae800e900 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf4151bc0 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaf406fb00 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdae800eaa0 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf17fa640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdad80678f0 0x7fdad8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fdae8085c60 con 0x7fdaf415c7a0 2024-04-22T01:19:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf8966640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdad80678f0 0x7fdad8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.027+0000 7fdaf8966640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdad80678f0 0x7fdad8069db0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fdaf412a9a0 tx=0x7fdae0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.031+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fdae8018030 con 0x7fdaf415c7a0 2024-04-22T01:19:07.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdaf404f510 con 0x7fdaf415c7a0 2024-04-22T01:19:07.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdaf17fa640 1 -- 172.21.15.70:0/2725512715 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v103) ==== 80+0+0 (secure 0 0 0) 0x7fdae8056610 con 0x7fdaf415c7a0 2024-04-22T01:19:07.198 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdad80678f0 msgr2=0x7fdad8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdad80678f0 0x7fdad8069db0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fdaf412a9a0 tx=0x7fdae0009040 comp rx=0 tx=0).stop 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 msgr2=0x7fdaf415cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 0x7fdaf415cc20 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fdae80108f0 tx=0x7fdae8010db0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 shutdown_connections 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fdad80678f0 0x7fdad8069db0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf406f140 0x7fdaf406f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaf415c7a0 0x7fdaf415cc20 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 --2- 172.21.15.70:0/2725512715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdae400b540 0x7fdaf4153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.199+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 >> 172.21.15.70:0/2725512715 conn(0x7fdaf40cf9e0 msgr2=0x7fdaf40ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.203+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 shutdown_connections 2024-04-22T01:19:07.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.203+0000 7fdafabca640 1 -- 172.21.15.70:0/2725512715 wait complete. 2024-04-22T01:19:07.217 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-22T01:19:07.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:07.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-22T01:19:07.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi098.stderr:ceph-fuse[34806]: fuse finished with error 0 and tester_r 0 2024-04-22T01:19:07.232 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:07.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:07.233 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-22T01:19:07.355 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:19:07.355 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:19:07.355 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:07.361 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:19:07.361 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:07.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:07.361 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-22T01:19:07.409 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-22T01:19:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:07.410 INFO:tasks.cephfs.mount:Terminating background process 2024-04-22T01:19:07.410 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:07.411 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:07.416 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:19:07.417 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:07.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:07.417 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-22T01:19:07.465 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-22T01:19:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:07.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:19:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 Processor -- start 2024-04-22T01:19:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 -- start start 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f836415dfb0 0x7f83640b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 0x7f83640ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f836415e390 con 0x7f83640cea80 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f83640cf420 con 0x7f836415dfb0 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f836b2e7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8364161e00 con 0x7f83640d3120 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f8368882640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f83640cea80 0x7f83640b5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58558/0 (socket says 172.21.15.70:58558) 2024-04-22T01:19:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f8368882640 1 -- 172.21.15.70:0/3149659339 learned_addr learned my addr 172.21.15.70:0/3149659339 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f8369884640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 0x7f83640ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f8369083640 1 --2- 172.21.15.70:0/3149659339 >> v2:172.21.15.98:3300/0 conn(0x7f836415dfb0 0x7f83640b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955998309 0 0) 0x7f836415e390 con 0x7f83640cea80 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8350003530 con 0x7f83640cea80 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3764735080 0 0) 0x7f8350003530 con 0x7f83640cea80 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 msgr2=0x7f83640ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 0x7f83640ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 >> v2:172.21.15.98:3300/0 conn(0x7f836415dfb0 msgr2=0x7f83640b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 --2- 172.21.15.70:0/3149659339 >> v2:172.21.15.98:3300/0 conn(0x7f836415dfb0 0x7f83640b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83641620e0 con 0x7f83640cea80 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f83540016a0 con 0x7f83640cea80 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 >> v1:172.21.15.70:6789/0 conn(0x7f83640cea80 legacy=0x7f83640b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83500039c0 0x7f8350003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 0x7f8350004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f83500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8364161e00 con 0x7f8350004d50 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f835000b9d0 con 0x7f83500039c0 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.587+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f835000bb50 con 0x7f83640d3120 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f83500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8368882640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 0x7f8350004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369083640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83500039c0 0x7f8350003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 -- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 msgr2=0x7f8350004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83640d3120 0x7f8350004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 -- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83500039c0 msgr2=0x7f8350003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83500039c0 0x7f8350003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 -- 172.21.15.70:0/3149659339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8350003530 con 0x7f8350004d50 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f83500051d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f835c009fc0 tx=0x7f835c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2356859a7d6155b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c0b2070 con 0x7f8350004d50 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f835c0a5c50 con 0x7f8350004d50 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f835bfff640 1 -- 172.21.15.70:0/3149659339 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c009c10 con 0x7f8350004d50 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f836b2e7640 1 -- 172.21.15.70:0/3149659339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 msgr2=0x7f83500051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f836b2e7640 1 --2- 172.21.15.70:0/3149659339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f83500051d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f835c009fc0 tx=0x7f835c01cc40 comp rx=0 tx=0).stop 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f8369884640 1 -- 172.21.15.70:0/3149659339 reap_dead start 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f836b2e7640 1 -- 172.21.15.70:0/3149659339 shutdown_connections 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f836b2e7640 1 -- 172.21.15.70:0/3149659339 >> 172.21.15.70:0/3149659339 conn(0x7f836406c9b0 msgr2=0x7f836406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f836b2e7640 1 -- 172.21.15.70:0/3149659339 shutdown_connections 2024-04-22T01:19:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.591+0000 7f836b2e7640 1 -- 172.21.15.70:0/3149659339 wait complete. 2024-04-22T01:19:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 Processor -- start 2024-04-22T01:19:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- start start 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f8364159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83640b75a0 0x7f83640b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f836415f360 0x7f836415f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83640b9b10 con 0x7f8350004d50 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83641626a0 con 0x7f83640b75a0 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8364161a90 con 0x7f836415f360 2024-04-22T01:19:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f8364159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f8364159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55370/0 (socket says 172.21.15.70:55370) 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 -- 172.21.15.70:0/2873401504 learned_addr learned my addr 172.21.15.70:0/2873401504 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369884640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f836415f360 0x7f836415f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8368882640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83640b75a0 0x7f83640b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 -- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f836415f360 msgr2=0x7f836415f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f836415f360 0x7f836415f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 -- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83640b75a0 msgr2=0x7f83640b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83640b75a0 0x7f83640b7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 -- 172.21.15.70:0/2873401504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8364160040 con 0x7f8350004d50 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8368882640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83640b75a0 0x7f83640b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8369083640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f8364159750 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f8360011610 tx=0x7f8360011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836000f8f0 con 0x7f8350004d50 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f836000fa90 con 0x7f8350004d50 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8364160320 con 0x7f8350004d50 2024-04-22T01:19:07.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836000fe40 con 0x7f8350004d50 2024-04-22T01:19:07.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83641608e0 con 0x7f8350004d50 2024-04-22T01:19:07.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8364068210 con 0x7f8350004d50 2024-04-22T01:19:07.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8360015460 con 0x7f8350004d50 2024-04-22T01:19:07.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.595+0000 7f8359ffb640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678f0 0x7f8348069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.599+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8360086810 con 0x7f8350004d50 2024-04-22T01:19:07.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.599+0000 7f8368882640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678f0 0x7f8348069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.599+0000 7f8368882640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678f0 0x7f8348069db0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8354001060 tx=0x7f8354000ff0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.603+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f8360019030 con 0x7f8350004d50 2024-04-22T01:19:07.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83640b6970 con 0x7f8350004d50 2024-04-22T01:19:07.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f8359ffb640 1 -- 172.21.15.70:0/2873401504 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f83600571c0 con 0x7f8350004d50 2024-04-22T01:19:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:07.782 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:51.626962+0000","modified":"2024-04-22T01:18:53.648845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7629":{"gid":7629,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3578750329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3578750329},{"type":"v1","addr":"172.21.15.70:6837","nonce":3578750329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7629,"qdb_cluster":[7629]},"id":7}]} 2024-04-22T01:19:07.782 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 50 2024-04-22T01:19:07.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678f0 msgr2=0x7f8348069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678f0 0x7f8348069db0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8354001060 tx=0x7f8354000ff0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 msgr2=0x7f8364159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f8364159750 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f8360011610 tx=0x7f8360011ad0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 shutdown_connections 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f83480678f0 0x7f8348069db0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f836415f360 0x7f836415f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83640b75a0 0x7f83640b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 --2- 172.21.15.70:0/2873401504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8350004d50 0x7f8364159750 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.783+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 >> 172.21.15.70:0/2873401504 conn(0x7f836406c9b0 msgr2=0x7f83640d3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.787+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 shutdown_connections 2024-04-22T01:19:07.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.787+0000 7f836b2e7640 1 -- 172.21.15.70:0/2873401504 wait complete. 2024-04-22T01:19:07.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:19:07.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.911+0000 7f10a5cad640 1 Processor -- start 2024-04-22T01:19:07.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.911+0000 7f10a5cad640 1 -- start start 2024-04-22T01:19:07.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f10a5cad640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10a015bc70 0x7f10a015f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f10a5cad640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f10a00cd740 0x7f10a006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f10a5cad640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10a006bb50 con 0x7f10a006b730 2024-04-22T01:19:07.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f10a5cad640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f10a00cfd10 con 0x7f10a00cd740 2024-04-22T01:19:07.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f10a5cad640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f10a015fd10 con 0x7f10a015bc70 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f10a00cd740 0x7f10a006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f10a00cd740 0x7f10a006b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55378/0 (socket says 172.21.15.70:55378) 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f10a006b730 0x7f10a00ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58572/0 (socket says 172.21.15.70:58572) 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 -- 172.21.15.70:0/2692649597 learned_addr learned my addr 172.21.15.70:0/2692649597 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109f7fe640 1 --2- 172.21.15.70:0/2692649597 >> v2:172.21.15.98:3300/0 conn(0x7f10a015bc70 0x7f10a015f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109e7fc640 1 -- 172.21.15.70:0/2692649597 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719923174 0 0) 0x7f10a006bb50 con 0x7f10a006b730 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 -- 172.21.15.70:0/2692649597 >> v2:172.21.15.98:3300/0 conn(0x7f10a015bc70 msgr2=0x7f10a015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 --2- 172.21.15.70:0/2692649597 >> v2:172.21.15.98:3300/0 conn(0x7f10a015bc70 0x7f10a015f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 -- 172.21.15.70:0/2692649597 >> v1:172.21.15.70:6789/0 conn(0x7f10a006b730 legacy=0x7f10a00ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 -- 172.21.15.70:0/2692649597 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a015c050 con 0x7f10a00cd740 2024-04-22T01:19:07.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109effd640 1 --2- 172.21.15.70:0/2692649597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a00cd740 0x7f10a006b1f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f10a00ce550 tx=0x7f109401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3784f597961c408a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109e7fc640 1 -- 172.21.15.70:0/2692649597 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10940aa650 con 0x7f10a00cd740 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.915+0000 7f109e7fc640 1 -- 172.21.15.70:0/2692649597 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10940af070 con 0x7f10a00cd740 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109e7fc640 1 -- 172.21.15.70:0/2692649597 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10940aaa10 con 0x7f10a00cd740 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/2692649597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a00cd740 msgr2=0x7f10a006b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 --2- 172.21.15.70:0/2692649597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a00cd740 0x7f10a006b1f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f10a00ce550 tx=0x7f109401c4f0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/2692649597 shutdown_connections 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 --2- 172.21.15.70:0/2692649597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a00cd740 0x7f10a006b1f0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 --2- 172.21.15.70:0/2692649597 >> v2:172.21.15.98:3300/0 conn(0x7f10a015bc70 0x7f10a015f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/2692649597 >> 172.21.15.70:0/2692649597 conn(0x7f10a0099a60 msgr2=0x7f10a0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/2692649597 shutdown_connections 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/2692649597 wait complete. 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 Processor -- start 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- start start 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10a006b730 0x7f10a0154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a00cd740 0x7f10a01f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 0x7f10a00b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0168b70 con 0x7f10a015bc70 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10a00cff60 con 0x7f10a00cd740 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f10a0168cf0 con 0x7f10a006b730 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 0x7f10a00b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 0x7f10a00b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55388/0 (socket says 172.21.15.70:55388) 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 -- 172.21.15.70:0/22063517 learned_addr learned my addr 172.21.15.70:0/22063517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109f7fe640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10a006b730 0x7f10a0154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109effd640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a00cd740 0x7f10a01f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 -- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10a006b730 msgr2=0x7f10a0154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10a006b730 0x7f10a0154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 -- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a00cd740 msgr2=0x7f10a01f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a00cd740 0x7f10a01f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 -- 172.21.15.70:0/22063517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a01f8860 con 0x7f10a015bc70 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f109ffff640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 0x7f10a00b01e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f109800c7d0 tx=0x7f109800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098018070 con 0x7f10a015bc70 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f109800a640 con 0x7f10a015bc70 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a01f8b40 con 0x7f10a015bc70 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109800aa70 con 0x7f10a015bc70 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.919+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a01548c0 con 0x7f10a015bc70 2024-04-22T01:19:07.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.923+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10a0068920 con 0x7f10a015bc70 2024-04-22T01:19:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.923+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f109801b070 con 0x7f10a015bc70 2024-04-22T01:19:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.923+0000 7f1087fff640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f108c0678c0 0x7f108c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.923+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f1098085440 con 0x7f10a015bc70 2024-04-22T01:19:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.923+0000 7f109f7fe640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f108c0678c0 0x7f108c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:07.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.923+0000 7f109f7fe640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f108c0678c0 0x7f108c069d80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f109000b930 tx=0x7f1090008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:07.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:07.927+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1098055df0 con 0x7f10a015bc70 2024-04-22T01:19:08.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.075+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10a00cd520 con 0x7f10a015bc70 2024-04-22T01:19:08.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.075+0000 7f1087fff640 1 -- 172.21.15.70:0/22063517 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7f10980556f0 con 0x7f10a015bc70 2024-04-22T01:19:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:08.075 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":103,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:53.308712+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:19:08.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f108c0678c0 msgr2=0x7f108c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f108c0678c0 0x7f108c069d80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f109000b930 tx=0x7f1090008040 comp rx=0 tx=0).stop 2024-04-22T01:19:08.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 msgr2=0x7f10a00b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 0x7f10a00b01e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f109800c7d0 tx=0x7f109800cc90 comp rx=0 tx=0).stop 2024-04-22T01:19:08.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 shutdown_connections 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f108c0678c0 0x7f108c069d80 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10a015bc70 0x7f10a00b01e0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10a00cd740 0x7f10a01f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 --2- 172.21.15.70:0/22063517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10a006b730 0x7f10a0154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 >> 172.21.15.70:0/22063517 conn(0x7f10a0099a60 msgr2=0x7f10a015ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 shutdown_connections 2024-04-22T01:19:08.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.079+0000 7f10a5cad640 1 -- 172.21.15.70:0/22063517 wait complete. 2024-04-22T01:19:08.092 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:14:34.662032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:18:49.869911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:18:50.621566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:19:08.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:19:08.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 Processor -- start 2024-04-22T01:19:08.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 -- start start 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31100b30e0 0x7f31100b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31100b5da0 0x7f31100b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f311006d740 con 0x7f311015e1e0 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f311006d2b0 con 0x7f31100b30e0 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f3117b5f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f311006c980 con 0x7f31100b5da0 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31100b30e0 0x7f31100b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31100b30e0 0x7f31100b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55402/0 (socket says 172.21.15.70:55402) 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31158fb640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f311015e1e0 0x7f31100d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58584/0 (socket says 172.21.15.70:58584) 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 -- 172.21.15.70:0/2781334322 learned_addr learned my addr 172.21.15.70:0/2781334322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31160fc640 1 --2- 172.21.15.70:0/2781334322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31100b5da0 0x7f31100b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 -- 172.21.15.70:0/2781334322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31100b5da0 msgr2=0x7f31100b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 --2- 172.21.15.70:0/2781334322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31100b5da0 0x7f31100b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 -- 172.21.15.70:0/2781334322 >> v1:172.21.15.70:6789/0 conn(0x7f311015e1e0 legacy=0x7f31100d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31150fa640 1 -- 172.21.15.70:0/2781334322 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311006cc60 con 0x7f31100b30e0 2024-04-22T01:19:08.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.207+0000 7f31148f9640 1 -- 172.21.15.70:0/2781334322 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236548945 0 0) 0x7f311006d740 con 0x7f311015e1e0 2024-04-22T01:19:08.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31150fa640 1 --2- 172.21.15.70:0/2781334322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100b5860 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f3100009870 tx=0x7f310001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9d6f692989084e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31148f9640 1 -- 172.21.15.70:0/2781334322 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000b2070 con 0x7f31100b30e0 2024-04-22T01:19:08.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31148f9640 1 -- 172.21.15.70:0/2781334322 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31000a9070 con 0x7f31100b30e0 2024-04-22T01:19:08.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31148f9640 1 -- 172.21.15.70:0/2781334322 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000ad880 con 0x7f31100b30e0 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/2781334322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 msgr2=0x7f31100b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 --2- 172.21.15.70:0/2781334322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100b5860 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f3100009870 tx=0x7f310001c560 comp rx=0 tx=0).stop 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/2781334322 shutdown_connections 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 --2- 172.21.15.70:0/2781334322 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31100b5da0 0x7f31100b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 --2- 172.21.15.70:0/2781334322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100b5860 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/2781334322 >> 172.21.15.70:0/2781334322 conn(0x7f3110073110 msgr2=0x7f31100d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/2781334322 shutdown_connections 2024-04-22T01:19:08.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/2781334322 wait complete. 2024-04-22T01:19:08.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 Processor -- start 2024-04-22T01:19:08.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- start start 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100c7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b5da0 0x7f311014d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f311015e1e0 0x7f31100cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31100b7f40 con 0x7f31100b30e0 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31100b7dc0 con 0x7f31100b5da0 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31100b61b0 con 0x7f311015e1e0 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100c7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55416/0 (socket says 172.21.15.70:55416) 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 -- 172.21.15.70:0/4285740953 learned_addr learned my addr 172.21.15.70:0/4285740953 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:08.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 -- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f311015e1e0 msgr2=0x7f31100cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31160fc640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f311015e1e0 0x7f31100cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31150fa640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b5da0 0x7f311014d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f311015e1e0 0x7f31100cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 -- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b5da0 msgr2=0x7f311014d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b5da0 0x7f311014d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 -- 172.21.15.70:0/4285740953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31100c76c0 con 0x7f31100b30e0 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31158fb640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100c7180 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f310c0011f0 tx=0x7f310c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c018070 con 0x7f31100b30e0 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f310c0074e0 con 0x7f31100b30e0 2024-04-22T01:19:08.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c004a10 con 0x7f31100b30e0 2024-04-22T01:19:08.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311014d7b0 con 0x7f31100b30e0 2024-04-22T01:19:08.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.211+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31101607b0 con 0x7f31100b30e0 2024-04-22T01:19:08.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.215+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f310c01b070 con 0x7f31100b30e0 2024-04-22T01:19:08.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.215+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3110068210 con 0x7f31100b30e0 2024-04-22T01:19:08.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.215+0000 7f31067fc640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f30f40678c0 0x7f30f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.215+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f310c0850d0 con 0x7f31100b30e0 2024-04-22T01:19:08.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.215+0000 7f31150fa640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f30f40678c0 0x7f30f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.215+0000 7f31150fa640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f30f40678c0 0x7f30f4069d80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f31000af000 tx=0x7f31000a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.219+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f310c055b10 con 0x7f31100b30e0 2024-04-22T01:19:08.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.383+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f311015d060 con 0x7f31100b30e0 2024-04-22T01:19:08.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f31067fc640 1 -- 172.21.15.70:0/4285740953 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7f310c055410 con 0x7f31100b30e0 2024-04-22T01:19:08.385 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-22T01:19:08.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f30f40678c0 msgr2=0x7f30f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f30f40678c0 0x7f30f4069d80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f31000af000 tx=0x7f31000a7000 comp rx=0 tx=0).stop 2024-04-22T01:19:08.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 msgr2=0x7f31100c7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100c7180 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f310c0011f0 tx=0x7f310c00c9b0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 shutdown_connections 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f30f40678c0 0x7f30f4069d80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f311015e1e0 0x7f31100cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31100b5da0 0x7f311014d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 --2- 172.21.15.70:0/4285740953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31100b30e0 0x7f31100c7180 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 >> 172.21.15.70:0/4285740953 conn(0x7f3110073110 msgr2=0x7f311015d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 shutdown_connections 2024-04-22T01:19:08.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.387+0000 7f3117b5f640 1 -- 172.21.15.70:0/4285740953 wait complete. 2024-04-22T01:19:08.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:19:08.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 Processor -- start 2024-04-22T01:19:08.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 -- start start 2024-04-22T01:19:08.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015e6e0 0x7f69e015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69e00ce920 0x7f69e0154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f69e00ceda0 0x7f69e01551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f69e015f000 con 0x7f69e00ceda0 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f69e00b3800 con 0x7f69e00ce920 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e6cf6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f69e00b3ae0 con 0x7f69e015e6e0 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f69e00ceda0 0x7f69e01551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f69e00ceda0 0x7f69e01551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55426/0 (socket says 172.21.15.70:55426) 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 -- 172.21.15.70:0/2625221510 learned_addr learned my addr 172.21.15.70:0/2625221510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69dffff640 1 --2- 172.21.15.70:0/2625221510 >> v2:172.21.15.98:3300/0 conn(0x7f69e00ce920 0x7f69e0154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e4a92640 1 --2- 172.21.15.70:0/2625221510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015e6e0 0x7f69e015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 -- 172.21.15.70:0/2625221510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015e6e0 msgr2=0x7f69e015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 --2- 172.21.15.70:0/2625221510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015e6e0 0x7f69e015eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 -- 172.21.15.70:0/2625221510 >> v2:172.21.15.98:3300/0 conn(0x7f69e00ce920 msgr2=0x7f69e0154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:08.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 --2- 172.21.15.70:0/2625221510 >> v2:172.21.15.98:3300/0 conn(0x7f69e00ce920 0x7f69e0154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 -- 172.21.15.70:0/2625221510 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e00cde40 con 0x7f69e00ceda0 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.519+0000 7f69e5293640 1 --2- 172.21.15.70:0/2625221510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00ceda0 0x7f69e01551a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f69e0072e50 tx=0x7f69d801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ece56851e242f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69df7fe640 1 -- 172.21.15.70:0/2625221510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d80a5840 con 0x7f69e00ceda0 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69df7fe640 1 -- 172.21.15.70:0/2625221510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d80a59e0 con 0x7f69e00ceda0 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69df7fe640 1 -- 172.21.15.70:0/2625221510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d80a5db0 con 0x7f69e00ceda0 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/2625221510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00ceda0 msgr2=0x7f69e01551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/2625221510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00ceda0 0x7f69e01551a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f69e0072e50 tx=0x7f69d801c9c0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/2625221510 shutdown_connections 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/2625221510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e00ceda0 0x7f69e01551a0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/2625221510 >> v2:172.21.15.98:3300/0 conn(0x7f69e00ce920 0x7f69e0154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/2625221510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e015e6e0 0x7f69e015eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/2625221510 >> 172.21.15.70:0/2625221510 conn(0x7f69e00b1d40 msgr2=0x7f69e00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/2625221510 shutdown_connections 2024-04-22T01:19:08.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/2625221510 wait complete. 2024-04-22T01:19:08.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 Processor -- start 2024-04-22T01:19:08.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- start start 2024-04-22T01:19:08.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e00ce920 0x7f69e00b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00ceda0 0x7f69e014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 0x7f69e014eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69e00b2c30 con 0x7f69e015e6e0 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69e00b2ab0 con 0x7f69e00ceda0 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f69e00b2db0 con 0x7f69e00ce920 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 0x7f69e014eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 0x7f69e014eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55442/0 (socket says 172.21.15.70:55442) 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 -- 172.21.15.70:0/3356091346 learned_addr learned my addr 172.21.15.70:0/3356091346 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 -- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e00ce920 msgr2=0x7f69e00b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69dffff640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00ceda0 0x7f69e014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e00ce920 0x7f69e00b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 -- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00ceda0 msgr2=0x7f69e014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00ceda0 0x7f69e014e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 -- 172.21.15.70:0/3356091346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e00af5b0 con 0x7f69e015e6e0 2024-04-22T01:19:08.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e5293640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 0x7f69e014eee0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f69d801b070 tx=0x7f69d8019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d80bc020 con 0x7f69e015e6e0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d80b8670 con 0x7f69e015e6e0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e00af850 con 0x7f69e015e6e0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d80b8a40 con 0x7f69e015e6e0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.523+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e00afe10 con 0x7f69e015e6e0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.527+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e0068210 con 0x7f69e015e6e0 2024-04-22T01:19:08.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.527+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69d80bf020 con 0x7f69e015e6e0 2024-04-22T01:19:08.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.527+0000 7f69dd7fa640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69c40678f0 0x7f69c4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.527+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f69d8120b40 con 0x7f69e015e6e0 2024-04-22T01:19:08.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.527+0000 7f69e4a92640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69c40678f0 0x7f69c4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.527+0000 7f69e4a92640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69c40678f0 0x7f69c4069db0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f69d400a810 tx=0x7f69d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.531+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f69d80165e0 con 0x7f69e015e6e0 2024-04-22T01:19:08.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69e0158da0 con 0x7f69e015e6e0 2024-04-22T01:19:08.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69dd7fa640 1 -- 172.21.15.70:0/3356091346 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f69d80f1580 con 0x7f69e015e6e0 2024-04-22T01:19:08.678 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-22T01:19:08.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69c40678f0 msgr2=0x7f69c4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69c40678f0 0x7f69c4069db0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f69d400a810 tx=0x7f69d4008040 comp rx=0 tx=0).stop 2024-04-22T01:19:08.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 msgr2=0x7f69e014eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 0x7f69e014eee0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f69d801b070 tx=0x7f69d8019760 comp rx=0 tx=0).stop 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 shutdown_connections 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f69c40678f0 0x7f69c4069db0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69e015e6e0 0x7f69e014eee0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e00ceda0 0x7f69e014e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 --2- 172.21.15.70:0/3356091346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69e00ce920 0x7f69e00b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 >> 172.21.15.70:0/3356091346 conn(0x7f69e00b1d40 msgr2=0x7f69e00b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.679+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 shutdown_connections 2024-04-22T01:19:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.683+0000 7f69e6cf6640 1 -- 172.21.15.70:0/3356091346 wait complete. 2024-04-22T01:19:08.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:19:08.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 Processor -- start 2024-04-22T01:19:08.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 -- start start 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8d40b98a0 0x7ff8d40d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40cea10 0x7ff8d40cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d40b9ce0 con 0x7ff8d415de20 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff8d40cf3c0 con 0x7ff8d40b98a0 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d9d7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d40b75e0 con 0x7ff8d40cea10 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d3fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40cea10 0x7ff8d40cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8d40b98a0 0x7ff8d40d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff8d415de20 0x7ff8d40b6db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:41672/0 (socket says 172.21.15.70:41672) 2024-04-22T01:19:08.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d37fe640 1 -- 172.21.15.70:0/3704075520 learned_addr learned my addr 172.21.15.70:0/3704075520 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:08.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3704075520 >> v2:172.21.15.98:3300/0 conn(0x7ff8d40b98a0 msgr2=0x7ff8d40d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:08.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d3fff640 1 --2- 172.21.15.70:0/3704075520 >> v2:172.21.15.98:3300/0 conn(0x7ff8d40b98a0 0x7ff8d40d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3704075520 >> v1:172.21.15.98:6789/0 conn(0x7ff8d415de20 legacy=0x7ff8d40b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.807+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3704075520 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d40b78c0 con 0x7ff8d40cea10 2024-04-22T01:19:08.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d3fff640 1 --2- 172.21.15.70:0/3704075520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40cea10 0x7ff8d40cee80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009d90 tx=0x7ff8cc01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8016500e7197f39 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d27fc640 1 -- 172.21.15.70:0/3704075520 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0b4070 con 0x7ff8d40cea10 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d27fc640 1 -- 172.21.15.70:0/3704075520 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8cc0af600 con 0x7ff8d40cea10 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d27fc640 1 -- 172.21.15.70:0/3704075520 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0af9d0 con 0x7ff8d40cea10 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3704075520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40cea10 msgr2=0x7ff8d40cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3704075520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40cea10 0x7ff8d40cee80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009d90 tx=0x7ff8cc01f150 comp rx=0 tx=0).stop 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3704075520 shutdown_connections 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3704075520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40cea10 0x7ff8d40cee80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3704075520 >> v2:172.21.15.98:3300/0 conn(0x7ff8d40b98a0 0x7ff8d40d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3704075520 >> 172.21.15.70:0/3704075520 conn(0x7ff8d4072d20 msgr2=0x7ff8d4073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3704075520 shutdown_connections 2024-04-22T01:19:08.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3704075520 wait complete. 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 Processor -- start 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- start start 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40b98a0 0x7ff8d40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8d40cea10 0x7ff8d40c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 0x7ff8d4161b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d40b8160 con 0x7ff8d415de20 2024-04-22T01:19:08.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d40b7e80 con 0x7ff8d40cea10 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d9d7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d415e200 con 0x7ff8d40b98a0 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 0x7ff8d4161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 0x7ff8d4161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55450/0 (socket says 172.21.15.70:55450) 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.811+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3033700123 learned_addr learned my addr 172.21.15.70:0/3033700123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d2ffd640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8d40cea10 0x7ff8d40c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d37fe640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40b98a0 0x7ff8d40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40b98a0 msgr2=0x7ff8d40cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d3fff640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40b98a0 0x7ff8d40cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8d40cea10 msgr2=0x7ff8d40c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d3fff640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8d40cea10 0x7ff8d40c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d3fff640 1 -- 172.21.15.70:0/3033700123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d40cb4a0 con 0x7ff8d415de20 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d3fff640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 0x7ff8d4161b90 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc01cd80 tx=0x7ff8cc01bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0b4070 con 0x7ff8d415de20 2024-04-22T01:19:08.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8cc0c1070 con 0x7ff8d415de20 2024-04-22T01:19:08.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d40cb780 con 0x7ff8d415de20 2024-04-22T01:19:08.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0bb8d0 con 0x7ff8d415de20 2024-04-22T01:19:08.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d40cbd40 con 0x7ff8d415de20 2024-04-22T01:19:08.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d4068210 con 0x7ff8d415de20 2024-04-22T01:19:08.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff8cc0bba70 con 0x7ff8d415de20 2024-04-22T01:19:08.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8bbfff640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8c00678c0 0x7ff8c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:08.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d37fe640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8c00678c0 0x7ff8c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:08.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7ff8cc002020 con 0x7ff8d415de20 2024-04-22T01:19:08.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.815+0000 7ff8d37fe640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8c00678c0 0x7ff8c0069d80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4000f70 tx=0x7ff8c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:08.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.819+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7ff8cc018cb0 con 0x7ff8d415de20 2024-04-22T01:19:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.967+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff8d406b9b0 con 0x7ff8d415de20 2024-04-22T01:19:08.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8bbfff640 1 -- 172.21.15.70:0/3033700123 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7ff8cc0f5170 con 0x7ff8d415de20 2024-04-22T01:19:08.969 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-22T01:19:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8c00678c0 msgr2=0x7ff8c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8c00678c0 0x7ff8c0069d80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4000f70 tx=0x7ff8c4009040 comp rx=0 tx=0).stop 2024-04-22T01:19:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 msgr2=0x7ff8d4161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:08.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 0x7ff8d4161b90 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc01cd80 tx=0x7ff8cc01bef0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 shutdown_connections 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff8c00678c0 0x7ff8c0069d80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8d415de20 0x7ff8d4161b90 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8d40cea10 0x7ff8d40c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 --2- 172.21.15.70:0/3033700123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8d40b98a0 0x7ff8d40cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 >> 172.21.15.70:0/3033700123 conn(0x7ff8d4072d20 msgr2=0x7ff8d4073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 shutdown_connections 2024-04-22T01:19:08.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:08.971+0000 7ff8d9d7a640 1 -- 172.21.15.70:0/3033700123 wait complete. 2024-04-22T01:19:08.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:19:09.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa199554640 1 Processor -- start 2024-04-22T01:19:09.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa199554640 1 -- start start 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa199554640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 0x7fa194166b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa199554640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1940cd7b0 con 0x7fa1941455b0 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa199554640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1940cc710 con 0x7fa1940cc150 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa199554640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa1940cc9f0 con 0x7fa194072230 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa1941455b0 0x7fa1940cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58600/0 (socket says 172.21.15.70:58600) 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 -- 172.21.15.70:0/4191750644 learned_addr learned my addr 172.21.15.70:0/4191750644 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1937fe640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 0x7fa194166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790507995 0 0) 0x7fa1940cd7b0 con 0x7fa1941455b0 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa180003530 con 0x7fa1941455b0 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 893214715 0 0) 0x7fa180003530 con 0x7fa1941455b0 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 msgr2=0x7fa194166b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 0x7fa194166b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 >> v1:172.21.15.98:6789/0 conn(0x7fa1940cc150 legacy=0x7fa194071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1940cccd0 con 0x7fa1941455b0 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fa184001680 con 0x7fa1941455b0 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 >> v1:172.21.15.70:6789/0 conn(0x7fa1941455b0 legacy=0x7fa1940cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 0x7fa180004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa180004d50 0x7fa1800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1940cc9f0 con 0x7fa1800039f0 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa18000bbc0 con 0x7fa180004d50 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa18000bdc0 con 0x7fa194072230 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa18bfff640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 0x7fa180004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1937fe640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa180004d50 0x7fa1800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 -- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 msgr2=0x7fa180004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194072230 0x7fa180004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 -- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa180004d50 msgr2=0x7fa1800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa180004d50 0x7fa1800051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa192ffd640 1 -- 172.21.15.70:0/4191750644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa180003530 con 0x7fa1800039f0 2024-04-22T01:19:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.107+0000 7fa1937fe640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa180004d50 0x7fa1800051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa192ffd640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa18400c240 tx=0x7fa18401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154d6bc7c688d751 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184007d20 con 0x7fa1800039f0 2024-04-22T01:19:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1840a9070 con 0x7fa1800039f0 2024-04-22T01:19:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa1927fc640 1 -- 172.21.15.70:0/4191750644 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184005ce0 con 0x7fa1800039f0 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- 172.21.15.70:0/4191750644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 msgr2=0x7fa180003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 --2- 172.21.15.70:0/4191750644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa18400c240 tx=0x7fa18401cc80 comp rx=0 tx=0).stop 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa1937fe640 1 -- 172.21.15.70:0/4191750644 reap_dead start 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- 172.21.15.70:0/4191750644 shutdown_connections 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- 172.21.15.70:0/4191750644 >> 172.21.15.70:0/4191750644 conn(0x7fa19406b1f0 msgr2=0x7fa19406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- 172.21.15.70:0/4191750644 shutdown_connections 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- 172.21.15.70:0/4191750644 wait complete. 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 Processor -- start 2024-04-22T01:19:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- start start 2024-04-22T01:19:09.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa194160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1940b0230 0x7fa19415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194139bb0 0x7fa19415d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941676d0 con 0x7fa1800039f0 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa194167550 con 0x7fa1940b0230 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa199554640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa194167850 con 0x7fa194139bb0 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa192ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa194160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa192ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa194160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55466/0 (socket says 172.21.15.70:55466) 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.111+0000 7fa192ffd640 1 -- 172.21.15.70:0/3455729579 learned_addr learned my addr 172.21.15.70:0/3455729579 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18bfff640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1940b0230 0x7fa19415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa1937fe640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194139bb0 0x7fa19415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa192ffd640 1 -- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194139bb0 msgr2=0x7fa19415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa192ffd640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194139bb0 0x7fa19415d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa192ffd640 1 -- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1940b0230 msgr2=0x7fa19415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa192ffd640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1940b0230 0x7fa19415b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa192ffd640 1 -- 172.21.15.70:0/3455729579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1940b0710 con 0x7fa1800039f0 2024-04-22T01:19:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa192ffd640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa194160860 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa18401cab0 tx=0x7fa180005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184001020 con 0x7fa1800039f0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1840047f0 con 0x7fa1800039f0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1940b09f0 con 0x7fa1800039f0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184004ba0 con 0x7fa1800039f0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1940b0cd0 con 0x7fa1800039f0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa194068210 con 0x7fa1800039f0 2024-04-22T01:19:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa184007070 con 0x7fa1800039f0 2024-04-22T01:19:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18b7fe640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1740678f0 0x7fa174069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa18411f540 con 0x7fa1800039f0 2024-04-22T01:19:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18bfff640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1740678f0 0x7fa174069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.115+0000 7fa18bfff640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1740678f0 0x7fa174069db0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa1941403e0 tx=0x7fa17c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.119+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa184016990 con 0x7fa1800039f0 2024-04-22T01:19:09.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.315+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa19415f510 con 0x7fa1800039f0 2024-04-22T01:19:09.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.315+0000 7fa18b7fe640 1 -- 172.21.15.70:0/3455729579 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7fa1840eff80 con 0x7fa1800039f0 2024-04-22T01:19:09.316 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:09.316 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:51.626962+0000","modified":"2024-04-22T01:18:53.648845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7629":{"gid":7629,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3578750329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3578750329},{"type":"v1","addr":"172.21.15.70:6837","nonce":3578750329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7629,"qdb_cluster":[7629]},"id":7}]} 2024-04-22T01:19:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 50 2024-04-22T01:19:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1740678f0 msgr2=0x7fa174069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1740678f0 0x7fa174069db0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa1941403e0 tx=0x7fa17c009040 comp rx=0 tx=0).stop 2024-04-22T01:19:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 msgr2=0x7fa194160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa194160860 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa18401cab0 tx=0x7fa180005840 comp rx=0 tx=0).stop 2024-04-22T01:19:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 shutdown_connections 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa1740678f0 0x7fa174069db0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa194139bb0 0x7fa19415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1940b0230 0x7fa19415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 --2- 172.21.15.70:0/3455729579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1800039f0 0x7fa194160860 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 >> 172.21.15.70:0/3455729579 conn(0x7fa19406b1f0 msgr2=0x7fa1940cf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 shutdown_connections 2024-04-22T01:19:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.319+0000 7fa199554640 1 -- 172.21.15.70:0/3455729579 wait complete. 2024-04-22T01:19:09.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:19:09.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 Processor -- start 2024-04-22T01:19:09.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 -- start start 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f04b40aee40 0x7f04b40b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 0x7f04b40b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04b406dbd0 con 0x7f04b415aa30 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f04b406de70 con 0x7f04b40aee40 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04bc0bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f04b406e150 con 0x7f04b415bb50 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9e58640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f04b415aa30 0x7f04b40ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58608/0 (socket says 172.21.15.70:58608) 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9e58640 1 -- 172.21.15.70:0/134988909 learned_addr learned my addr 172.21.15.70:0/134988909 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 --2- 172.21.15.70:0/134988909 >> v2:172.21.15.98:3300/0 conn(0x7f04b40aee40 0x7f04b40b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04ba659640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 0x7f04b40b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648167080 0 0) 0x7f04b406dbd0 con 0x7f04b415aa30 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f049c003530 con 0x7f04b415aa30 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1392402079 0 0) 0x7f049c003530 con 0x7f04b415aa30 2024-04-22T01:19:09.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 msgr2=0x7f04b40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 0x7f04b40b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 >> v2:172.21.15.98:3300/0 conn(0x7f04b40aee40 msgr2=0x7f04b40b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 --2- 172.21.15.70:0/134988909 >> v2:172.21.15.98:3300/0 conn(0x7f04b40aee40 0x7f04b40b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b406e430 con 0x7f04b415aa30 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f04a4001a20 con 0x7f04b415aa30 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 >> v1:172.21.15.70:6789/0 conn(0x7f04b415aa30 legacy=0x7f04b40ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0039c0 0x7f049c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f049c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 0x7f049c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04b406de70 con 0x7f049c004810 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04b406e150 con 0x7f049c0039c0 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f049c00b9d0 con 0x7f04b415bb50 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f049c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9e58640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0039c0 0x7f049c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04ba659640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 0x7f049c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 -- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 msgr2=0x7f049c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b415bb50 0x7f049c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 -- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0039c0 msgr2=0x7f049c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0039c0 0x7f049c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.451+0000 7f04b9657640 1 -- 172.21.15.70:0/134988909 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c003530 con 0x7f049c004810 2024-04-22T01:19:09.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9657640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f049c004c90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009ab0 tx=0x7f04a801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b291d62e3ee9251 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a80096b0 con 0x7f049c004810 2024-04-22T01:19:09.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a80a64a0 con 0x7f049c004810 2024-04-22T01:19:09.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b8e56640 1 -- 172.21.15.70:0/134988909 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a80a6890 con 0x7f049c004810 2024-04-22T01:19:09.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- 172.21.15.70:0/134988909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 msgr2=0x7f049c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/134988909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f049c004c90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009ab0 tx=0x7f04a801ca90 comp rx=0 tx=0).stop 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04ba659640 1 -- 172.21.15.70:0/134988909 reap_dead start 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- 172.21.15.70:0/134988909 shutdown_connections 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- 172.21.15.70:0/134988909 >> 172.21.15.70:0/134988909 conn(0x7f04b406b1f0 msgr2=0x7f04b406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- 172.21.15.70:0/134988909 shutdown_connections 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- 172.21.15.70:0/134988909 wait complete. 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 Processor -- start 2024-04-22T01:19:09.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- start start 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f04b414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b40cdce0 0x7f04b40afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04b4139260 0x7f04b414e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04b415ae10 con 0x7f049c004810 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04b40b4d00 con 0x7f04b4139260 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04bc0bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f04b415af90 con 0x7f04b40cdce0 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f04b414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f04b414b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55486/0 (socket says 172.21.15.70:55486) 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 -- 172.21.15.70:0/4150246840 learned_addr learned my addr 172.21.15.70:0/4150246840 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9657640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b40cdce0 0x7f04b40afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04ba659640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04b4139260 0x7f04b414e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 -- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b40cdce0 msgr2=0x7f04b40afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b40cdce0 0x7f04b40afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 -- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04b4139260 msgr2=0x7f04b414e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04b4139260 0x7f04b414e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9e58640 1 -- 172.21.15.70:0/4150246840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b40b1670 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04ba659640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04b4139260 0x7f04b414e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.455+0000 7f04b9657640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b40cdce0 0x7f04b40afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b9e58640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f04b414b2a0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f04a40117f0 tx=0x7f04a4011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4010040 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a400fb60 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b414b9d0 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4015410 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b40b1bf0 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b4068210 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f04a40155b0 con 0x7f049c004810 2024-04-22T01:19:09.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b27fc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04a00678c0 0x7f04a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f04a4089210 con 0x7f049c004810 2024-04-22T01:19:09.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b9657640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04a00678c0 0x7f04a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.459+0000 7f04b9657640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04a00678c0 0x7f04a0069d80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f04a80ad000 tx=0x7f04a80046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.463+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f04a4059bc0 con 0x7f049c004810 2024-04-22T01:19:09.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.611+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04b4072750 con 0x7f049c004810 2024-04-22T01:19:09.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04b27fc640 1 -- 172.21.15.70:0/4150246840 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7f04a40594c0 con 0x7f049c004810 2024-04-22T01:19:09.614 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:09.614 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":103,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:53.308712+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04a00678c0 msgr2=0x7f04a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04a00678c0 0x7f04a0069d80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f04a80ad000 tx=0x7f04a80046d0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 msgr2=0x7f04b414b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f04b414b2a0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f04a40117f0 tx=0x7f04a4011cb0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 shutdown_connections 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f04a00678c0 0x7f04a0069d80 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04b4139260 0x7f04b414e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04b40cdce0 0x7f04b40afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 --2- 172.21.15.70:0/4150246840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c004810 0x7f04b414b2a0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 >> 172.21.15.70:0/4150246840 conn(0x7f04b406b1f0 msgr2=0x7f04b406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:09.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 shutdown_connections 2024-04-22T01:19:09.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.615+0000 7f04bc0bc640 1 -- 172.21.15.70:0/4150246840 wait complete. 2024-04-22T01:19:09.631 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:19:09.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:19:09.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.743+0000 7f057661a640 1 Processor -- start 2024-04-22T01:19:09.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.743+0000 7f057661a640 1 -- start start 2024-04-22T01:19:09.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f057661a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f057015aa30 0x7f05700b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f057661a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05700aee40 0x7f05700b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f057661a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057006dbd0 con 0x7f057015bb50 2024-04-22T01:19:09.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f057661a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f057006de70 con 0x7f05700aee40 2024-04-22T01:19:09.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f057661a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f057006e150 con 0x7f057015aa30 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05700aee40 0x7f05700b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05700aee40 0x7f05700b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55488/0 (socket says 172.21.15.70:55488) 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 -- 172.21.15.70:0/3293053026 learned_addr learned my addr 172.21.15.70:0/3293053026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f056ffff640 1 --2- 172.21.15.70:0/3293053026 >> v2:172.21.15.98:3300/0 conn(0x7f057015aa30 0x7f05700b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 -- 172.21.15.70:0/3293053026 >> v2:172.21.15.98:3300/0 conn(0x7f057015aa30 msgr2=0x7f05700b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 --2- 172.21.15.70:0/3293053026 >> v2:172.21.15.98:3300/0 conn(0x7f057015aa30 0x7f05700b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 -- 172.21.15.70:0/3293053026 >> v1:172.21.15.70:6789/0 conn(0x7f057015bb50 legacy=0x7f05700ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 -- 172.21.15.70:0/3293053026 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057006e430 con 0x7f05700aee40 2024-04-22T01:19:09.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f0574bb7640 1 --2- 172.21.15.70:0/3293053026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05700aee40 0x7f05700b28f0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f057006ecd0 tx=0x7f056001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e76b5d18d80116 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f056effd640 1 -- 172.21.15.70:0/3293053026 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05600a58a0 con 0x7f05700aee40 2024-04-22T01:19:09.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f056effd640 1 -- 172.21.15.70:0/3293053026 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05600a5a40 con 0x7f05700aee40 2024-04-22T01:19:09.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.747+0000 7f056effd640 1 -- 172.21.15.70:0/3293053026 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05600a5e10 con 0x7f05700aee40 2024-04-22T01:19:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/3293053026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05700aee40 msgr2=0x7f05700b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 --2- 172.21.15.70:0/3293053026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05700aee40 0x7f05700b28f0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f057006ecd0 tx=0x7f056001c9c0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/3293053026 shutdown_connections 2024-04-22T01:19:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 --2- 172.21.15.70:0/3293053026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05700aee40 0x7f05700b28f0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 --2- 172.21.15.70:0/3293053026 >> v2:172.21.15.98:3300/0 conn(0x7f057015aa30 0x7f05700b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/3293053026 >> 172.21.15.70:0/3293053026 conn(0x7f057006b1f0 msgr2=0x7f057006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/3293053026 shutdown_connections 2024-04-22T01:19:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/3293053026 wait complete. 2024-04-22T01:19:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 Processor -- start 2024-04-22T01:19:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- start start 2024-04-22T01:19:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05700aee40 0x7f0570150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 0x7f05701423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f057015bb50 0x7f0570071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f057006ead0 con 0x7f057015aa30 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f057006e950 con 0x7f05700aee40 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0570167a90 con 0x7f057015bb50 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 0x7f05701423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 0x7f05701423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55502/0 (socket says 172.21.15.70:55502) 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 -- 172.21.15.70:0/820294298 learned_addr learned my addr 172.21.15.70:0/820294298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056ffff640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05700aee40 0x7f0570150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 -- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f057015bb50 msgr2=0x7f0570071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f057015bb50 0x7f0570071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 -- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05700aee40 msgr2=0x7f0570150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05700aee40 0x7f0570150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 -- 172.21.15.70:0/820294298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0570152130 con 0x7f057015aa30 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056f7fe640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 0x7f05701423d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f055800c480 tx=0x7f055800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580052d0 con 0x7f057015aa30 2024-04-22T01:19:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0558005490 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05701515b0 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558012850 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.751+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0570151b70 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.755+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0570068210 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.755+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05580129f0 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.755+0000 7f056cff9640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05500678f0 0x7f0550069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.755+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f0558085a20 con 0x7f057015aa30 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.755+0000 7f056ffff640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05500678f0 0x7f0550069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.755+0000 7f056ffff640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05500678f0 0x7f0550069db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f056800b930 tx=0x7f0568008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:09.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.759+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0558056460 con 0x7f057015aa30 2024-04-22T01:19:09.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.939+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f057006f190 con 0x7f057015aa30 2024-04-22T01:19:09.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.939+0000 7f056cff9640 1 -- 172.21.15.70:0/820294298 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f0558055d60 con 0x7f057015aa30 2024-04-22T01:19:09.940 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:09.940 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:51.626962+0000","modified":"2024-04-22T01:18:53.648845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7629":{"gid":7629,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3578750329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3578750329},{"type":"v1","addr":"172.21.15.70:6837","nonce":3578750329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7629,"qdb_cluster":[7629]},"id":7}]} 2024-04-22T01:19:09.940 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 50 2024-04-22T01:19:09.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05500678f0 msgr2=0x7f0550069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05500678f0 0x7f0550069db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f056800b930 tx=0x7f0568008040 comp rx=0 tx=0).stop 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 msgr2=0x7f05701423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 0x7f05701423d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f055800c480 tx=0x7f055800c940 comp rx=0 tx=0).stop 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 shutdown_connections 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f05500678f0 0x7f0550069db0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f057015bb50 0x7f0570071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f057015aa30 0x7f05701423d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 --2- 172.21.15.70:0/820294298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05700aee40 0x7f0570150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 >> 172.21.15.70:0/820294298 conn(0x7f057006b1f0 msgr2=0x7f057015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 shutdown_connections 2024-04-22T01:19:09.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:09.943+0000 7f057661a640 1 -- 172.21.15.70:0/820294298 wait complete. 2024-04-22T01:19:09.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:19:10.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.067+0000 7f909ac0e640 1 Processor -- start 2024-04-22T01:19:10.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.067+0000 7f909ac0e640 1 -- start start 2024-04-22T01:19:10.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90941455b0 0x7f90940cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f90940cc150 0x7f9094071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f90940724b0 con 0x7f90940cc150 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f90940d0250 con 0x7f90941455b0 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f90940cd7b0 con 0x7f9094072060 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f90940cc150 0x7f9094071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f90940cc150 0x7f9094071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55510/0 (socket says 172.21.15.70:55510) 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 -- 172.21.15.70:0/1516947893 learned_addr learned my addr 172.21.15.70:0/1516947893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f90989aa640 1 --2- 172.21.15.70:0/1516947893 >> v2:172.21.15.98:3300/0 conn(0x7f90941455b0 0x7f90940cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f90991ab640 1 --2- 172.21.15.70:0/1516947893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 -- 172.21.15.70:0/1516947893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 msgr2=0x7f909415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 --2- 172.21.15.70:0/1516947893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 -- 172.21.15.70:0/1516947893 >> v2:172.21.15.98:3300/0 conn(0x7f90941455b0 msgr2=0x7f90940cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 --2- 172.21.15.70:0/1516947893 >> v2:172.21.15.98:3300/0 conn(0x7f90941455b0 0x7f90940cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 -- 172.21.15.70:0/1516947893 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90940cc710 con 0x7f90940cc150 2024-04-22T01:19:10.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f9093fff640 1 --2- 172.21.15.70:0/1516947893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90940cc150 0x7f9094071b20 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f90940d0700 tx=0x7f908801c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed842aae4a0f67b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f90937fe640 1 -- 172.21.15.70:0/1516947893 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90880b1070 con 0x7f90940cc150 2024-04-22T01:19:10.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f90937fe640 1 -- 172.21.15.70:0/1516947893 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90880ac440 con 0x7f90940cc150 2024-04-22T01:19:10.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f90937fe640 1 -- 172.21.15.70:0/1516947893 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90880ac810 con 0x7f90940cc150 2024-04-22T01:19:10.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- 172.21.15.70:0/1516947893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90940cc150 msgr2=0x7f9094071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1516947893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90940cc150 0x7f9094071b20 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f90940d0700 tx=0x7f908801c140 comp rx=0 tx=0).stop 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- 172.21.15.70:0/1516947893 shutdown_connections 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1516947893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1516947893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90940cc150 0x7f9094071b20 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1516947893 >> v2:172.21.15.98:3300/0 conn(0x7f90941455b0 0x7f90940cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- 172.21.15.70:0/1516947893 >> 172.21.15.70:0/1516947893 conn(0x7f909406b1f0 msgr2=0x7f909406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- 172.21.15.70:0/1516947893 shutdown_connections 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.071+0000 7f909ac0e640 1 -- 172.21.15.70:0/1516947893 wait complete. 2024-04-22T01:19:10.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 Processor -- start 2024-04-22T01:19:10.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 -- start start 2024-04-22T01:19:10.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90941455b0 0x7f909415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 0x7f9094136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f909406e380 con 0x7f909413f2a0 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f909406e200 con 0x7f90941455b0 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f909406e500 con 0x7f9094072060 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 0x7f9094136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 0x7f9094136b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55514/0 (socket says 172.21.15.70:55514) 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 -- 172.21.15.70:0/1857288343 learned_addr learned my addr 172.21.15.70:0/1857288343 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f9093fff640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90941455b0 0x7f909415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90989aa640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 -- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 msgr2=0x7f909415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 -- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90941455b0 msgr2=0x7f909415e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90941455b0 0x7f909415e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 -- 172.21.15.70:0/1857288343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90940af670 con 0x7f909413f2a0 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f9093fff640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90941455b0 0x7f909415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90991ab640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 0x7f9094136b40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c5c0 tx=0x7f908c00ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c015070 con 0x7f909413f2a0 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c004220 con 0x7f909413f2a0 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c003620 con 0x7f909413f2a0 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90940af950 con 0x7f909413f2a0 2024-04-22T01:19:10.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.075+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90940aff10 con 0x7f909413f2a0 2024-04-22T01:19:10.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.079+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90940af3e0 con 0x7f909413f2a0 2024-04-22T01:19:10.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.079+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f908c0037c0 con 0x7f909413f2a0 2024-04-22T01:19:10.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.079+0000 7f90917fa640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f90800678c0 0x7f9080069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.079+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f908c085c40 con 0x7f909413f2a0 2024-04-22T01:19:10.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.079+0000 7f90989aa640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f90800678c0 0x7f9080069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.079+0000 7f90989aa640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f90800678c0 0x7f9080069d80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f90940cdbf0 tx=0x7f90840083c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.083+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f908c0565f0 con 0x7f909413f2a0 2024-04-22T01:19:10.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.263+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9094068210 con 0x7f909413f2a0 2024-04-22T01:19:10.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f90917fa640 1 -- 172.21.15.70:0/1857288343 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5180 (secure 0 0 0) 0x7f908c055ef0 con 0x7f909413f2a0 2024-04-22T01:19:10.265 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:10.265 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:18:51.626962+0000","modified":"2024-04-22T01:18:53.648845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7629":{"gid":7629,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3578750329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3578750329},{"type":"v1","addr":"172.21.15.70:6837","nonce":3578750329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7629,"qdb_cluster":[7629]},"id":7}]} 2024-04-22T01:19:10.266 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 50 2024-04-22T01:19:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f90800678c0 msgr2=0x7f9080069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f90800678c0 0x7f9080069d80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f90940cdbf0 tx=0x7f90840083c0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 msgr2=0x7f9094136b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 0x7f9094136b40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c5c0 tx=0x7f908c00ca80 comp rx=0 tx=0).stop 2024-04-22T01:19:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 shutdown_connections 2024-04-22T01:19:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f90800678c0 0x7f9080069d80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f909413f2a0 0x7f9094136b40 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90941455b0 0x7f909415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 --2- 172.21.15.70:0/1857288343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9094072060 0x7f909415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 >> 172.21.15.70:0/1857288343 conn(0x7f909406b1f0 msgr2=0x7f90940744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:10.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 shutdown_connections 2024-04-22T01:19:10.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.267+0000 7f909ac0e640 1 -- 172.21.15.70:0/1857288343 wait complete. 2024-04-22T01:19:10.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:19:10.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf198f3640 1 Processor -- start 2024-04-22T01:19:10.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf198f3640 1 -- start start 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf198f3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbf1415dfb0 0x7fbf140b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf198f3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf140b6550 con 0x7fbf140d3120 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf198f3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf1415e390 con 0x7fbf140cea80 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf198f3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbf14161e00 con 0x7fbf1415dfb0 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf12ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbf1415dfb0 0x7fbf140b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf137fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbf140d3120 0x7fbf140b5e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58632/0 (socket says 172.21.15.70:58632) 2024-04-22T01:19:10.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf137fe640 1 -- 172.21.15.70:0/2376090562 learned_addr learned my addr 172.21.15.70:0/2376090562 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:10.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf12ffd640 1 -- 172.21.15.70:0/2376090562 >> v1:172.21.15.98:6789/0 conn(0x7fbf140cea80 legacy=0x7fbf140b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:10.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf12ffd640 1 -- 172.21.15.70:0/2376090562 >> v1:172.21.15.70:6789/0 conn(0x7fbf140d3120 legacy=0x7fbf140b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf12ffd640 1 -- 172.21.15.70:0/2376090562 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf141620e0 con 0x7fbf1415dfb0 2024-04-22T01:19:10.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf127fc640 1 -- 172.21.15.70:0/2376090562 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3120558576 0 0) 0x7fbf140b6550 con 0x7fbf140d3120 2024-04-22T01:19:10.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.407+0000 7fbf12ffd640 1 --2- 172.21.15.70:0/2376090562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf1415dfb0 0x7fbf140b95d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fbf040097c0 tx=0x7fbf0401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3e371b3e3016af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf127fc640 1 -- 172.21.15.70:0/2376090562 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040b0070 con 0x7fbf1415dfb0 2024-04-22T01:19:10.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf127fc640 1 -- 172.21.15.70:0/2376090562 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf040ab600 con 0x7fbf1415dfb0 2024-04-22T01:19:10.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf127fc640 1 -- 172.21.15.70:0/2376090562 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040ab9d0 con 0x7fbf1415dfb0 2024-04-22T01:19:10.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- 172.21.15.70:0/2376090562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf1415dfb0 msgr2=0x7fbf140b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 --2- 172.21.15.70:0/2376090562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf1415dfb0 0x7fbf140b95d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fbf040097c0 tx=0x7fbf0401c460 comp rx=0 tx=0).stop 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- 172.21.15.70:0/2376090562 shutdown_connections 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 --2- 172.21.15.70:0/2376090562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf1415dfb0 0x7fbf140b95d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- 172.21.15.70:0/2376090562 >> 172.21.15.70:0/2376090562 conn(0x7fbf1406c9b0 msgr2=0x7fbf1406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- 172.21.15.70:0/2376090562 shutdown_connections 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- 172.21.15.70:0/2376090562 wait complete. 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 Processor -- start 2024-04-22T01:19:10.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- start start 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf140cea80 0x7fbf141582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 0x7fbf1415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf1415dfb0 0x7fbf14146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14161a90 con 0x7fbf140d3120 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf140cf2f0 con 0x7fbf140cea80 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbf1416c5f0 con 0x7fbf1415dfb0 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 0x7fbf1415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 0x7fbf1415bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55540/0 (socket says 172.21.15.70:55540) 2024-04-22T01:19:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 -- 172.21.15.70:0/3526525248 learned_addr learned my addr 172.21.15.70:0/3526525248 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf137fe640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf1415dfb0 0x7fbf14146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf12ffd640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf140cea80 0x7fbf141582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 -- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf1415dfb0 msgr2=0x7fbf14146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf1415dfb0 0x7fbf14146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 -- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf140cea80 msgr2=0x7fbf141582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf140cea80 0x7fbf141582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 -- 172.21.15.70:0/3526525248 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1415c950 con 0x7fbf140d3120 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf0a7fc640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 0x7fbf1415bfe0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00b410 tx=0x7fbefc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc015030 con 0x7fbf140d3120 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbefc004020 con 0x7fbf140d3120 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc004410 con 0x7fbf140d3120 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.411+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1415cc30 con 0x7fbf140d3120 2024-04-22T01:19:10.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf14158820 con 0x7fbf140d3120 2024-04-22T01:19:10.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf14068210 con 0x7fbf140d3120 2024-04-22T01:19:10.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbefc018070 con 0x7fbf140d3120 2024-04-22T01:19:10.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf10ff9640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbef40678c0 0x7fbef4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fbefc0859b0 con 0x7fbf140d3120 2024-04-22T01:19:10.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf12ffd640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbef40678c0 0x7fbef4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.415+0000 7fbf12ffd640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbef40678c0 0x7fbef4069d80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbf040ad000 tx=0x7fbf040a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.419+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fbefc056360 con 0x7fbf140d3120 2024-04-22T01:19:10.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.579+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf141599b0 con 0x7fbf140d3120 2024-04-22T01:19:10.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.579+0000 7fbf10ff9640 1 -- 172.21.15.70:0/3526525248 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7fbefc055c60 con 0x7fbf140d3120 2024-04-22T01:19:10.579 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:10.579 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":103,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:18:53.308712+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:18:49.869911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-22T01:18:50.621566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbef40678c0 msgr2=0x7fbef4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbef40678c0 0x7fbef4069d80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbf040ad000 tx=0x7fbf040a6000 comp rx=0 tx=0).stop 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 msgr2=0x7fbf1415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 0x7fbf1415bfe0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00b410 tx=0x7fbefc00b8d0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 shutdown_connections 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fbef40678c0 0x7fbef4069d80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf1415dfb0 0x7fbf14146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf140d3120 0x7fbf1415bfe0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 --2- 172.21.15.70:0/3526525248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf140cea80 0x7fbf141582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 >> 172.21.15.70:0/3526525248 conn(0x7fbf1406c9b0 msgr2=0x7fbf140b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 shutdown_connections 2024-04-22T01:19:10.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.583+0000 7fbf198f3640 1 -- 172.21.15.70:0/3526525248 wait complete. 2024-04-22T01:19:10.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:19:10.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 Processor -- start 2024-04-22T01:19:10.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 -- start start 2024-04-22T01:19:10.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f23580b3db0 0x7f23580ae990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b29d0 0x7f235806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23580adb70 con 0x7f235815e000 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f23580b2e40 con 0x7f23580b3db0 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f235d4a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f235806db50 con 0x7f23580b29d0 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23567fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f23580b3db0 0x7f23580ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f2356ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f235815e000 0x7f23580b36a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58648/0 (socket says 172.21.15.70:58648) 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f2356ffd640 1 -- 172.21.15.70:0/4102891447 learned_addr learned my addr 172.21.15.70:0/4102891447 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23577fe640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b29d0 0x7f235806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23567fc640 1 -- 172.21.15.70:0/4102891447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b29d0 msgr2=0x7f235806dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23567fc640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b29d0 0x7f235806dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23567fc640 1 -- 172.21.15.70:0/4102891447 >> v1:172.21.15.70:6789/0 conn(0x7f235815e000 legacy=0x7f23580b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23567fc640 1 -- 172.21.15.70:0/4102891447 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23580addf0 con 0x7f23580b3db0 2024-04-22T01:19:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.723+0000 7f23577fe640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b29d0 0x7f235806dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:10.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23567fc640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580b3db0 0x7f23580ae990 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f23400097c0 tx=0x7f234001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0da32aaa719281 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f2355ffb640 1 -- 172.21.15.70:0/4102891447 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400b1070 con 0x7f23580b3db0 2024-04-22T01:19:10.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f2355ffb640 1 -- 172.21.15.70:0/4102891447 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23400ac600 con 0x7f23580b3db0 2024-04-22T01:19:10.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f2355ffb640 1 -- 172.21.15.70:0/4102891447 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400ac9d0 con 0x7f23580b3db0 2024-04-22T01:19:10.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- 172.21.15.70:0/4102891447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580b3db0 msgr2=0x7f23580ae990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580b3db0 0x7f23580ae990 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f23400097c0 tx=0x7f234001c380 comp rx=0 tx=0).stop 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- 172.21.15.70:0/4102891447 shutdown_connections 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b29d0 0x7f235806dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 --2- 172.21.15.70:0/4102891447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580b3db0 0x7f23580ae990 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- 172.21.15.70:0/4102891447 >> 172.21.15.70:0/4102891447 conn(0x7f23580cf9e0 msgr2=0x7f23580cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- 172.21.15.70:0/4102891447 shutdown_connections 2024-04-22T01:19:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- 172.21.15.70:0/4102891447 wait complete. 2024-04-22T01:19:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 Processor -- start 2024-04-22T01:19:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- start start 2024-04-22T01:19:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23580b29d0 0x7f2358159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b3db0 0x7f235814d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 0x7f235814f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f235806f100 con 0x7f235815e000 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f235806ef80 con 0x7f23580b29d0 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f235d4a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f235806f280 con 0x7f23580b3db0 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 0x7f235814f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 0x7f235814f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55554/0 (socket says 172.21.15.70:55554) 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 -- 172.21.15.70:0/1544598684 learned_addr learned my addr 172.21.15.70:0/1544598684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f2356ffd640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23580b29d0 0x7f2358159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 -- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b3db0 msgr2=0x7f235814d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b3db0 0x7f235814d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 -- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23580b29d0 msgr2=0x7f2358159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23580b29d0 0x7f2358159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.727+0000 7f23577fe640 1 -- 172.21.15.70:0/1544598684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235815f550 con 0x7f235815e000 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f23577fe640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 0x7f235814f320 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f235000c820 tx=0x7f235000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350015020 con 0x7f235815e000 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2350004240 con 0x7f235815e000 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235815f830 con 0x7f235815e000 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350003620 con 0x7f235815e000 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235815fdf0 con 0x7f235815e000 2024-04-22T01:19:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235804f510 con 0x7f235815e000 2024-04-22T01:19:10.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2350018020 con 0x7f235815e000 2024-04-22T01:19:10.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f234f7fe640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23380678c0 0x7f2338069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:10.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f2350085e70 con 0x7f235815e000 2024-04-22T01:19:10.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f2356ffd640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23380678c0 0x7f2338069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:10.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.731+0000 7f2356ffd640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23380678c0 0x7f2338069d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f235815f300 tx=0x7f2348009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:10.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.735+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f235001f020 con 0x7f235815e000 2024-04-22T01:19:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.935+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f235806e2d0 con 0x7f235815e000 2024-04-22T01:19:10.935 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-22T01:19:10.935+0000 7fb6cf031640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:19:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.939+0000 7f234f7fe640 1 -- 172.21.15.70:0/1544598684 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) ==== 161+0+0 (secure 0 0 0) 0x7f23500568b0 con 0x7f235815e000 2024-04-22T01:19:10.937 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:19:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.943+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23380678c0 msgr2=0x7f2338069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.943+0000 7f235d4a4640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23380678c0 0x7f2338069d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f235815f300 tx=0x7f2348009040 comp rx=0 tx=0).stop 2024-04-22T01:19:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.943+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 msgr2=0x7f235814f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:10.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.943+0000 7f235d4a4640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 0x7f235814f320 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f235000c820 tx=0x7f235000cce0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 shutdown_connections 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f23380678c0 0x7f2338069d80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f235815e000 0x7f235814f320 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23580b3db0 0x7f235814d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 --2- 172.21.15.70:0/1544598684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23580b29d0 0x7f2358159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 >> 172.21.15.70:0/1544598684 conn(0x7f23580cf9e0 msgr2=0x7f23580b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 shutdown_connections 2024-04-22T01:19:10.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:10.947+0000 7f235d4a4640 1 -- 172.21.15.70:0/1544598684 wait complete. 2024-04-22T01:19:10.963 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:19:10.992 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-22T01:19:11.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 Processor -- start 2024-04-22T01:19:11.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 -- start start 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f150c099d30 0x7f150c09a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f150c0d01c0 0x7f150c0d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f150c0d0c70 con 0x7f150c0b13d0 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f150c0cd380 con 0x7f150c099d30 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f15146bd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f150c0cc7b0 con 0x7f150c0d01c0 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f150c099d30 0x7f150c09a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f150c099d30 0x7f150c09a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55576/0 (socket says 172.21.15.70:55576) 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 -- 172.21.15.70:0/410723159 learned_addr learned my addr 172.21.15.70:0/410723159 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1511c58640 1 --2- 172.21.15.70:0/410723159 >> v2:172.21.15.98:3300/0 conn(0x7f150c0d01c0 0x7f150c0d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:11.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 -- 172.21.15.70:0/410723159 >> v2:172.21.15.98:3300/0 conn(0x7f150c0d01c0 msgr2=0x7f150c0d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:11.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 --2- 172.21.15.70:0/410723159 >> v2:172.21.15.98:3300/0 conn(0x7f150c0d01c0 0x7f150c0d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 -- 172.21.15.70:0/410723159 >> v1:172.21.15.70:6789/0 conn(0x7f150c0b13d0 legacy=0x7f150c0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:11.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.075+0000 7f1512459640 1 -- 172.21.15.70:0/410723159 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c09a650 con 0x7f150c099d30 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512459640 1 --2- 172.21.15.70:0/410723159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c099d30 0x7f150c09a110 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f150c09b650 tx=0x7f150801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10461b7c1454b830 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1511457640 1 -- 172.21.15.70:0/410723159 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15080ab600 con 0x7f150c099d30 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1511457640 1 -- 172.21.15.70:0/410723159 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15080b0070 con 0x7f150c099d30 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1511457640 1 -- 172.21.15.70:0/410723159 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15080ab9c0 con 0x7f150c099d30 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- 172.21.15.70:0/410723159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c099d30 msgr2=0x7f150c09a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 --2- 172.21.15.70:0/410723159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c099d30 0x7f150c09a110 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f150c09b650 tx=0x7f150801c4f0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- 172.21.15.70:0/410723159 shutdown_connections 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 --2- 172.21.15.70:0/410723159 >> v2:172.21.15.98:3300/0 conn(0x7f150c0d01c0 0x7f150c0d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 --2- 172.21.15.70:0/410723159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c099d30 0x7f150c09a110 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- 172.21.15.70:0/410723159 >> 172.21.15.70:0/410723159 conn(0x7f150c0b3700 msgr2=0x7f150c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:11.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- 172.21.15.70:0/410723159 shutdown_connections 2024-04-22T01:19:11.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- 172.21.15.70:0/410723159 wait complete. 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 Processor -- start 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- start start 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f150c099d30 0x7f150c147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f150c0b13d0 0x7f150c165190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 0x7f150c1656d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f150c0cdf20 con 0x7f150c0d01c0 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f150c0b10a0 con 0x7f150c099d30 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f150c0ce0a0 con 0x7f150c0b13d0 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 0x7f150c1656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 0x7f150c1656d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55580/0 (socket says 172.21.15.70:55580) 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 -- 172.21.15.70:0/2434283399 learned_addr learned my addr 172.21.15.70:0/2434283399 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512459640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f150c099d30 0x7f150c147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1511c58640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f150c0b13d0 0x7f150c165190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 -- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f150c0b13d0 msgr2=0x7f150c165190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f150c0b13d0 0x7f150c165190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 -- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f150c099d30 msgr2=0x7f150c147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f150c099d30 0x7f150c147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 -- 172.21.15.70:0/2434283399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c06d0e0 con 0x7f150c0d01c0 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512459640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f150c099d30 0x7f150c147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1512c5a640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 0x7f150c1656d0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f150400c710 tx=0x7f150400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504016020 con 0x7f150c0d01c0 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1504006d60 con 0x7f150c0d01c0 2024-04-22T01:19:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c06d3c0 con 0x7f150c0d01c0 2024-04-22T01:19:11.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.079+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504003bc0 con 0x7f150c0d01c0 2024-04-22T01:19:11.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c06d980 con 0x7f150c0d01c0 2024-04-22T01:19:11.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c068920 con 0x7f150c0d01c0 2024-04-22T01:19:11.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f150400a850 con 0x7f150c0d01c0 2024-04-22T01:19:11.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f1502ffd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f00678c0 0x7f14f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:11.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6187+0+0 (secure 0 0 0) 0x7f1504085ad0 con 0x7f150c0d01c0 2024-04-22T01:19:11.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f1512459640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f00678c0 0x7f14f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:11.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.083+0000 7f1512459640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f00678c0 0x7f14f0069d80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f15080af000 tx=0x7f1508008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:11.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.087+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1504020020 con 0x7f150c0d01c0 2024-04-22T01:19:11.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.267+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f150c158820 con 0x7f150c0d01c0 2024-04-22T01:19:11.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.939+0000 7f1502ffd640 1 -- 172.21.15.70:0/2434283399 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) ==== 104+0+0 (secure 0 0 0) 0x7f15040564c0 con 0x7f150c0d01c0 2024-04-22T01:19:11.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f00678c0 msgr2=0x7f14f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:11.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f00678c0 0x7f14f0069d80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f15080af000 tx=0x7f1508008040 comp rx=0 tx=0).stop 2024-04-22T01:19:11.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 msgr2=0x7f150c1656d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:11.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 0x7f150c1656d0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f150400c710 tx=0x7f150400cbd0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 shutdown_connections 2024-04-22T01:19:11.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f14f00678c0 0x7f14f0069d80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c0d01c0 0x7f150c1656d0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f150c0b13d0 0x7f150c165190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 --2- 172.21.15.70:0/2434283399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f150c099d30 0x7f150c147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:11.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 >> 172.21.15.70:0/2434283399 conn(0x7f150c0b3700 msgr2=0x7f150c0afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:11.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 shutdown_connections 2024-04-22T01:19:11.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:11.943+0000 7f15146bd640 1 -- 172.21.15.70:0/2434283399 wait complete. 2024-04-22T01:19:11.966 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:19:12.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368e590640 1 Processor -- start 2024-04-22T01:19:12.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368e590640 1 -- start start 2024-04-22T01:19:12.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368e590640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f368816d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368e590640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f368806e340 con 0x7f368806f3e0 2024-04-22T01:19:12.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368e590640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f368806e620 con 0x7f3688162350 2024-04-22T01:19:12.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368e590640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f368806e900 con 0x7f3688072280 2024-04-22T01:19:12.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f36877fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f368806f3e0 0x7f3688071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58672/0 (socket says 172.21.15.70:58672) 2024-04-22T01:19:12.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f36877fe640 1 -- 172.21.15.70:0/1700453440 learned_addr learned my addr 172.21.15.70:0/1700453440 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f368cb2d640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f368816d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185005205 0 0) 0x7f368806e340 con 0x7f368806f3e0 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3670003530 con 0x7f368806f3e0 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 815704184 0 0) 0x7f3670003530 con 0x7f368806f3e0 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 msgr2=0x7f368816d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f368816d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 >> v1:172.21.15.98:6789/0 conn(0x7f3688162350 legacy=0x7f368806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368806ebe0 con 0x7f368806f3e0 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f367c001680 con 0x7f368806f3e0 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 >> v1:172.21.15.70:6789/0 conn(0x7f368806f3e0 legacy=0x7f3688071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f36700039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3670004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670004d50 0x7f36700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f368806e900 con 0x7f3670003f70 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f367000bab0 con 0x7f3670004d50 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f367000bcb0 con 0x7f3688072280 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.139+0000 7f36877fe640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3670004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f3687fff640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f36700039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368cb2d640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670004d50 0x7f36700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f36877fe640 1 -- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 msgr2=0x7f36700039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f36877fe640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f36700039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f36877fe640 1 -- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670004d50 msgr2=0x7f36700051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f36877fe640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670004d50 0x7f36700051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f36877fe640 1 -- 172.21.15.70:0/1700453440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3670003530 con 0x7f3670003f70 2024-04-22T01:19:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f3687fff640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688072280 0x7f36700039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:19:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f36877fe640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3670004810 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c240 tx=0x7f367c01f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6aa5427d66863a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c007d20 con 0x7f3670003f70 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f367c005ae0 con 0x7f3670003f70 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f3686ffd640 1 -- 172.21.15.70:0/1700453440 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c007070 con 0x7f3670003f70 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368e590640 1 -- 172.21.15.70:0/1700453440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 msgr2=0x7f3670004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368e590640 1 --2- 172.21.15.70:0/1700453440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3670004810 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c240 tx=0x7f367c01f3f0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368cb2d640 1 -- 172.21.15.70:0/1700453440 reap_dead start 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368e590640 1 -- 172.21.15.70:0/1700453440 shutdown_connections 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368e590640 1 -- 172.21.15.70:0/1700453440 >> 172.21.15.70:0/1700453440 conn(0x7f36880b1d40 msgr2=0x7f36880cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368e590640 1 -- 172.21.15.70:0/1700453440 shutdown_connections 2024-04-22T01:19:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.143+0000 7f368e590640 1 -- 172.21.15.70:0/1700453440 wait complete. 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 Processor -- start 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- start start 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3688164c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3688151140 0x7f36881503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688150b80 0x7f3688152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36880b36f0 con 0x7f3670003f70 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f368806e000 con 0x7f3688151140 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f36880b3870 con 0x7f3688150b80 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3688164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3688164c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55598/0 (socket says 172.21.15.70:55598) 2024-04-22T01:19:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 -- 172.21.15.70:0/1879522084 learned_addr learned my addr 172.21.15.70:0/1879522084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368cb2d640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688150b80 0x7f3688152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f36877fe640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3688151140 0x7f36881503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 -- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688150b80 msgr2=0x7f3688152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688150b80 0x7f3688152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 -- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3688151140 msgr2=0x7f36881503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3688151140 0x7f36881503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 -- 172.21.15.70:0/1879522084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36881f8d30 con 0x7f3670003f70 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f36877fe640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3688151140 0x7f36881503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3687fff640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3688164c50 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f367800e490 tx=0x7f367800e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678019070 con 0x7f3670003f70 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36780052e0 con 0x7f3670003f70 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3688163bb0 con 0x7f3670003f70 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678003840 con 0x7f3670003f70 2024-04-22T01:19:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3688164170 con 0x7f3670003f70 2024-04-22T01:19:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3688068210 con 0x7f3670003f70 2024-04-22T01:19:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.147+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f367801c070 con 0x7f3670003f70 2024-04-22T01:19:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.151+0000 7f3684ff9640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f36740678c0 0x7f3674069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.151+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6187+0+0 (secure 0 0 0) 0x7f3678085480 con 0x7f3670003f70 2024-04-22T01:19:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.151+0000 7f36877fe640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f36740678c0 0x7f3674069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.151+0000 7f36877fe640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f36740678c0 0x7f3674069d80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f367c0af000 tx=0x7f367c0b2040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.155+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f3678055770 con 0x7f3670003f70 2024-04-22T01:19:12.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.315+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3688165b00 con 0x7f3670003f70 2024-04-22T01:19:12.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.947+0000 7f3684ff9640 1 -- 172.21.15.70:0/1879522084 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) ==== 181+0+0 (secure 0 0 0) 0x7f3678059a70 con 0x7f3670003f70 2024-04-22T01:19:12.949 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:19:12.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f36740678c0 msgr2=0x7f3674069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f36740678c0 0x7f3674069d80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f367c0af000 tx=0x7f367c0b2040 comp rx=0 tx=0).stop 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 msgr2=0x7f3688164c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3688164c50 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f367800e490 tx=0x7f367800e950 comp rx=0 tx=0).stop 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 shutdown_connections 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f36740678c0 0x7f3674069d80 secure :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f367c0af000 tx=0x7f367c0b2040 comp rx=0 tx=0).stop 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3688150b80 0x7f3688152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3688151140 0x7f36881503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 --2- 172.21.15.70:0/1879522084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3670003f70 0x7f3688164c50 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:12.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.951+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 >> 172.21.15.70:0/1879522084 conn(0x7f36880b1d40 msgr2=0x7f36880b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:12.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.955+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 shutdown_connections 2024-04-22T01:19:12.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:12.955+0000 7f368e590640 1 -- 172.21.15.70:0/1879522084 wait complete. 2024-04-22T01:19:12.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:19:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 Processor -- start 2024-04-22T01:19:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 -- start start 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8980b4450 0x7fb898071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8980720b0 0x7fb8980ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980cd050 0x7fb89815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb89815c290 con 0x7fb8980b4450 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb89815c570 con 0x7fb8980720b0 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89d244640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb89815c850 con 0x7fb8980cd050 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8980b4450 0x7fb898071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8980b4450 0x7fb898071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55612/0 (socket says 172.21.15.70:55612) 2024-04-22T01:19:13.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 -- 172.21.15.70:0/420092452 learned_addr learned my addr 172.21.15.70:0/420092452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:13.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89659c640 1 --2- 172.21.15.70:0/420092452 >> v2:172.21.15.98:3300/0 conn(0x7fb8980720b0 0x7fb8980ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89759e640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980cd050 0x7fb89815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 -- 172.21.15.70:0/420092452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980cd050 msgr2=0x7fb89815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980cd050 0x7fb89815f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 -- 172.21.15.70:0/420092452 >> v2:172.21.15.98:3300/0 conn(0x7fb8980720b0 msgr2=0x7fb8980ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 --2- 172.21.15.70:0/420092452 >> v2:172.21.15.98:3300/0 conn(0x7fb8980720b0 0x7fb8980ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb896d9d640 1 -- 172.21.15.70:0/420092452 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89815cb30 con 0x7fb8980b4450 2024-04-22T01:19:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.103+0000 7fb89759e640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980cd050 0x7fb89815f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:13.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb896d9d640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980b4450 0x7fb898071b70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb8980cf9a0 tx=0x7fb88801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a435d0f5034047 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:13.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb895d9b640 1 -- 172.21.15.70:0/420092452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b0070 con 0x7fb8980b4450 2024-04-22T01:19:13.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb895d9b640 1 -- 172.21.15.70:0/420092452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8880ab600 con 0x7fb8980b4450 2024-04-22T01:19:13.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb895d9b640 1 -- 172.21.15.70:0/420092452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880ab9d0 con 0x7fb8980b4450 2024-04-22T01:19:13.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 -- 172.21.15.70:0/420092452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980b4450 msgr2=0x7fb898071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:13.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980b4450 0x7fb898071b70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb8980cf9a0 tx=0x7fb88801c260 comp rx=0 tx=0).stop 2024-04-22T01:19:13.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 -- 172.21.15.70:0/420092452 shutdown_connections 2024-04-22T01:19:13.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980cd050 0x7fb89815f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 --2- 172.21.15.70:0/420092452 >> v2:172.21.15.98:3300/0 conn(0x7fb8980720b0 0x7fb8980ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 --2- 172.21.15.70:0/420092452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980b4450 0x7fb898071b70 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 -- 172.21.15.70:0/420092452 >> 172.21.15.70:0/420092452 conn(0x7fb8980b1d40 msgr2=0x7fb8980cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:13.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 -- 172.21.15.70:0/420092452 shutdown_connections 2024-04-22T01:19:13.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 -- 172.21.15.70:0/420092452 wait complete. 2024-04-22T01:19:13.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 Processor -- start 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.107+0000 7fb89d244640 1 -- start start 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 0x7fb89814cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980b4450 0x7fb898151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980cd050 0x7fb89814b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8980cd980 con 0x7fb8980720b0 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8980cd800 con 0x7fb8980cd050 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8980b2a40 con 0x7fb8980b4450 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 0x7fb89814cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 0x7fb89814cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55628/0 (socket says 172.21.15.70:55628) 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 -- 172.21.15.70:0/1239048814 learned_addr learned my addr 172.21.15.70:0/1239048814 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89759e640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980cd050 0x7fb89814b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 -- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980b4450 msgr2=0x7fb898151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:13.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89659c640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980b4450 0x7fb898151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980b4450 0x7fb898151140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 -- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980cd050 msgr2=0x7fb89814b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980cd050 0x7fb89814b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 -- 172.21.15.70:0/1239048814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8981518a0 con 0x7fb8980720b0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89759e640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980cd050 0x7fb89814b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb896d9d640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 0x7fb89814cda0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb8880ae040 tx=0x7fb888019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b0040 con 0x7fb8980720b0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8880b9070 con 0x7fb8980720b0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb898151b80 con 0x7fb8980720b0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b4680 con 0x7fb8980720b0 2024-04-22T01:19:13.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb898152140 con 0x7fb8980720b0 2024-04-22T01:19:13.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb898068210 con 0x7fb8980720b0 2024-04-22T01:19:13.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb8880b4820 con 0x7fb8980720b0 2024-04-22T01:19:13.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb87f7fe640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8840678c0 0x7fb884069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:13.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5749+0+0 (secure 0 0 0) 0x7fb88811fa70 con 0x7fb8980720b0 2024-04-22T01:19:13.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89659c640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8840678c0 0x7fb884069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:13.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.111+0000 7fb89659c640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8840678c0 0x7fb884069d80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb8980cc300 tx=0x7fb88c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:13.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.115+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb888016110 con 0x7fb8980720b0 2024-04-22T01:19:13.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:13.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:13.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.275+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb8980d01e0 con 0x7fb8980720b0 2024-04-22T01:19:13.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.951+0000 7fb87f7fe640 1 -- 172.21.15.70:0/1239048814 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) ==== 169+0+0 (secure 0 0 0) 0x7fb8880f0580 con 0x7fb8980720b0 2024-04-22T01:19:13.951 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-22T01:19:13.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8840678c0 msgr2=0x7fb884069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:13.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8840678c0 0x7fb884069d80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb8980cc300 tx=0x7fb88c008040 comp rx=0 tx=0).stop 2024-04-22T01:19:13.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 msgr2=0x7fb89814cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:13.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 0x7fb89814cda0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb8880ae040 tx=0x7fb888019000 comp rx=0 tx=0).stop 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 shutdown_connections 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb8840678c0 0x7fb884069d80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980cd050 0x7fb89814b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8980b4450 0x7fb898151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 --2- 172.21.15.70:0/1239048814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8980720b0 0x7fb89814cda0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 >> 172.21.15.70:0/1239048814 conn(0x7fb8980b1d40 msgr2=0x7fb898155890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 shutdown_connections 2024-04-22T01:19:13.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:13.959+0000 7fb89d244640 1 -- 172.21.15.70:0/1239048814 wait complete. 2024-04-22T01:19:13.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-22T01:19:14.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 Processor -- start 2024-04-22T01:19:14.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 -- start start 2024-04-22T01:19:14.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b7415a320 0x7f1b7406f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b740ce420 0x7f1b74072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7406da70 con 0x7f1b74071b70 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1b7406dd50 con 0x7f1b7415a320 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7c086640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1b7406e030 con 0x7f1b740ce420 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b79621640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1b74071b70 0x7f1b740b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58680/0 (socket says 172.21.15.70:58680) 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b79621640 1 -- 172.21.15.70:0/2274708438 learned_addr learned my addr 172.21.15.70:0/2274708438 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b79e22640 1 --2- 172.21.15.70:0/2274708438 >> v2:172.21.15.70:3300/0 conn(0x7f1b7415a320 0x7f1b7406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b7a623640 1 --2- 172.21.15.70:0/2274708438 >> v2:172.21.15.98:3300/0 conn(0x7f1b740ce420 0x7f1b74072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b78e20640 1 -- 172.21.15.70:0/2274708438 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980153933 0 0) 0x7f1b7406da70 con 0x7f1b74071b70 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.146+0000 7f1b78e20640 1 -- 172.21.15.70:0/2274708438 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b60003530 con 0x7f1b74071b70 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b79e22640 1 -- 172.21.15.70:0/2274708438 >> v2:172.21.15.98:3300/0 conn(0x7f1b740ce420 msgr2=0x7f1b74072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b79e22640 1 --2- 172.21.15.70:0/2274708438 >> v2:172.21.15.98:3300/0 conn(0x7f1b740ce420 0x7f1b74072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b79e22640 1 -- 172.21.15.70:0/2274708438 >> v1:172.21.15.70:6789/0 conn(0x7f1b74071b70 legacy=0x7f1b740b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b79e22640 1 -- 172.21.15.70:0/2274708438 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7406e310 con 0x7f1b7415a320 2024-04-22T01:19:14.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b79e22640 1 --2- 172.21.15.70:0/2274708438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b7406f150 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1b74071b00 tx=0x7f1b7001d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463756b59d5f3f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:14.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b78e20640 1 -- 172.21.15.70:0/2274708438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b700b3070 con 0x7f1b7415a320 2024-04-22T01:19:14.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b78e20640 1 -- 172.21.15.70:0/2274708438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b700a67b0 con 0x7f1b7415a320 2024-04-22T01:19:14.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b78e20640 1 -- 172.21.15.70:0/2274708438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7000b040 con 0x7f1b7415a320 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 -- 172.21.15.70:0/2274708438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 msgr2=0x7f1b7406f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 --2- 172.21.15.70:0/2274708438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b7406f150 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1b74071b00 tx=0x7f1b7001d850 comp rx=0 tx=0).stop 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 -- 172.21.15.70:0/2274708438 shutdown_connections 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 --2- 172.21.15.70:0/2274708438 >> v2:172.21.15.98:3300/0 conn(0x7f1b740ce420 0x7f1b74072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 --2- 172.21.15.70:0/2274708438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b7406f150 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 -- 172.21.15.70:0/2274708438 >> 172.21.15.70:0/2274708438 conn(0x7f1b740b1d40 msgr2=0x7f1b740cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 -- 172.21.15.70:0/2274708438 shutdown_connections 2024-04-22T01:19:14.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.150+0000 7f1b7c086640 1 -- 172.21.15.70:0/2274708438 wait complete. 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 Processor -- start 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- start start 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b74071b70 0x7f1b74144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b740ce420 0x7f1b74158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b741585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7406f810 con 0x7f1b7415a320 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7406f690 con 0x7f1b74071b70 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1b740b4470 con 0x7f1b740ce420 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b741585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b741585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55650/0 (socket says 172.21.15.70:55650) 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 -- 172.21.15.70:0/1419741883 learned_addr learned my addr 172.21.15.70:0/1419741883 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:14.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b79e22640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b74071b70 0x7f1b74144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 -- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b740ce420 msgr2=0x7f1b74158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b740ce420 0x7f1b74158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 -- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b74071b70 msgr2=0x7f1b74144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b74071b70 0x7f1b74144730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 -- 172.21.15.70:0/1419741883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b741f7d10 con 0x7f1b7415a320 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7a623640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b741585c0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009dd0 tx=0x7f1b6c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c018020 con 0x7f1b7415a320 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b6c014480 con 0x7f1b7415a320 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b740b0f00 con 0x7f1b7415a320 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c014870 con 0x7f1b7415a320 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b741f7fb0 con 0x7f1b7415a320 2024-04-22T01:19:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b74068210 con 0x7f1b7415a320 2024-04-22T01:19:14.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b6c01b020 con 0x7f1b7415a320 2024-04-22T01:19:14.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b6a7fc640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b580678c0 0x7f1b58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f1b6c0865d0 con 0x7f1b7415a320 2024-04-22T01:19:14.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.154+0000 7f1b79e22640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b580678c0 0x7f1b58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.158+0000 7f1b79e22640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b580678c0 0x7f1b58069d80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1b700b0000 tx=0x7f1b7000c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:14.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.158+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f1b6c022070 con 0x7f1b7415a320 2024-04-22T01:19:14.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.318+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f1b7414e610 con 0x7f1b7415a320 2024-04-22T01:19:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.322+0000 7f1b6a7fc640 1 -- 172.21.15.70:0/1419741883 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f1b6c010040 con 0x7f1b7415a320 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b580678c0 msgr2=0x7f1b58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b580678c0 0x7f1b58069d80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1b700b0000 tx=0x7f1b7000c040 comp rx=0 tx=0).stop 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 msgr2=0x7f1b741585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b741585c0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009dd0 tx=0x7f1b6c00c490 comp rx=0 tx=0).stop 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 shutdown_connections 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f1b580678c0 0x7f1b58069d80 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b7415a320 0x7f1b741585c0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b740ce420 0x7f1b74158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 --2- 172.21.15.70:0/1419741883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b74071b70 0x7f1b74144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 >> 172.21.15.70:0/1419741883 conn(0x7f1b740b1d40 msgr2=0x7f1b7406b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 shutdown_connections 2024-04-22T01:19:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.326+0000 7f1b7c086640 1 -- 172.21.15.70:0/1419741883 wait complete. 2024-04-22T01:19:14.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write' 2024-04-22T01:19:14.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.462+0000 7f2e76f25640 1 Processor -- start 2024-04-22T01:19:14.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e76f25640 1 -- start start 2024-04-22T01:19:14.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e76f25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e701455b0 0x7f2e700cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e76f25640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e70072230 0x7f2e7006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e76f25640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e700d0250 con 0x7f2e700cc150 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e76f25640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2e700cd7b0 con 0x7f2e70072230 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e76f25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e700cc710 con 0x7f2e701455b0 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e754c2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e70072230 0x7f2e7006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e6ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2e700cc150 0x7f2e70071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:58692/0 (socket says 172.21.15.70:58692) 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e6ffff640 1 -- 172.21.15.70:0/762344940 learned_addr learned my addr 172.21.15.70:0/762344940 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e754c2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e70072230 0x7f2e7006e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55658/0 (socket says 172.21.15.70:55658) 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e74cc1640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e701455b0 0x7f2e700cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e754c2640 1 -- 172.21.15.70:0/762344940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e701455b0 msgr2=0x7f2e700cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e754c2640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e701455b0 0x7f2e700cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e754c2640 1 -- 172.21.15.70:0/762344940 >> v1:172.21.15.70:6789/0 conn(0x7f2e700cc150 legacy=0x7f2e70071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e754c2640 1 -- 172.21.15.70:0/762344940 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e700cc9f0 con 0x7f2e70072230 2024-04-22T01:19:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e6f7fe640 1 -- 172.21.15.70:0/762344940 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120049153 0 0) 0x7f2e700d0250 con 0x7f2e700cc150 2024-04-22T01:19:14.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.466+0000 7f2e74cc1640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e701455b0 0x7f2e700cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:14.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e754c2640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e70072230 0x7f2e7006e200 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e6801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112f3604a04a3e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:14.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e6f7fe640 1 -- 172.21.15.70:0/762344940 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680b2070 con 0x7f2e70072230 2024-04-22T01:19:14.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e6f7fe640 1 -- 172.21.15.70:0/762344940 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e680a5710 con 0x7f2e70072230 2024-04-22T01:19:14.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e6f7fe640 1 -- 172.21.15.70:0/762344940 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680a5ae0 con 0x7f2e70072230 2024-04-22T01:19:14.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- 172.21.15.70:0/762344940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e70072230 msgr2=0x7f2e7006e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e70072230 0x7f2e7006e200 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e6801c850 comp rx=0 tx=0).stop 2024-04-22T01:19:14.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- 172.21.15.70:0/762344940 shutdown_connections 2024-04-22T01:19:14.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e70072230 0x7f2e7006e200 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 --2- 172.21.15.70:0/762344940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e701455b0 0x7f2e700cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- 172.21.15.70:0/762344940 >> 172.21.15.70:0/762344940 conn(0x7f2e7006b1f0 msgr2=0x7f2e7006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- 172.21.15.70:0/762344940 shutdown_connections 2024-04-22T01:19:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- 172.21.15.70:0/762344940 wait complete. 2024-04-22T01:19:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 Processor -- start 2024-04-22T01:19:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- start start 2024-04-22T01:19:14.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e70072230 0x7f2e700b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e700cc150 0x7f2e700af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 0x7f2e7013c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7006f2e0 con 0x7f2e701455b0 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7006f160 con 0x7f2e700cc150 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e76f25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2e70168b40 con 0x7f2e70072230 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e754c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 0x7f2e7013c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e754c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 0x7f2e7013c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55666/0 (socket says 172.21.15.70:55666) 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e754c2640 1 -- 172.21.15.70:0/4241515812 learned_addr learned my addr 172.21.15.70:0/4241515812 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e6ffff640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e700cc150 0x7f2e700af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e74cc1640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e70072230 0x7f2e700b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e754c2640 1 -- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e70072230 msgr2=0x7f2e700b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.470+0000 7f2e754c2640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e70072230 0x7f2e700b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e754c2640 1 -- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e700cc150 msgr2=0x7f2e700af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e754c2640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e700cc150 0x7f2e700af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e754c2640 1 -- 172.21.15.70:0/4241515812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e700afd20 con 0x7f2e701455b0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6ffff640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e700cc150 0x7f2e700af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e754c2640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 0x7f2e7013c020 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e680195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680b2050 con 0x7f2e701455b0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e680be070 con 0x7f2e701455b0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e700b0000 con 0x7f2e701455b0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680ada90 con 0x7f2e701455b0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e700b05c0 con 0x7f2e701455b0 2024-04-22T01:19:14.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e70068210 con 0x7f2e701455b0 2024-04-22T01:19:14.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e680a5590 con 0x7f2e701455b0 2024-04-22T01:19:14.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6d7fa640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e5c0678c0 0x7f2e5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:14.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f2e68120b90 con 0x7f2e701455b0 2024-04-22T01:19:14.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e74cc1640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e5c0678c0 0x7f2e5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:14.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.474+0000 7f2e74cc1640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e5c0678c0 0x7f2e5c069d80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2e60006fd0 tx=0x7f2e60008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:14.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.478+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2e68016560 con 0x7f2e701455b0 2024-04-22T01:19:14.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.698+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]} v 0) -- 0x7f2e7015f510 con 0x7f2e701455b0 2024-04-22T01:19:14.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.962+0000 7f2e6d7fa640 1 -- 172.21.15.70:0/4241515812 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]}]=0 v266) ==== 137+0+0 (secure 0 0 0) 0x7f2e680ab340 con 0x7f2e701455b0 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e5c0678c0 msgr2=0x7f2e5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e5c0678c0 0x7f2e5c069d80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2e60006fd0 tx=0x7f2e60008040 comp rx=0 tx=0).stop 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 msgr2=0x7f2e7013c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 0x7f2e7013c020 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e680195f0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 shutdown_connections 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2e5c0678c0 0x7f2e5c069d80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e701455b0 0x7f2e7013c020 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e700cc150 0x7f2e700af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 --2- 172.21.15.70:0/4241515812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e70072230 0x7f2e700b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 >> 172.21.15.70:0/4241515812 conn(0x7f2e7006b1f0 msgr2=0x7f2e7013fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 shutdown_connections 2024-04-22T01:19:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:14.966+0000 7f2e76f25640 1 -- 172.21.15.70:0/4241515812 wait complete. 2024-04-22T01:19:14.987 INFO:tasks.cephfs_test_runner:That when a client modifies the quota on a directory used ... ok 2024-04-22T01:19:14.987 INFO:tasks.cephfs_test_runner: 2024-04-22T01:19:14.988 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-22T01:19:14.988 INFO:tasks.cephfs_test_runner:Ran 6 tests in 216.959s 2024-04-22T01:19:14.988 INFO:tasks.cephfs_test_runner: 2024-04-22T01:19:14.988 INFO:tasks.cephfs_test_runner:OK 2024-04-22T01:19:14.988 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-22T01:19:14.998 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-22T01:19:15.016 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-22T01:19:15.016 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:19:15.016 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:15.022 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:19:15.022 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-22T01:19:15.022 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:15.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336460k,nr_inodes=4084115,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16220 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274676k,nr_inodes=818669,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274680k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-22T01:19:15.070 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:15.070 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:15.077 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:19:15.077 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T01:19:15.077 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:19:15.126 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336504k,nr_inodes=4084126,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16230 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-22T01:19:15.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.126 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-22T01:19:15.132 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-22T01:19:15.132 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:19:15.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-22T01:19:15.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:19:15.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-04-22T01:19:15.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.435 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T01:19:15.441 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-22T01:19:15.441 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-22T01:19:15.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-22T01:19:15.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T01:19:15.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:19:15.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-22T01:19:15.716 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T01:19:15.726 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T01:19:15.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:19:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 Processor -- start 2024-04-22T01:19:15.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 -- start start 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c1455b0 0x7f085c0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f085c0cc150 0x7f085c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f085c0d0250 con 0x7f085c072060 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f085c0724d0 con 0x7f085c0cc150 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0861eb0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f085c0cd7b0 con 0x7f085c1455b0 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f085c0cc150 0x7f085c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f085c0cc150 0x7f085c071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55674/0 (socket says 172.21.15.70:55674) 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 -- 172.21.15.70:0/2915405704 learned_addr learned my addr 172.21.15.70:0/2915405704 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:15.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 -- 172.21.15.70:0/2915405704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c1455b0 msgr2=0x7f085c0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 --2- 172.21.15.70:0/2915405704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c1455b0 0x7f085c0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 -- 172.21.15.70:0/2915405704 >> v1:172.21.15.98:6789/0 conn(0x7f085c072060 legacy=0x7f085c15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:15.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 -- 172.21.15.70:0/2915405704 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c0cc710 con 0x7f085c0cc150 2024-04-22T01:19:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.858+0000 7f0852ffd640 1 --2- 172.21.15.70:0/2915405704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c0cc150 0x7f085c071b20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f0844009870 tx=0x7f084401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23db0a447392ad61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085affd640 1 -- 172.21.15.70:0/2915405704 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440b0070 con 0x7f085c0cc150 2024-04-22T01:19:15.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085affd640 1 -- 172.21.15.70:0/2915405704 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08440ab600 con 0x7f085c0cc150 2024-04-22T01:19:15.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085affd640 1 -- 172.21.15.70:0/2915405704 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440ab9d0 con 0x7f085c0cc150 2024-04-22T01:19:15.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2915405704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c0cc150 msgr2=0x7f085c071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:15.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2915405704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c0cc150 0x7f085c071b20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f0844009870 tx=0x7f084401c400 comp rx=0 tx=0).stop 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2915405704 shutdown_connections 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2915405704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c0cc150 0x7f085c071b20 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2915405704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c1455b0 0x7f085c0cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2915405704 >> 172.21.15.70:0/2915405704 conn(0x7f085c06b1f0 msgr2=0x7f085c06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2915405704 shutdown_connections 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2915405704 wait complete. 2024-04-22T01:19:15.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 Processor -- start 2024-04-22T01:19:15.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- start start 2024-04-22T01:19:15.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085c072060 0x7f085c1573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c0cc150 0x7f085c143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 0x7f085c143d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f085c168b40 con 0x7f085c1455b0 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f085c06f210 con 0x7f085c072060 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f085c168cc0 con 0x7f085c0cc150 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 0x7f085c143d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 0x7f085c143d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55682/0 (socket says 172.21.15.70:55682) 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 -- 172.21.15.70:0/2074636369 learned_addr learned my addr 172.21.15.70:0/2074636369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085b7fe640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085c072060 0x7f085c1573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0852ffd640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c0cc150 0x7f085c143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 -- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c0cc150 msgr2=0x7f085c143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c0cc150 0x7f085c143820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 -- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085c072060 msgr2=0x7f085c1573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085c072060 0x7f085c1573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 -- 172.21.15.70:0/2074636369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c141f50 con 0x7f085c1455b0 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f085bfff640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 0x7f085c143d60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f085400b410 tx=0x7f085400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854015020 con 0x7f085c1455b0 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0854011830 con 0x7f085c1455b0 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854011c20 con 0x7f085c1455b0 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085c141960 con 0x7f085c1455b0 2024-04-22T01:19:15.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.862+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085c1424d0 con 0x7f085c1455b0 2024-04-22T01:19:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.866+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085c068210 con 0x7f085c1455b0 2024-04-22T01:19:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.866+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0854018020 con 0x7f085c1455b0 2024-04-22T01:19:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.866+0000 7f0858ff9640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f083c0678c0 0x7f083c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.866+0000 7f085b7fe640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f083c0678c0 0x7f083c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.866+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f08540859c0 con 0x7f085c1455b0 2024-04-22T01:19:15.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.866+0000 7f085b7fe640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f083c0678c0 0x7f083c069d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f084c009f20 tx=0x7f084c0094f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:15.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:15.870+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f0854087e20 con 0x7f085c1455b0 2024-04-22T01:19:16.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.038+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f085c072770 con 0x7f083c0678c0 2024-04-22T01:19:16.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0858ff9640 1 -- 172.21.15.70:0/2074636369 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13505 (secure 0 0 0) 0x7f085c072770 con 0x7f083c0678c0 2024-04-22T01:19:16.041 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:16.041 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:19:16.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f083c0678c0 msgr2=0x7f083c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f083c0678c0 0x7f083c069d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f084c009f20 tx=0x7f084c0094f0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 msgr2=0x7f085c143d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 0x7f085c143d60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f085400b410 tx=0x7f085400b8d0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 shutdown_connections 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f083c0678c0 0x7f083c069d80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f085c1455b0 0x7f085c143d60 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f085c0cc150 0x7f085c143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 --2- 172.21.15.70:0/2074636369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f085c072060 0x7f085c1573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 >> 172.21.15.70:0/2074636369 conn(0x7f085c06b1f0 msgr2=0x7f085c143040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 shutdown_connections 2024-04-22T01:19:16.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.042+0000 7f0861eb0640 1 -- 172.21.15.70:0/2074636369 wait complete. 2024-04-22T01:19:16.060 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-04-22T01:19:14.665025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282712,"kb_used_data":16624,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692008,"statfs":{"total":773094113280,"available":772804616192,"internally_reserved":0,"allocated":17022976,"data_stored":11231838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2770109,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-221,"num_write_kb":-23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001781"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":320,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T01:19:11.414006+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:19:11.414006+0000","last_peered":"2024-04-22T01:19:11.414006+0000","last_clean":"2024-04-22T01:19:11.414006+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:19:11.414006+0000","last_undegraded":"2024-04-22T01:19:11.414006+0000","last_fullsized":"2024-04-22T01:19:11.414006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1186881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35184,"kb_used_data":4004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336656,"statfs":{"total":96636764160,"available":96600735744,"internally_reserved":0,"allocated":4100096,"data_stored":3366237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":602746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":596445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1302528,"data_stored":596926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":602742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39056,"kb_used_data":3716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332784,"statfs":{"total":96636764160,"available":96596770816,"internally_reserved":0,"allocated":3805184,"data_stored":3086752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":1193109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:19:16.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:19:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.174+0000 7f2ccb9f1640 1 Processor -- start 2024-04-22T01:19:16.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.174+0000 7f2ccb9f1640 1 -- start start 2024-04-22T01:19:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2ccb9f1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc40cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2ccb9f1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2cc40cc150 0x7f2cc4071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2ccb9f1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc40d0250 con 0x7f2cc4072060 2024-04-22T01:19:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2ccb9f1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc40724d0 con 0x7f2cc40cc150 2024-04-22T01:19:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2ccb9f1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc40cd7b0 con 0x7f2cc41455b0 2024-04-22T01:19:16.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2cc40cc150 0x7f2cc4071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2cc40cc150 0x7f2cc4071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45066/0 (socket says 172.21.15.70:45066) 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/1530994402 learned_addr learned my addr 172.21.15.70:0/1530994402 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc978d640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc40cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/1530994402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 msgr2=0x7f2cc40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc40cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/1530994402 >> v1:172.21.15.98:6789/0 conn(0x7f2cc4072060 legacy=0x7f2cc415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/1530994402 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc40cc710 con 0x7f2cc40cc150 2024-04-22T01:19:16.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc978d640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:19:16.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cc8f8c640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc4071b20 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4009870 tx=0x7f2cb401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfa95393acf12c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cbbfff640 1 -- 172.21.15.70:0/1530994402 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40b0070 con 0x7f2cc40cc150 2024-04-22T01:19:16.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cbbfff640 1 -- 172.21.15.70:0/1530994402 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb40ab600 con 0x7f2cc40cc150 2024-04-22T01:19:16.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.178+0000 7f2cbbfff640 1 -- 172.21.15.70:0/1530994402 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40ab9d0 con 0x7f2cc40cc150 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/1530994402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 msgr2=0x7f2cc4071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc4071b20 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4009870 tx=0x7f2cb401c440 comp rx=0 tx=0).stop 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/1530994402 shutdown_connections 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc4071b20 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/1530994402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/1530994402 >> 172.21.15.70:0/1530994402 conn(0x7f2cc406b1f0 msgr2=0x7f2cc406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/1530994402 shutdown_connections 2024-04-22T01:19:16.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/1530994402 wait complete. 2024-04-22T01:19:16.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 Processor -- start 2024-04-22T01:19:16.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- start start 2024-04-22T01:19:16.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc4072060 0x7f2cc415a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc41437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc4168b40 con 0x7f2cc40cc150 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc406f210 con 0x7f2cc4072060 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc4168cc0 con 0x7f2cc41455b0 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45082/0 (socket says 172.21.15.70:45082) 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/2341495231 learned_addr learned my addr 172.21.15.70:0/2341495231 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc978d640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc4072060 0x7f2cc415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc9f8e640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc41437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 msgr2=0x7f2cc41437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc41437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc4072060 msgr2=0x7f2cc415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc4072060 0x7f2cc415a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 -- 172.21.15.70:0/2341495231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc415bc60 con 0x7f2cc40cc150 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc9f8e640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc41437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cc8f8c640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc415ccb0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40ae040 tx=0x7f2cb40191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40b8020 con 0x7f2cc40cc150 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb40b4410 con 0x7f2cc40cc150 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc415c1f0 con 0x7f2cc40cc150 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40b0040 con 0x7f2cc40cc150 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.182+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc4144860 con 0x7f2cc40cc150 2024-04-22T01:19:16.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.186+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc4068210 con 0x7f2cc40cc150 2024-04-22T01:19:16.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.186+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2cb40bb020 con 0x7f2cc40cc150 2024-04-22T01:19:16.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.186+0000 7f2cb9ffb640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ca80678c0 0x7f2ca8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.186+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f2cb411f390 con 0x7f2cc40cc150 2024-04-22T01:19:16.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.186+0000 7f2cc978d640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ca80678c0 0x7f2ca8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.186+0000 7f2cc978d640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ca80678c0 0x7f2ca8069d80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000d8b0 tx=0x7f2cc0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.190+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f2cb40c2070 con 0x7f2cc40cc150 2024-04-22T01:19:16.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.350+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2cc415f510 con 0x7f2ca80678c0 2024-04-22T01:19:16.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/2341495231 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13505 (secure 0 0 0) 0x7f2cc415f510 con 0x7f2ca80678c0 2024-04-22T01:19:16.353 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:16.353 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ca80678c0 msgr2=0x7f2ca8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ca80678c0 0x7f2ca8069d80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000d8b0 tx=0x7f2cc0008040 comp rx=0 tx=0).stop 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 msgr2=0x7f2cc415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc415ccb0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40ae040 tx=0x7f2cb40191e0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 shutdown_connections 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f2ca80678c0 0x7f2ca8069d80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2cc41455b0 0x7f2cc41437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cc40cc150 0x7f2cc415ccb0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 --2- 172.21.15.70:0/2341495231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc4072060 0x7f2cc415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 >> 172.21.15.70:0/2341495231 conn(0x7f2cc406b1f0 msgr2=0x7f2cc4164c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 shutdown_connections 2024-04-22T01:19:16.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.354+0000 7f2ccb9f1640 1 -- 172.21.15.70:0/2341495231 wait complete. 2024-04-22T01:19:16.371 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-04-22T01:19:14.665025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282712,"kb_used_data":16624,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692008,"statfs":{"total":773094113280,"available":772804616192,"internally_reserved":0,"allocated":17022976,"data_stored":11231838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2770109,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-221,"num_write_kb":-23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001781"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":320,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T01:19:11.414006+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:19:11.414006+0000","last_peered":"2024-04-22T01:19:11.414006+0000","last_clean":"2024-04-22T01:19:11.414006+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:19:11.414006+0000","last_undegraded":"2024-04-22T01:19:11.414006+0000","last_fullsized":"2024-04-22T01:19:11.414006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1186881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35184,"kb_used_data":4004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336656,"statfs":{"total":96636764160,"available":96600735744,"internally_reserved":0,"allocated":4100096,"data_stored":3366237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":602746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":596445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1302528,"data_stored":596926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":602742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39056,"kb_used_data":3716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332784,"statfs":{"total":96636764160,"available":96596770816,"internally_reserved":0,"allocated":3805184,"data_stored":3086752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":1193109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:19:16.371 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T01:19:16.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:19:16.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.478+0000 7f81a8d05640 1 Processor -- start 2024-04-22T01:19:16.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a8d05640 1 -- start start 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a8d05640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81a41455b0 0x7f81a40cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a8d05640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a40d0250 con 0x7f81a40cc150 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a8d05640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a40cd7b0 con 0x7f81a4072230 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a8d05640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f81a40cc710 con 0x7f81a41455b0 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f81a40cc150 0x7f81a4071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36486/0 (socket says 172.21.15.70:36486) 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 -- 172.21.15.70:0/3328478279 learned_addr learned my addr 172.21.15.70:0/3328478279 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096703292 0 0) 0x7f81a40d0250 con 0x7f81a40cc150 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a259c640 1 --2- 172.21.15.70:0/3328478279 >> v2:172.21.15.98:3300/0 conn(0x7f81a41455b0 0x7f81a40cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8190003530 con 0x7f81a40cc150 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2768888605 0 0) 0x7f8190003530 con 0x7f81a40cc150 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 >> v2:172.21.15.98:3300/0 conn(0x7f81a41455b0 msgr2=0x7f81a40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 --2- 172.21.15.70:0/3328478279 >> v2:172.21.15.98:3300/0 conn(0x7f81a41455b0 0x7f81a40cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 >> v1:172.21.15.98:6789/0 conn(0x7f81a4072230 legacy=0x7f81a406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a40cc9f0 con 0x7f81a40cc150 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f8194001680 con 0x7f81a40cc150 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a259c640 1 --2- 172.21.15.70:0/3328478279 >> v2:172.21.15.98:3300/0 conn(0x7f81a41455b0 0x7f81a40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 >> v1:172.21.15.70:6789/0 conn(0x7f81a40cc150 legacy=0x7f81a4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900039c0 0x7f8190003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8190004810 0x7f8190004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81900053d0 0x7f819000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81a40cc710 con 0x7f8190004810 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f819000c260 con 0x7f81900039c0 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f819000c460 con 0x7f81900053d0 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8190004810 0x7f8190004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a259c640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900039c0 0x7f8190003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 -- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81900053d0 msgr2=0x7f819000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a2d9d640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81900053d0 0x7f819000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81900053d0 0x7f819000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 -- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900039c0 msgr2=0x7f8190003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900039c0 0x7f8190003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f8199d9b640 1 -- 172.21.15.70:0/3328478279 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8190003530 con 0x7f8190004810 2024-04-22T01:19:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.482+0000 7f81a2d9d640 1 -- 172.21.15.70:0/3328478279 reap_dead start 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f8199d9b640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8190004810 0x7f8190004c90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f819400c210 tx=0x7f819401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84afaa1abed50ece server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194007d40 con 0x7f8190004810 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81940a9070 con 0x7f8190004810 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a1d9b640 1 -- 172.21.15.70:0/3328478279 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194005d80 con 0x7f8190004810 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- 172.21.15.70:0/3328478279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8190004810 msgr2=0x7f8190004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8190004810 0x7f8190004c90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f819400c210 tx=0x7f819401cca0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- 172.21.15.70:0/3328478279 shutdown_connections 2024-04-22T01:19:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 --2- 172.21.15.70:0/3328478279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8190004810 0x7f8190004c90 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- 172.21.15.70:0/3328478279 >> 172.21.15.70:0/3328478279 conn(0x7f81a406b1f0 msgr2=0x7f81a406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- 172.21.15.70:0/3328478279 shutdown_connections 2024-04-22T01:19:16.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- 172.21.15.70:0/3328478279 wait complete. 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 Processor -- start 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- start start 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8190004810 0x7f81a41f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a41f81c0 0x7f81a41f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 0x7f81a415de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81a4160130 con 0x7f81a41f73c0 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81a415ffb0 con 0x7f81a41f81c0 2024-04-22T01:19:16.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a8d05640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f81a41602b0 con 0x7f8190004810 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 0x7f81a415de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 0x7f81a415de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45102/0 (socket says 172.21.15.70:45102) 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.486+0000 7f81a2d9d640 1 -- 172.21.15.70:0/2482810510 learned_addr learned my addr 172.21.15.70:0/2482810510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f8199d9b640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a41f81c0 0x7f81a41f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a259c640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8190004810 0x7f81a41f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a2d9d640 1 -- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8190004810 msgr2=0x7f81a41f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a2d9d640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8190004810 0x7f81a41f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a2d9d640 1 -- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a41f81c0 msgr2=0x7f81a41f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a2d9d640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a41f81c0 0x7f81a41f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a2d9d640 1 -- 172.21.15.70:0/2482810510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a40b3f00 con 0x7f81a41f73c0 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a2d9d640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 0x7f81a415de10 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f81a40b1580 tx=0x7f819c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c018070 con 0x7f81a41f73c0 2024-04-22T01:19:16.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f819c003b80 con 0x7f81a41f73c0 2024-04-22T01:19:16.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c005070 con 0x7f81a41f73c0 2024-04-22T01:19:16.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a41f8d30 con 0x7f81a41f73c0 2024-04-22T01:19:16.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a41f92f0 con 0x7f81a41f73c0 2024-04-22T01:19:16.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a4068210 con 0x7f81a41f73c0 2024-04-22T01:19:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f819c01b070 con 0x7f81a41f73c0 2024-04-22T01:19:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f819b7fe640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81840678c0 0x7f8184069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a259c640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81840678c0 0x7f8184069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f819c085130 con 0x7f81a41f73c0 2024-04-22T01:19:16.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.490+0000 7f81a259c640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81840678c0 0x7f8184069d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f81a40b4bd0 tx=0x7f818c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.494+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f819c055e70 con 0x7f81a41f73c0 2024-04-22T01:19:16.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.654+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81a415bc10 con 0x7f81840678c0 2024-04-22T01:19:16.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.654+0000 7f819b7fe640 1 -- 172.21.15.70:0/2482810510 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13505 (secure 0 0 0) 0x7f81a415bc10 con 0x7f81840678c0 2024-04-22T01:19:16.656 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:16.657 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:19:16.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81840678c0 msgr2=0x7f8184069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81840678c0 0x7f8184069d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f81a40b4bd0 tx=0x7f818c008040 comp rx=0 tx=0).stop 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 msgr2=0x7f81a415de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 0x7f81a415de10 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f81a40b1580 tx=0x7f819c00db80 comp rx=0 tx=0).stop 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 shutdown_connections 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f81840678c0 0x7f8184069d80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a41f73c0 0x7f81a415de10 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81a41f81c0 0x7f81a41f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 --2- 172.21.15.70:0/2482810510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8190004810 0x7f81a41f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 >> 172.21.15.70:0/2482810510 conn(0x7f81a406b1f0 msgr2=0x7f81a406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 shutdown_connections 2024-04-22T01:19:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.658+0000 7f81a8d05640 1 -- 172.21.15.70:0/2482810510 wait complete. 2024-04-22T01:19:16.675 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-04-22T01:19:14.665025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282712,"kb_used_data":16624,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692008,"statfs":{"total":773094113280,"available":772804616192,"internally_reserved":0,"allocated":17022976,"data_stored":11231838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2770109,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-221,"num_write_kb":-23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001781"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":320,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T01:19:11.414006+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:19:11.414006+0000","last_peered":"2024-04-22T01:19:11.414006+0000","last_clean":"2024-04-22T01:19:11.414006+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:19:11.414006+0000","last_undegraded":"2024-04-22T01:19:11.414006+0000","last_fullsized":"2024-04-22T01:19:11.414006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1186881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35184,"kb_used_data":4004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336656,"statfs":{"total":96636764160,"available":96600735744,"internally_reserved":0,"allocated":4100096,"data_stored":3366237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":602746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1327104,"data_stored":596445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1302528,"data_stored":596926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":602742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39056,"kb_used_data":3716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332784,"statfs":{"total":96636764160,"available":96596770816,"internally_reserved":0,"allocated":3805184,"data_stored":3086752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":1193109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:19:16.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:19:16.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 Processor -- start 2024-04-22T01:19:16.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 -- start start 2024-04-22T01:19:16.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0940159380 0x7f09400b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 0x7f09400cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0940162a60 con 0x7f0940163500 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f09400b3f40 con 0x7f0940159380 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0947886640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0940162d00 con 0x7f09400b3a50 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945e23640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0940163500 0x7f094006e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36502/0 (socket says 172.21.15.70:36502) 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945e23640 1 -- 172.21.15.70:0/1358061258 learned_addr learned my addr 172.21.15.70:0/1358061258 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 --2- 172.21.15.70:0/1358061258 >> v2:172.21.15.98:3300/0 conn(0x7f0940159380 0x7f09400b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0944e21640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 0x7f09400cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193812873 0 0) 0x7f0940162a60 con 0x7f0940163500 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f092c003530 con 0x7f0940163500 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3133754034 0 0) 0x7f092c003530 con 0x7f0940163500 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 msgr2=0x7f09400cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 0x7f09400cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 >> v2:172.21.15.98:3300/0 conn(0x7f0940159380 msgr2=0x7f09400b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 --2- 172.21.15.70:0/1358061258 >> v2:172.21.15.98:3300/0 conn(0x7f0940159380 0x7f09400b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0940162fe0 con 0x7f0940163500 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f0934001680 con 0x7f0940163500 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 >> v1:172.21.15.70:6789/0 conn(0x7f0940163500 legacy=0x7f094006e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092c0039c0 0x7f092c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092c004810 0x7f092c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 0x7f092c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09400b3f40 con 0x7f092c0039c0 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0940162d00 con 0x7f092c004810 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f092c00b9d0 con 0x7f09400b3a50 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092c0039c0 0x7f092c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945e23640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 0x7f092c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0944e21640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092c004810 0x7f092c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 -- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 msgr2=0x7f092c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400b3a50 0x7f092c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 -- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092c004810 msgr2=0x7f092c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:16.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f092c004810 0x7f092c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.790+0000 7f0945622640 1 -- 172.21.15.70:0/1358061258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092c003530 con 0x7f092c0039c0 2024-04-22T01:19:16.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0945622640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092c0039c0 0x7f092c003de0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f093800e170 tx=0x7f093801cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b37ec4e9c3da12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938009c10 con 0x7f092c0039c0 2024-04-22T01:19:16.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09380a5b60 con 0x7f092c0039c0 2024-04-22T01:19:16.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0933fff640 1 -- 172.21.15.70:0/1358061258 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09380ad3f0 con 0x7f092c0039c0 2024-04-22T01:19:16.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- 172.21.15.70:0/1358061258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092c0039c0 msgr2=0x7f092c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 --2- 172.21.15.70:0/1358061258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f092c0039c0 0x7f092c003de0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f093800e170 tx=0x7f093801cc50 comp rx=0 tx=0).stop 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0945e23640 1 -- 172.21.15.70:0/1358061258 reap_dead start 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- 172.21.15.70:0/1358061258 shutdown_connections 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- 172.21.15.70:0/1358061258 >> 172.21.15.70:0/1358061258 conn(0x7f094006b1f0 msgr2=0x7f094006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- 172.21.15.70:0/1358061258 shutdown_connections 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- 172.21.15.70:0/1358061258 wait complete. 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 Processor -- start 2024-04-22T01:19:16.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- start start 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 0x7f09401fba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09401fa8e0 0x7f09400cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940165300 0x7f0940165780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09400cf750 con 0x7f09401fd890 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0940166e60 con 0x7f0940165300 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0947886640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f09400cf8d0 con 0x7f09401fa8e0 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0945622640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 0x7f09401fba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0945622640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 0x7f09401fba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45120/0 (socket says 172.21.15.70:45120) 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.794+0000 7f0945622640 1 -- 172.21.15.70:0/3508248557 learned_addr learned my addr 172.21.15.70:0/3508248557 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945e23640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940165300 0x7f0940165780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0944e21640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09401fa8e0 0x7f09400cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945622640 1 -- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09401fa8e0 msgr2=0x7f09400cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945622640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09401fa8e0 0x7f09400cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945622640 1 -- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940165300 msgr2=0x7f0940165780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945622640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940165300 0x7f0940165780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945622640 1 -- 172.21.15.70:0/3508248557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09401fdc70 con 0x7f09401fd890 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0944e21640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09401fa8e0 0x7f09400cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945e23640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940165300 0x7f0940165780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0945622640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 0x7f09401fba90 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f093801c710 tx=0x7f0938009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938009c10 con 0x7f09401fd890 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09380be070 con 0x7f09401fd890 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09401fae80 con 0x7f09401fd890 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09380adc10 con 0x7f09401fd890 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09401fb440 con 0x7f09401fd890 2024-04-22T01:19:16.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0940068210 con 0x7f09401fd890 2024-04-22T01:19:16.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f09380b8470 con 0x7f09401fd890 2024-04-22T01:19:16.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0931ffb640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09240678c0 0x7f0924069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:16.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f0938121010 con 0x7f09401fd890 2024-04-22T01:19:16.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0944e21640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09240678c0 0x7f0924069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:16.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.798+0000 7f0944e21640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09240678c0 0x7f0924069d80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0940164260 tx=0x7f093c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:16.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.802+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f09380c8020 con 0x7f09401fd890 2024-04-22T01:19:16.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.966+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f094006e040 con 0x7f09401fd890 2024-04-22T01:19:16.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.970+0000 7f0931ffb640 1 -- 172.21.15.70:0/3508248557 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+13267 (secure 0 0 0) 0x7f09401fdc70 con 0x7f09401fd890 2024-04-22T01:19:16.971 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:16.971 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":106,"fsid":"b126cadd-d35d-4fad-9fb6-c51f861f8653","created":"2024-04-22T01:14:22.033212+0000","modified":"2024-04-22T01:19:13.951085+0000","last_up_change":"2024-04-22T01:14:33.127410+0000","last_in_change":"2024-04-22T01:14:24.343579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:14:34.662032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50e61bd7-7279-4366-9afa-06822339ded6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6811","nonce":3468450268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6815","nonce":3468450268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6821","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6823","nonce":3468450268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6817","nonce":3468450268},{"type":"v1","addr":"172.21.15.70:6819","nonce":3468450268}]},"public_addr":"172.21.15.70:6811/3468450268","cluster_addr":"172.21.15.70:6815/3468450268","heartbeat_back_addr":"172.21.15.70:6823/3468450268","heartbeat_front_addr":"172.21.15.70:6819/3468450268","state":["exists","up"]},{"osd":1,"uuid":"21575948-910b-45c7-903d-7cb907f90dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6801","nonce":2050830481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6803","nonce":2050830481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6807","nonce":2050830481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2050830481},{"type":"v1","addr":"172.21.15.70:6805","nonce":2050830481}]},"public_addr":"172.21.15.70:6801/2050830481","cluster_addr":"172.21.15.70:6803/2050830481","heartbeat_back_addr":"172.21.15.70:6807/2050830481","heartbeat_front_addr":"172.21.15.70:6805/2050830481","state":["exists","up"]},{"osd":2,"uuid":"78c44d13-1d12-40d9-88b3-6df647d83dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6825","nonce":3866658519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6827","nonce":3866658519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6831","nonce":3866658519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3866658519},{"type":"v1","addr":"172.21.15.70:6829","nonce":3866658519}]},"public_addr":"172.21.15.70:6825/3866658519","cluster_addr":"172.21.15.70:6827/3866658519","heartbeat_back_addr":"172.21.15.70:6831/3866658519","heartbeat_front_addr":"172.21.15.70:6829/3866658519","state":["exists","up"]},{"osd":3,"uuid":"bd245b3d-f0f8-4d36-bfc5-6e1b46ef214e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6810","nonce":1842545244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6814","nonce":1842545244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6822","nonce":1842545244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1842545244},{"type":"v1","addr":"172.21.15.70:6818","nonce":1842545244}]},"public_addr":"172.21.15.70:6810/1842545244","cluster_addr":"172.21.15.70:6814/1842545244","heartbeat_back_addr":"172.21.15.70:6822/1842545244","heartbeat_front_addr":"172.21.15.70:6818/1842545244","state":["exists","up"]},{"osd":4,"uuid":"310bb8df-5bcd-4966-9e02-98bcd61e596c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6805","nonce":3056066255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6807","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6809","nonce":3056066255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6815","nonce":3056066255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3056066255},{"type":"v1","addr":"172.21.15.98:6812","nonce":3056066255}]},"public_addr":"172.21.15.98:6805/3056066255","cluster_addr":"172.21.15.98:6809/3056066255","heartbeat_back_addr":"172.21.15.98:6815/3056066255","heartbeat_front_addr":"172.21.15.98:6812/3056066255","state":["exists","up"]},{"osd":5,"uuid":"d9619c39-d326-4500-a754-a2fbe3a2070f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6817","nonce":3651416447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6819","nonce":3651416447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6823","nonce":3651416447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3651416447},{"type":"v1","addr":"172.21.15.98:6821","nonce":3651416447}]},"public_addr":"172.21.15.98:6817/3651416447","cluster_addr":"172.21.15.98:6819/3651416447","heartbeat_back_addr":"172.21.15.98:6823/3651416447","heartbeat_front_addr":"172.21.15.98:6821/3651416447","state":["exists","up"]},{"osd":6,"uuid":"e8096831-1495-4589-aefc-91205b3cb8c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6801","nonce":3535628973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6803","nonce":3535628973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6813","nonce":3535628973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3535628973},{"type":"v1","addr":"172.21.15.98:6808","nonce":3535628973}]},"public_addr":"172.21.15.98:6801/3535628973","cluster_addr":"172.21.15.98:6803/3535628973","heartbeat_back_addr":"172.21.15.98:6813/3535628973","heartbeat_front_addr":"172.21.15.98:6808/3535628973","state":["exists","up"]},{"osd":7,"uuid":"620872a8-a49b-46ea-853a-b83de44daa0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6825","nonce":2897852499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6827","nonce":2897852499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6831","nonce":2897852499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2897852499},{"type":"v1","addr":"172.21.15.98:6829","nonce":2897852499}]},"public_addr":"172.21.15.98:6825/2897852499","cluster_addr":"172.21.15.98:6827/2897852499","heartbeat_back_addr":"172.21.15.98:6831/2897852499","heartbeat_front_addr":"172.21.15.98:6829/2897852499","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.963551+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.870133+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.993572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.891515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.805030+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.817754+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.970583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:14:29.887949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:6837/3578750329":"2024-04-23T01:19:10.937899+0000","172.21.15.70:6836/3578750329":"2024-04-23T01:19:10.937899+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09240678c0 msgr2=0x7f0924069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09240678c0 0x7f0924069d80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0940164260 tx=0x7f093c008040 comp rx=0 tx=0).stop 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 msgr2=0x7f09401fba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 0x7f09401fba90 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f093801c710 tx=0x7f0938009750 comp rx=0 tx=0).stop 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 shutdown_connections 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f09240678c0 0x7f0924069d80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940165300 0x7f0940165780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09401fa8e0 0x7f09400cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 --2- 172.21.15.70:0/3508248557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09401fd890 0x7f09401fba90 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 >> 172.21.15.70:0/3508248557 conn(0x7f094006b1f0 msgr2=0x7f094006e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 shutdown_connections 2024-04-22T01:19:16.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:16.974+0000 7f0947886640 1 -- 172.21.15.70:0/3508248557 wait complete. 2024-04-22T01:19:17.992 INFO:tasks.ceph:Scrubbing osd.0 2024-04-22T01:19:17.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:18.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 Processor -- start 2024-04-22T01:19:18.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 -- start start 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 0x7f8df815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8df80ce920 0x7f8df8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df815f000 con 0x7f8df80ceda0 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8df80b3800 con 0x7f8df80ce920 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfec49640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8df80b3ae0 con 0x7f8df815e6e0 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfd1e6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8df80ceda0 0x7f8df806d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36506/0 (socket says 172.21.15.70:36506) 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2145473825 learned_addr learned my addr 172.21.15.70:0/2145473825 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2145473825 >> v2:172.21.15.98:3300/0 conn(0x7f8df80ce920 0x7f8df8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8dfc9e5640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 0x7f8df815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 849163052 0 0) 0x7f8df815f000 con 0x7f8df80ceda0 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8de0003530 con 0x7f8df80ceda0 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3769679393 0 0) 0x7f8de0003530 con 0x7f8df80ceda0 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 msgr2=0x7f8df815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 0x7f8df815eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 >> v2:172.21.15.98:3300/0 conn(0x7f8df80ce920 msgr2=0x7f8df8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 --2- 172.21.15.70:0/2145473825 >> v2:172.21.15.98:3300/0 conn(0x7f8df80ce920 0x7f8df8154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df80cde40 con 0x7f8df80ceda0 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f8df0001a00 con 0x7f8df80ceda0 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 >> v1:172.21.15.70:6789/0 conn(0x7f8df80ceda0 legacy=0x7f8df806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 0x7f8de00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de0003f40 0x7f8de0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de0004d50 0x7f8de00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8df80b3800 con 0x7f8de0003f40 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8df80b3ae0 con 0x7f8de0004d50 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.126+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8de000dbe0 con 0x7f8df815e6e0 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de0003f40 0x7f8de0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfc9e5640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 0x7f8de00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfd1e6640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de0004d50 0x7f8de00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 -- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 msgr2=0x7f8de00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8df815e6e0 0x7f8de00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 -- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de0004d50 msgr2=0x7f8de00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de0004d50 0x7f8de00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 -- 172.21.15.70:0/2145473825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de0003530 con 0x7f8de0003f40 2024-04-22T01:19:18.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de0003f40 0x7f8de0004810 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009ae0 tx=0x7f8dec01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5aa7f40426a9846 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0b2070 con 0x7f8de0003f40 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dec0a5590 con 0x7f8de0003f40 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8df77fe640 1 -- 172.21.15.70:0/2145473825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0096b0 con 0x7f8de0003f40 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfec49640 1 -- 172.21.15.70:0/2145473825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de0003f40 msgr2=0x7f8de0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfec49640 1 --2- 172.21.15.70:0/2145473825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de0003f40 0x7f8de0004810 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009ae0 tx=0x7f8dec01ca50 comp rx=0 tx=0).stop 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2145473825 reap_dead start 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfec49640 1 -- 172.21.15.70:0/2145473825 shutdown_connections 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfec49640 1 -- 172.21.15.70:0/2145473825 >> 172.21.15.70:0/2145473825 conn(0x7f8df80b1d40 msgr2=0x7f8df80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfec49640 1 -- 172.21.15.70:0/2145473825 shutdown_connections 2024-04-22T01:19:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.130+0000 7f8dfec49640 1 -- 172.21.15.70:0/2145473825 wait complete. 2024-04-22T01:19:18.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 Processor -- start 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- start start 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8de0003f40 0x7f8df8154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8153b30 0x7f8df80ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8df814c380 0x7f8df815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8df806f0e0 con 0x7f8df814c380 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8df80ce400 con 0x7f8df8153b30 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8df806f260 con 0x7f8de0003f40 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8df814c380 0x7f8df815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8df814c380 0x7f8df815b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45144/0 (socket says 172.21.15.70:45144) 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2623496654 learned_addr learned my addr 172.21.15.70:0/2623496654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8153b30 0x7f8df80ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfc9e5640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8de0003f40 0x7f8df8154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8de0003f40 msgr2=0x7f8df8154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8de0003f40 0x7f8df8154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8153b30 msgr2=0x7f8df80ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8153b30 0x7f8df80ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2623496654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df80aef50 con 0x7f8df814c380 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8153b30 0x7f8df80ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfd1e6640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8df814c380 0x7f8df815b0f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f8df0002070 tx=0x7f8df0011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0010040 con 0x7f8df814c380 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df001a070 con 0x7f8df814c380 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df8154270 con 0x7f8df814c380 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df000fb60 con 0x7f8df814c380 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df80aec40 con 0x7f8df814c380 2024-04-22T01:19:18.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8df81540c0 con 0x7f8df814c380 2024-04-22T01:19:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.134+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8df000fd00 con 0x7f8df814c380 2024-04-22T01:19:18.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df57fa640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de40678c0 0x7f8de4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8dfc9e5640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de40678c0 0x7f8de4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f8df00895c0 con 0x7f8df814c380 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df57fa640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7f8de406dd60 0x7f8de40701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 --> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8de4070870 con 0x7f8de406dd60 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f8df0089a50 con 0x7f8df814c380 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7f8de406dd60 0x7f8de40701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8dfc9e5640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de40678c0 0x7f8de4069d80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009890 tx=0x7f8de8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df7fff640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7f8de406dd60 0x7f8de40701a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.138+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== osd.0 v2:172.21.15.70:6809/3468450268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8de4070870 con 0x7f8de406dd60 2024-04-22T01:19:18.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 --> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8df8068210 con 0x7f8de406dd60 2024-04-22T01:19:18.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8df57fa640 1 -- 172.21.15.70:0/2623496654 <== osd.0 v2:172.21.15.70:6809/3468450268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8df8068210 con 0x7f8de406dd60 2024-04-22T01:19:18.152 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:18.152 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:18.152 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7f8de406dd60 msgr2=0x7f8de40701a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6809/3468450268,v1:172.21.15.70:6811/3468450268] conn(0x7f8de406dd60 0x7f8de40701a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de40678c0 msgr2=0x7f8de4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f8de40678c0 0x7f8de4069d80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009890 tx=0x7f8de8008040 comp rx=0 tx=0).stop 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8df814c380 msgr2=0x7f8df815b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 --2- 172.21.15.70:0/2623496654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8df814c380 0x7f8df815b0f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f8df0002070 tx=0x7f8df0011c20 comp rx=0 tx=0).stop 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfd1e6640 1 -- 172.21.15.70:0/2623496654 reap_dead start 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 shutdown_connections 2024-04-22T01:19:18.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 >> 172.21.15.70:0/2623496654 conn(0x7f8df80b1d40 msgr2=0x7f8df80b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 shutdown_connections 2024-04-22T01:19:18.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.150+0000 7f8dfec49640 1 -- 172.21.15.70:0/2623496654 wait complete. 2024-04-22T01:19:18.168 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-22T01:19:18.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe0641f8640 1 Processor -- start 2024-04-22T01:19:18.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe0641f8640 1 -- start start 2024-04-22T01:19:18.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe0641f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe05c162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe0641f8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c06b450 con 0x7fe05c161b90 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe0641f8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c0b77d0 con 0x7fe05c1615a0 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe0641f8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe05c162550 con 0x7fe05c0cf3e0 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe05c161b90 0x7fe05c0cecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36512/0 (socket says 172.21.15.70:36512) 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 -- 172.21.15.70:0/860840038 learned_addr learned my addr 172.21.15.70:0/860840038 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe062795640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe05c162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333918389 0 0) 0x7fe05c06b450 con 0x7fe05c161b90 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe048003530 con 0x7fe05c161b90 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1218587110 0 0) 0x7fe048003530 con 0x7fe05c161b90 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 msgr2=0x7fe05c162010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe05c162010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 >> v1:172.21.15.98:6789/0 conn(0x7fe05c1615a0 legacy=0x7fe05c0b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c0ce830 con 0x7fe05c161b90 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fe04c001680 con 0x7fe05c161b90 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 >> v1:172.21.15.70:6789/0 conn(0x7fe05c161b90 legacy=0x7fe05c0cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe048003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe048004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048004d50 0x7fe0480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c162550 con 0x7fe048003fe0 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe04800bb30 con 0x7fe048004d50 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe04800bd30 con 0x7fe05c0cf3e0 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe048004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061f94640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe048003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe062795640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048004d50 0x7fe0480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 -- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 msgr2=0x7fe048003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe048003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 -- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048004d50 msgr2=0x7fe0480051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048004d50 0x7fe0480051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061793640 1 -- 172.21.15.70:0/860840038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe048003530 con 0x7fe048003fe0 2024-04-22T01:19:18.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe062795640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048004d50 0x7fe0480051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:18.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.282+0000 7fe061f94640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c0cf3e0 0x7fe048003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:18.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe061793640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe048004810 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00c210 tx=0x7fe04c01f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d67964121e8e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04c007d40 con 0x7fe048003fe0 2024-04-22T01:19:18.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe04c005b00 con 0x7fe048003fe0 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe060f92640 1 -- 172.21.15.70:0/860840038 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04c007070 con 0x7fe048003fe0 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 -- 172.21.15.70:0/860840038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 msgr2=0x7fe048004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 --2- 172.21.15.70:0/860840038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe048004810 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00c210 tx=0x7fe04c01f330 comp rx=0 tx=0).stop 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe062795640 1 -- 172.21.15.70:0/860840038 reap_dead start 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 -- 172.21.15.70:0/860840038 shutdown_connections 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 -- 172.21.15.70:0/860840038 >> 172.21.15.70:0/860840038 conn(0x7fe05c06c910 msgr2=0x7fe05c06ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 -- 172.21.15.70:0/860840038 shutdown_connections 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 -- 172.21.15.70:0/860840038 wait complete. 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 Processor -- start 2024-04-22T01:19:18.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.286+0000 7fe0641f8640 1 -- start start 2024-04-22T01:19:18.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe05c0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c15abd0 0x7fe05c0b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c15b100 0x7fe05c0b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c0d24a0 con 0x7fe048003fe0 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c161980 con 0x7fe05c15b100 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe05c162770 con 0x7fe05c15abd0 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe05c0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe05c0c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45164/0 (socket says 172.21.15.70:45164) 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 -- 172.21.15.70:0/4208458294 learned_addr learned my addr 172.21.15.70:0/4208458294 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe062795640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c15b100 0x7fe05c0b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061793640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c15abd0 0x7fe05c0b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 -- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c15abd0 msgr2=0x7fe05c0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c15abd0 0x7fe05c0b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 -- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c15b100 msgr2=0x7fe05c0b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c15b100 0x7fe05c0b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 -- 172.21.15.70:0/4208458294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c1e7ce0 con 0x7fe048003fe0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061793640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c15abd0 0x7fe05c0b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe062795640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c15b100 0x7fe05c0b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061f94640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe05c0c95b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe05800d7a0 tx=0x7fe05800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058005270 con 0x7fe048003fe0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe058017070 con 0x7fe048003fe0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05800b650 con 0x7fe048003fe0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe05c1e7fc0 con 0x7fe048003fe0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe05c0b4df0 con 0x7fe048003fe0 2024-04-22T01:19:18.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05c04f510 con 0x7fe048003fe0 2024-04-22T01:19:18.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe05800b7f0 con 0x7fe048003fe0 2024-04-22T01:19:18.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0527fc640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe0400678f0 0x7fe040069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7fe0580859b0 con 0x7fe048003fe0 2024-04-22T01:19:18.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.290+0000 7fe061793640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe0400678f0 0x7fe040069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.294+0000 7fe061793640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe0400678f0 0x7fe040069db0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0af000 tx=0x7fe04c0aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.294+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fe05c0b77d0 con 0x7fe048003fe0 2024-04-22T01:19:18.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe05c0d1b90 con 0x7fe0400678f0 2024-04-22T01:19:18.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0527fc640 1 -- 172.21.15.70:0/4208458294 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe05c0d1b90 con 0x7fe0400678f0 2024-04-22T01:19:18.454 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 0 to deep-scrub 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe0400678f0 msgr2=0x7fe040069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe0400678f0 0x7fe040069db0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0af000 tx=0x7fe04c0aa000 comp rx=0 tx=0).stop 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 msgr2=0x7fe05c0c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe05c0c95b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe05800d7a0 tx=0x7fe05800dc60 comp rx=0 tx=0).stop 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 shutdown_connections 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fe0400678f0 0x7fe040069db0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c15b100 0x7fe05c0b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe05c15abd0 0x7fe05c0b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 --2- 172.21.15.70:0/4208458294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe048003fe0 0x7fe05c0c95b0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 >> 172.21.15.70:0/4208458294 conn(0x7fe05c06c910 msgr2=0x7fe05c0cb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 shutdown_connections 2024-04-22T01:19:18.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.454+0000 7fe0641f8640 1 -- 172.21.15.70:0/4208458294 wait complete. 2024-04-22T01:19:18.471 INFO:tasks.ceph:Scrubbing osd.1 2024-04-22T01:19:18.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.582+0000 7f3403a8d640 1 Processor -- start 2024-04-22T01:19:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.582+0000 7f3403a8d640 1 -- start start 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.582+0000 7f3403a8d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0cdef0 0x7f33fc06b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.582+0000 7f3403a8d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33fc0d15a0 0x7f33fc06b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33fc0d0770 con 0x7f33fc0d0390 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f33fc0ccec0 con 0x7f33fc0d15a0 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0d0ac0 con 0x7f33fc0cdef0 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33fc0d15a0 0x7f33fc06b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f33fc0d15a0 0x7f33fc06b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45174/0 (socket says 172.21.15.70:45174) 2024-04-22T01:19:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 -- 172.21.15.70:0/376067513 learned_addr learned my addr 172.21.15.70:0/376067513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3401028640 1 --2- 172.21.15.70:0/376067513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0cdef0 0x7f33fc06b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 -- 172.21.15.70:0/376067513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0cdef0 msgr2=0x7f33fc06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 --2- 172.21.15.70:0/376067513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0cdef0 0x7f33fc06b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 -- 172.21.15.70:0/376067513 >> v1:172.21.15.98:6789/0 conn(0x7f33fc0d0390 legacy=0x7f33fc0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 -- 172.21.15.70:0/376067513 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc0d0da0 con 0x7f33fc0d15a0 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f340202a640 1 --2- 172.21.15.70:0/376067513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0d15a0 0x7f33fc06b730 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009870 tx=0x7f33f801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eced0b60c95b708e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3400827640 1 -- 172.21.15.70:0/376067513 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f80b0070 con 0x7f33fc0d15a0 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3400827640 1 -- 172.21.15.70:0/376067513 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f80ab600 con 0x7f33fc0d15a0 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3400827640 1 -- 172.21.15.70:0/376067513 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f80ab9d0 con 0x7f33fc0d15a0 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- 172.21.15.70:0/376067513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0d15a0 msgr2=0x7f33fc06b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 --2- 172.21.15.70:0/376067513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0d15a0 0x7f33fc06b730 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009870 tx=0x7f33f801c490 comp rx=0 tx=0).stop 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- 172.21.15.70:0/376067513 shutdown_connections 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 --2- 172.21.15.70:0/376067513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0d15a0 0x7f33fc06b730 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 --2- 172.21.15.70:0/376067513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0cdef0 0x7f33fc06b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- 172.21.15.70:0/376067513 >> 172.21.15.70:0/376067513 conn(0x7f33fc099a60 msgr2=0x7f33fc099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- 172.21.15.70:0/376067513 shutdown_connections 2024-04-22T01:19:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.586+0000 7f3403a8d640 1 -- 172.21.15.70:0/376067513 wait complete. 2024-04-22T01:19:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 Processor -- start 2024-04-22T01:19:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- start start 2024-04-22T01:19:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0cdef0 0x7f33fc14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0d0390 0x7f33fc14dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0d15a0 0x7f33fc0b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc15ce80 con 0x7f33fc0cdef0 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc15cd00 con 0x7f33fc0d0390 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc15d000 con 0x7f33fc0d15a0 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0cdef0 0x7f33fc14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0cdef0 0x7f33fc14e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45188/0 (socket says 172.21.15.70:45188) 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 -- 172.21.15.70:0/82060924 learned_addr learned my addr 172.21.15.70:0/82060924 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f340202a640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0d15a0 0x7f33fc0b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401028640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0d0390 0x7f33fc14dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 -- 172.21.15.70:0/82060924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0d15a0 msgr2=0x7f33fc0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0d15a0 0x7f33fc0b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 -- 172.21.15.70:0/82060924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0d0390 msgr2=0x7f33fc14dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc0d0390 0x7f33fc14dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 -- 172.21.15.70:0/82060924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc14edf0 con 0x7f33fc0cdef0 2024-04-22T01:19:18.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3401829640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0cdef0 0x7f33fc14e260 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f33f400ba80 tx=0x7f33f400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f400cc00 con 0x7f33fc0cdef0 2024-04-22T01:19:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f400cda0 con 0x7f33fc0cdef0 2024-04-22T01:19:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4003bc0 con 0x7f33fc0cdef0 2024-04-22T01:19:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc14f0d0 con 0x7f33fc0cdef0 2024-04-22T01:19:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc14f690 con 0x7f33fc0cdef0 2024-04-22T01:19:18.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f33fc068920 con 0x7f33fc0cdef0 2024-04-22T01:19:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.590+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f33f4006d10 con 0x7f33fc0cdef0 2024-04-22T01:19:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f33f27fc640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33e00678c0 0x7f33e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f33f40863a0 con 0x7f33fc0cdef0 2024-04-22T01:19:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f3401028640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33e00678c0 0x7f33e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f33f27fc640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7f33e006dd60 0x7f33e00701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 --> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33e0070870 con 0x7f33e006dd60 2024-04-22T01:19:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f33f4086830 con 0x7f33fc0cdef0 2024-04-22T01:19:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f340202a640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7f33e006dd60 0x7f33e00701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f3401028640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33e00678c0 0x7f33e0069d80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f33fc1fa6c0 tx=0x7f33ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f340202a640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7f33e006dd60 0x7f33e00701a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.594+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== osd.1 v2:172.21.15.70:6800/2050830481 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33e0070870 con 0x7f33e006dd60 2024-04-22T01:19:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 --> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f33fc0cab90 con 0x7f33e006dd60 2024-04-22T01:19:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f33f27fc640 1 -- 172.21.15.70:0/82060924 <== osd.1 v2:172.21.15.70:6800/2050830481 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f33fc0cab90 con 0x7f33e006dd60 2024-04-22T01:19:18.607 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:18.607 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:18.607 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7f33e006dd60 msgr2=0x7f33e00701a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6800/2050830481,v1:172.21.15.70:6801/2050830481] conn(0x7f33e006dd60 0x7f33e00701a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33e00678c0 msgr2=0x7f33e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f33e00678c0 0x7f33e0069d80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f33fc1fa6c0 tx=0x7f33ec008040 comp rx=0 tx=0).stop 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0cdef0 msgr2=0x7f33fc14e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 --2- 172.21.15.70:0/82060924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33fc0cdef0 0x7f33fc14e260 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f33f400ba80 tx=0x7f33f400bf40 comp rx=0 tx=0).stop 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f340202a640 1 -- 172.21.15.70:0/82060924 reap_dead start 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 shutdown_connections 2024-04-22T01:19:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 >> 172.21.15.70:0/82060924 conn(0x7f33fc099a60 msgr2=0x7f33fc154210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 shutdown_connections 2024-04-22T01:19:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.606+0000 7f3403a8d640 1 -- 172.21.15.70:0/82060924 wait complete. 2024-04-22T01:19:18.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-22T01:19:18.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 Processor -- start 2024-04-22T01:19:18.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 -- start start 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0b4ee0 0x7f752c06b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f752c0b52c0 0x7f752c0b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f752c15ee00 con 0x7f752c0af440 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f752c06fcf0 con 0x7f752c0b52c0 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7530f70640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f752c06e400 con 0x7f752c0b4ee0 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f752ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f752c0b52c0 0x7f752c0b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7529d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f752c0af440 0x7f752c0b2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36514/0 (socket says 172.21.15.70:36514) 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.734+0000 7f7529d9b640 1 -- 172.21.15.70:0/2924311096 learned_addr learned my addr 172.21.15.70:0/2924311096 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752a59c640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0b4ee0 0x7f752c06b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752ad9d640 1 -- 172.21.15.70:0/2924311096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0b4ee0 msgr2=0x7f752c06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:18.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752ad9d640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0b4ee0 0x7f752c06b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752959a640 1 -- 172.21.15.70:0/2924311096 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863970822 0 0) 0x7f752c15ee00 con 0x7f752c0af440 2024-04-22T01:19:18.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752ad9d640 1 -- 172.21.15.70:0/2924311096 >> v1:172.21.15.70:6789/0 conn(0x7f752c0af440 legacy=0x7f752c0b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752ad9d640 1 -- 172.21.15.70:0/2924311096 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c15b780 con 0x7f752c0b52c0 2024-04-22T01:19:18.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752a59c640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0b4ee0 0x7f752c06b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:18.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752ad9d640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c0b5750 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f751c0097c0 tx=0x7f751c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34e4e3266aa3dce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752959a640 1 -- 172.21.15.70:0/2924311096 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0b0070 con 0x7f752c0b52c0 2024-04-22T01:19:18.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752959a640 1 -- 172.21.15.70:0/2924311096 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751c0ab600 con 0x7f752c0b52c0 2024-04-22T01:19:18.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f752959a640 1 -- 172.21.15.70:0/2924311096 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0ab9d0 con 0x7f752c0b52c0 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 -- 172.21.15.70:0/2924311096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 msgr2=0x7f752c0b5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c0b5750 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f751c0097c0 tx=0x7f751c01c4a0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 -- 172.21.15.70:0/2924311096 shutdown_connections 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c0b5750 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 --2- 172.21.15.70:0/2924311096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0b4ee0 0x7f752c06b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 -- 172.21.15.70:0/2924311096 >> 172.21.15.70:0/2924311096 conn(0x7f752c0b3c00 msgr2=0x7f752c0b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 -- 172.21.15.70:0/2924311096 shutdown_connections 2024-04-22T01:19:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.738+0000 7f7530f70640 1 -- 172.21.15.70:0/2924311096 wait complete. 2024-04-22T01:19:18.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 Processor -- start 2024-04-22T01:19:18.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- start start 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0af440 0x7f752c15d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f752c0b4ee0 0x7f752c14de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c166260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f752c0b5c90 con 0x7f752c0b52c0 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f752c15e8d0 con 0x7f752c0b4ee0 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f752c0b2ba0 con 0x7f752c0af440 2024-04-22T01:19:18.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c166260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c166260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45208/0 (socket says 172.21.15.70:45208) 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 -- 172.21.15.70:0/3344729219 learned_addr learned my addr 172.21.15.70:0/3344729219 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752a59c640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0af440 0x7f752c15d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7529d9b640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f752c0b4ee0 0x7f752c14de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 -- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0af440 msgr2=0x7f752c15d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0af440 0x7f752c15d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 -- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f752c0b4ee0 msgr2=0x7f752c14de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f752c0b4ee0 0x7f752c14de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 -- 172.21.15.70:0/3344729219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c14e3c0 con 0x7f752c0b52c0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f752ad9d640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c166260 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f751c0ae040 tx=0x7f751c01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0b0040 con 0x7f752c0b52c0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751c004030 con 0x7f752c0b52c0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0043e0 con 0x7f752c0b52c0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c15bfe0 con 0x7f752c0b52c0 2024-04-22T01:19:18.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c15c5a0 con 0x7f752c0b52c0 2024-04-22T01:19:18.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752c04f510 con 0x7f752c0b52c0 2024-04-22T01:19:18.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.742+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f751c003620 con 0x7f752c0b52c0 2024-04-22T01:19:18.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.746+0000 7f7522ffd640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f750c0678c0 0x7f750c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:18.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.746+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f751c120320 con 0x7f752c0b52c0 2024-04-22T01:19:18.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.746+0000 7f752a59c640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f750c0678c0 0x7f750c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:18.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.746+0000 7f752a59c640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f750c0678c0 0x7f750c069d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f751400a9a0 tx=0x7f7514008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:18.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.750+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f751c016090 con 0x7f752c0b52c0 2024-04-22T01:19:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.894+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f752c0cd830 con 0x7f750c0678c0 2024-04-22T01:19:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7522ffd640 1 -- 172.21.15.70:0/3344729219 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f752c0cd830 con 0x7f750c0678c0 2024-04-22T01:19:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 1 to deep-scrub 2024-04-22T01:19:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f750c0678c0 msgr2=0x7f750c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f750c0678c0 0x7f750c069d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f751400a9a0 tx=0x7f7514008040 comp rx=0 tx=0).stop 2024-04-22T01:19:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 msgr2=0x7f752c166260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c166260 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f751c0ae040 tx=0x7f751c01ace0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 shutdown_connections 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f750c0678c0 0x7f750c069d80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f752c0b52c0 0x7f752c166260 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f752c0b4ee0 0x7f752c14de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 --2- 172.21.15.70:0/3344729219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f752c0af440 0x7f752c15d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 >> 172.21.15.70:0/3344729219 conn(0x7f752c0b3c00 msgr2=0x7f752c159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 shutdown_connections 2024-04-22T01:19:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:18.898+0000 7f7530f70640 1 -- 172.21.15.70:0/3344729219 wait complete. 2024-04-22T01:19:18.914 INFO:tasks.ceph:Scrubbing osd.2 2024-04-22T01:19:18.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:19.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 Processor -- start 2024-04-22T01:19:19.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 -- start start 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31c015f3e0 0x7f31c0099d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31c00cd280 0x7f31c009a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31c00c5720 con 0x7f31c00b43e0 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f31c00cd710 con 0x7f31c00cd280 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31c51cd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f31c00cd9f0 con 0x7f31c015f3e0 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31c00cd280 0x7f31c009a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31c00cd280 0x7f31c009a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45222/0 (socket says 172.21.15.70:45222) 2024-04-22T01:19:19.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 -- 172.21.15.70:0/1230276113 learned_addr learned my addr 172.21.15.70:0/1230276113 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 -- 172.21.15.70:0/1230276113 >> v2:172.21.15.98:3300/0 conn(0x7f31c015f3e0 msgr2=0x7f31c0099d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31bed9d640 1 --2- 172.21.15.70:0/1230276113 >> v2:172.21.15.98:3300/0 conn(0x7f31c015f3e0 0x7f31c0099d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 --2- 172.21.15.70:0/1230276113 >> v2:172.21.15.98:3300/0 conn(0x7f31c015f3e0 0x7f31c0099d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 -- 172.21.15.70:0/1230276113 >> v1:172.21.15.98:6789/0 conn(0x7f31c00b43e0 legacy=0x7f31c00c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.042+0000 7f31be59c640 1 -- 172.21.15.70:0/1230276113 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c00cdcd0 con 0x7f31c00cd280 2024-04-22T01:19:19.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31be59c640 1 --2- 172.21.15.70:0/1230276113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c00cd280 0x7f31c009a270 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f31c00733f0 tx=0x7f31b401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9c4c7c9655954d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31bdd9b640 1 -- 172.21.15.70:0/1230276113 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b40aa430 con 0x7f31c00cd280 2024-04-22T01:19:19.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31bdd9b640 1 -- 172.21.15.70:0/1230276113 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b40af070 con 0x7f31c00cd280 2024-04-22T01:19:19.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31bdd9b640 1 -- 172.21.15.70:0/1230276113 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b40aa7f0 con 0x7f31c00cd280 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 -- 172.21.15.70:0/1230276113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c00cd280 msgr2=0x7f31c009a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 --2- 172.21.15.70:0/1230276113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c00cd280 0x7f31c009a270 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f31c00733f0 tx=0x7f31b401c4b0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 -- 172.21.15.70:0/1230276113 shutdown_connections 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 --2- 172.21.15.70:0/1230276113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c00cd280 0x7f31c009a270 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 --2- 172.21.15.70:0/1230276113 >> v2:172.21.15.98:3300/0 conn(0x7f31c015f3e0 0x7f31c0099d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 -- 172.21.15.70:0/1230276113 >> 172.21.15.70:0/1230276113 conn(0x7f31c00b36c0 msgr2=0x7f31c00b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 -- 172.21.15.70:0/1230276113 shutdown_connections 2024-04-22T01:19:19.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 -- 172.21.15.70:0/1230276113 wait complete. 2024-04-22T01:19:19.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 Processor -- start 2024-04-22T01:19:19.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.046+0000 7f31c51cd640 1 -- start start 2024-04-22T01:19:19.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31c00b43e0 0x7f31c00b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31c015f3e0 0x7f31c0159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c0151f00 0x7f31c0152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31c00c6f10 con 0x7f31c0151f00 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31c00c6d90 con 0x7f31c00b43e0 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31c0168b30 con 0x7f31c015f3e0 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c0151f00 0x7f31c0152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c0151f00 0x7f31c0152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45232/0 (socket says 172.21.15.70:45232) 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 -- 172.21.15.70:0/2568744537 learned_addr learned my addr 172.21.15.70:0/2568744537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bed9d640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31c00b43e0 0x7f31c00b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 -- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31c015f3e0 msgr2=0x7f31c0159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31be59c640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31c015f3e0 0x7f31c0159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31c015f3e0 0x7f31c0159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 -- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31c00b43e0 msgr2=0x7f31c00b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31c00b43e0 0x7f31c00b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 -- 172.21.15.70:0/2568744537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c01528d0 con 0x7f31c0151f00 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bed9d640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31c00b43e0 0x7f31c00b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:19.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bf59e640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c0151f00 0x7f31c0152390 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c780 tx=0x7f31b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8004b70 con 0x7f31c0151f00 2024-04-22T01:19:19.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b8004d10 con 0x7f31c0151f00 2024-04-22T01:19:19.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c0152ee0 con 0x7f31c0151f00 2024-04-22T01:19:19.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800aa90 con 0x7f31c0151f00 2024-04-22T01:19:19.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c01534a0 con 0x7f31c0151f00 2024-04-22T01:19:19.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f31c0068920 con 0x7f31c0151f00 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f31b800ac30 con 0x7f31c0151f00 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31ac0678f0 0x7f31ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bed9d640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31ac0678f0 0x7f31ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f31b80864a0 con 0x7f31c0151f00 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f31ac06dd90 0x7f31ac0701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 --> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31b8005830 con 0x7f31ac06dd90 2024-04-22T01:19:19.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f31b8086930 con 0x7f31c0151f00 2024-04-22T01:19:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31be59c640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f31ac06dd90 0x7f31ac0701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31bed9d640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31ac0678f0 0x7f31ac069db0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f31b0006fd0 tx=0x7f31b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.050+0000 7f31be59c640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f31ac06dd90 0x7f31ac0701d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.054+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== osd.2 v2:172.21.15.70:6824/3866658519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31b8005830 con 0x7f31ac06dd90 2024-04-22T01:19:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 --> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f31c00cf2e0 con 0x7f31ac06dd90 2024-04-22T01:19:19.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31a77fe640 1 -- 172.21.15.70:0/2568744537 <== osd.2 v2:172.21.15.70:6824/3866658519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f31c00cf2e0 con 0x7f31ac06dd90 2024-04-22T01:19:19.066 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:19.066 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:19.066 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f31ac06dd90 msgr2=0x7f31ac0701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6824/3866658519,v1:172.21.15.70:6825/3866658519] conn(0x7f31ac06dd90 0x7f31ac0701d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31ac0678f0 msgr2=0x7f31ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f31ac0678f0 0x7f31ac069db0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f31b0006fd0 tx=0x7f31b0008040 comp rx=0 tx=0).stop 2024-04-22T01:19:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c0151f00 msgr2=0x7f31c0152390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 --2- 172.21.15.70:0/2568744537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31c0151f00 0x7f31c0152390 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c780 tx=0x7f31b800cc40 comp rx=0 tx=0).stop 2024-04-22T01:19:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31bf59e640 1 -- 172.21.15.70:0/2568744537 reap_dead start 2024-04-22T01:19:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 shutdown_connections 2024-04-22T01:19:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 >> 172.21.15.70:0/2568744537 conn(0x7f31c00b36c0 msgr2=0x7f31c00c7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 shutdown_connections 2024-04-22T01:19:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.066+0000 7f31c51cd640 1 -- 172.21.15.70:0/2568744537 wait complete. 2024-04-22T01:19:19.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-22T01:19:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a5b396640 1 Processor -- start 2024-04-22T01:19:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a5b396640 1 -- start start 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a5b396640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a5415bd10 0x7f6a540cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a5b396640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a540cfa30 con 0x7f6a5409a3d0 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a5b396640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a540ccce0 con 0x7f6a540cdd80 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a5b396640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6a540ce210 con 0x7f6a5415bd10 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59132640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a5415bd10 0x7f6a540cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59933640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6a5409a3d0 0x7f6a540c56f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36528/0 (socket says 172.21.15.70:36528) 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59132640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6a5415bd10 0x7f6a540cc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45246/0 (socket says 172.21.15.70:45246) 2024-04-22T01:19:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59933640 1 -- 172.21.15.70:0/3628943607 learned_addr learned my addr 172.21.15.70:0/3628943607 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59132640 1 -- 172.21.15.70:0/3628943607 >> v1:172.21.15.98:6789/0 conn(0x7f6a540cdd80 legacy=0x7f6a54099cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59132640 1 -- 172.21.15.70:0/3628943607 >> v1:172.21.15.70:6789/0 conn(0x7f6a5409a3d0 legacy=0x7f6a540c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a59132640 1 -- 172.21.15.70:0/3628943607 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a540ccfc0 con 0x7f6a5415bd10 2024-04-22T01:19:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.198+0000 7f6a47fff640 1 -- 172.21.15.70:0/3628943607 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239335623 0 0) 0x7f6a540cfa30 con 0x7f6a5409a3d0 2024-04-22T01:19:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a59132640 1 --2- 172.21.15.70:0/3628943607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5415bd10 0x7f6a540cc6b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009d90 tx=0x7f6a4c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b398e928a985e49e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a47fff640 1 -- 172.21.15.70:0/3628943607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0b2070 con 0x7f6a5415bd10 2024-04-22T01:19:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a47fff640 1 -- 172.21.15.70:0/3628943607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0a58e0 con 0x7f6a5415bd10 2024-04-22T01:19:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a47fff640 1 -- 172.21.15.70:0/3628943607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0a5cb0 con 0x7f6a5415bd10 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 -- 172.21.15.70:0/3628943607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5415bd10 msgr2=0x7f6a540cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 --2- 172.21.15.70:0/3628943607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5415bd10 0x7f6a540cc6b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009d90 tx=0x7f6a4c01c930 comp rx=0 tx=0).stop 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 -- 172.21.15.70:0/3628943607 shutdown_connections 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 --2- 172.21.15.70:0/3628943607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5415bd10 0x7f6a540cc6b0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 -- 172.21.15.70:0/3628943607 >> 172.21.15.70:0/3628943607 conn(0x7f6a540b3690 msgr2=0x7f6a540b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 -- 172.21.15.70:0/3628943607 shutdown_connections 2024-04-22T01:19:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 -- 172.21.15.70:0/3628943607 wait complete. 2024-04-22T01:19:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 Processor -- start 2024-04-22T01:19:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.202+0000 7f6a5b396640 1 -- start start 2024-04-22T01:19:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 0x7f6a54151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a540cdd80 0x7f6a540b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a5415bd10 0x7f6a540b6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5415c270 con 0x7f6a5409a3d0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5415c0f0 con 0x7f6a5415bd10 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6a54169c60 con 0x7f6a540cdd80 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 0x7f6a54151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 0x7f6a54151b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45262/0 (socket says 172.21.15.70:45262) 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 -- 172.21.15.70:0/274931746 learned_addr learned my addr 172.21.15.70:0/274931746 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59933640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a5415bd10 0x7f6a540b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a58931640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a540cdd80 0x7f6a540b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 -- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a540cdd80 msgr2=0x7f6a540b5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a540cdd80 0x7f6a540b5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 -- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a5415bd10 msgr2=0x7f6a540b6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a5415bd10 0x7f6a540b6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 -- 172.21.15.70:0/274931746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a540cbc30 con 0x7f6a5409a3d0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a59132640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 0x7f6a54151b10 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c019540 tx=0x7f6a4c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0bd020 con 0x7f6a5409a3d0 2024-04-22T01:19:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0b8670 con 0x7f6a5409a3d0 2024-04-22T01:19:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a540cb390 con 0x7f6a5409a3d0 2024-04-22T01:19:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0b2040 con 0x7f6a5409a3d0 2024-04-22T01:19:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a540cb950 con 0x7f6a5409a3d0 2024-04-22T01:19:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a5404f510 con 0x7f6a5409a3d0 2024-04-22T01:19:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6a4c0c0020 con 0x7f6a5409a3d0 2024-04-22T01:19:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a45ffb640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a380678f0 0x7f6a38069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f6a4c1209e0 con 0x7f6a5409a3d0 2024-04-22T01:19:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a58931640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a380678f0 0x7f6a38069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.206+0000 7f6a58931640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a380678f0 0x7f6a38069db0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6a541586e0 tx=0x7f6a48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.210+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f6a4c0c7070 con 0x7f6a5409a3d0 2024-04-22T01:19:19.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f6a540c5a50 con 0x7f6a380678f0 2024-04-22T01:19:19.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a45ffb640 1 -- 172.21.15.70:0/274931746 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6a540c5a50 con 0x7f6a380678f0 2024-04-22T01:19:19.362 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 2 to deep-scrub 2024-04-22T01:19:19.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a380678f0 msgr2=0x7f6a38069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a380678f0 0x7f6a38069db0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6a541586e0 tx=0x7f6a48009040 comp rx=0 tx=0).stop 2024-04-22T01:19:19.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 msgr2=0x7f6a54151b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 0x7f6a54151b10 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c019540 tx=0x7f6a4c0196d0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 shutdown_connections 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f6a380678f0 0x7f6a38069db0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a5415bd10 0x7f6a540b6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a540cdd80 0x7f6a540b5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 --2- 172.21.15.70:0/274931746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a5409a3d0 0x7f6a54151b10 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 >> 172.21.15.70:0/274931746 conn(0x7f6a540b3690 msgr2=0x7f6a540cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 shutdown_connections 2024-04-22T01:19:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.362+0000 7f6a5b396640 1 -- 172.21.15.70:0/274931746 wait complete. 2024-04-22T01:19:19.379 INFO:tasks.ceph:Scrubbing osd.3 2024-04-22T01:19:19.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:19.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 Processor -- start 2024-04-22T01:19:19.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 -- start start 2024-04-22T01:19:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ce920 0x7fc268154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2680ceda0 0x7fc2681551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2680b3800 con 0x7fc26815e6e0 2024-04-22T01:19:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc2680b3ae0 con 0x7fc2680ceda0 2024-04-22T01:19:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26c96b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2680cde40 con 0x7fc2680ce920 2024-04-22T01:19:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc266d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2680ceda0 0x7fc2681551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc26815e6e0 0x7fc26815eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36542/0 (socket says 172.21.15.70:36542) 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26659c640 1 -- 172.21.15.70:0/165292063 learned_addr learned my addr 172.21.15.70:0/165292063 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc265d9b640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ce920 0x7fc268154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc266d9d640 1 -- 172.21.15.70:0/165292063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ce920 msgr2=0x7fc268154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc266d9d640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ce920 0x7fc268154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc266d9d640 1 -- 172.21.15.70:0/165292063 >> v1:172.21.15.70:6789/0 conn(0x7fc26815e6e0 legacy=0x7fc26815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc26559a640 1 -- 172.21.15.70:0/165292063 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526961796 0 0) 0x7fc2680b3800 con 0x7fc26815e6e0 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc266d9d640 1 -- 172.21.15.70:0/165292063 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2680ce120 con 0x7fc2680ceda0 2024-04-22T01:19:19.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.502+0000 7fc265d9b640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ce920 0x7fc268154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:19.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc266d9d640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2680ceda0 0x7fc2681551a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fc260009d90 tx=0x7fc26001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aee4ed6fb3b8ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26559a640 1 -- 172.21.15.70:0/165292063 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2600b2070 con 0x7fc2680ceda0 2024-04-22T01:19:19.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26559a640 1 -- 172.21.15.70:0/165292063 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2600a5a20 con 0x7fc2680ceda0 2024-04-22T01:19:19.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26559a640 1 -- 172.21.15.70:0/165292063 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2600a5df0 con 0x7fc2680ceda0 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- 172.21.15.70:0/165292063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2680ceda0 msgr2=0x7fc2681551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2680ceda0 0x7fc2681551a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fc260009d90 tx=0x7fc26001c940 comp rx=0 tx=0).stop 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- 172.21.15.70:0/165292063 shutdown_connections 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2680ceda0 0x7fc2681551a0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 --2- 172.21.15.70:0/165292063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ce920 0x7fc268154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- 172.21.15.70:0/165292063 >> 172.21.15.70:0/165292063 conn(0x7fc2680b1d40 msgr2=0x7fc2680cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- 172.21.15.70:0/165292063 shutdown_connections 2024-04-22T01:19:19.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- 172.21.15.70:0/165292063 wait complete. 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 Processor -- start 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- start start 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2680ce920 0x7fc268151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ceda0 0x7fc26814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26815e6e0 0x7fc2680b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc2680b2e40 con 0x7fc26815e6e0 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2680b2cc0 con 0x7fc2680ce920 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc26c96b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2680b2fc0 con 0x7fc2680ceda0 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.506+0000 7fc266d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26815e6e0 0x7fc2680b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26815e6e0 0x7fc2680b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45288/0 (socket says 172.21.15.70:45288) 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 -- 172.21.15.70:0/3643265157 learned_addr learned my addr 172.21.15.70:0/3643265157 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 -- 172.21.15.70:0/3643265157 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ceda0 msgr2=0x7fc26814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2680ceda0 0x7fc26814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 -- 172.21.15.70:0/3643265157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2680ce920 msgr2=0x7fc268151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2680ce920 0x7fc268151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 -- 172.21.15.70:0/3643265157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc268165100 con 0x7fc26815e6e0 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc266d9d640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26815e6e0 0x7fc2680b0e70 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fc2600b0040 tx=0x7fc2600196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2600b2040 con 0x7fc26815e6e0 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2600ad5c0 con 0x7fc26815e6e0 2024-04-22T01:19:19.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2600ad970 con 0x7fc26815e6e0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2681653e0 con 0x7fc26815e6e0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2680b0670 con 0x7fc26815e6e0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc268068210 con 0x7fc26815e6e0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2600c1020 con 0x7fc26815e6e0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2480678c0 0x7fc248069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7fc260121340 con 0x7fc26815e6e0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc26659c640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2480678c0 0x7fc248069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7fc24806dd60 0x7fc2480701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 --> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc26000aea0 con 0x7fc24806dd60 2024-04-22T01:19:19.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fc2601217d0 con 0x7fc26815e6e0 2024-04-22T01:19:19.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc265d9b640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7fc24806dd60 0x7fc2480701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc26659c640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2480678c0 0x7fc248069d80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc2580015a0 tx=0x7fc258007970 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.510+0000 7fc265d9b640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7fc24806dd60 0x7fc2480701a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.514+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== osd.3 v2:172.21.15.70:6808/1842545244 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc26000aea0 con 0x7fc24806dd60 2024-04-22T01:19:19.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 --> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc268158d10 con 0x7fc24806dd60 2024-04-22T01:19:19.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc256ffd640 1 -- 172.21.15.70:0/3643265157 <== osd.3 v2:172.21.15.70:6808/1842545244 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc268158d10 con 0x7fc24806dd60 2024-04-22T01:19:19.526 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:19.526 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:19.526 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:19.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7fc24806dd60 msgr2=0x7fc2480701a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6808/1842545244,v1:172.21.15.70:6810/1842545244] conn(0x7fc24806dd60 0x7fc2480701a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2480678c0 msgr2=0x7fc248069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc2480678c0 0x7fc248069d80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc2580015a0 tx=0x7fc258007970 comp rx=0 tx=0).stop 2024-04-22T01:19:19.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26815e6e0 msgr2=0x7fc2680b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 --2- 172.21.15.70:0/3643265157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc26815e6e0 0x7fc2680b0e70 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fc2600b0040 tx=0x7fc2600196e0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc266d9d640 1 -- 172.21.15.70:0/3643265157 reap_dead start 2024-04-22T01:19:19.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 shutdown_connections 2024-04-22T01:19:19.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 >> 172.21.15.70:0/3643265157 conn(0x7fc2680b1d40 msgr2=0x7fc26815b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 shutdown_connections 2024-04-22T01:19:19.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.526+0000 7fc26c96b640 1 -- 172.21.15.70:0/3643265157 wait complete. 2024-04-22T01:19:19.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-22T01:19:19.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 Processor -- start 2024-04-22T01:19:19.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 -- start start 2024-04-22T01:19:19.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f501015bb50 0x7f50100cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f501015bf30 0x7f501006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f50101605e0 0x7f501006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f501015f8b0 con 0x7f50101605e0 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f50100cd7a0 con 0x7f501015bb50 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f50159f5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f501015fb00 con 0x7f501015bf30 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f50101605e0 0x7f501006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f50101605e0 0x7f501006f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45298/0 (socket says 172.21.15.70:45298) 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 -- 172.21.15.70:0/2771430253 learned_addr learned my addr 172.21.15.70:0/2771430253 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500effd640 1 --2- 172.21.15.70:0/2771430253 >> v2:172.21.15.98:3300/0 conn(0x7f501015bb50 0x7f50100cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 -- 172.21.15.70:0/2771430253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f501015bf30 msgr2=0x7f501006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 --2- 172.21.15.70:0/2771430253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f501015bf30 0x7f501006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 -- 172.21.15.70:0/2771430253 >> v2:172.21.15.98:3300/0 conn(0x7f501015bb50 msgr2=0x7f50100cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 --2- 172.21.15.70:0/2771430253 >> v2:172.21.15.98:3300/0 conn(0x7f501015bb50 0x7f50100cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.650+0000 7f500f7fe640 1 -- 172.21.15.70:0/2771430253 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501015fde0 con 0x7f50101605e0 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f500f7fe640 1 --2- 172.21.15.70:0/2771430253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50101605e0 0x7f501006f2b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f501006c470 tx=0x7f500801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b426e92a003a28 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f500dffb640 1 -- 172.21.15.70:0/2771430253 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50080ab5f0 con 0x7f50101605e0 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f500dffb640 1 -- 172.21.15.70:0/2771430253 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50080b0070 con 0x7f50101605e0 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f500dffb640 1 -- 172.21.15.70:0/2771430253 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50080ab9b0 con 0x7f50101605e0 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- 172.21.15.70:0/2771430253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50101605e0 msgr2=0x7f501006f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- 172.21.15.70:0/2771430253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50101605e0 0x7f501006f2b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f501006c470 tx=0x7f500801c520 comp rx=0 tx=0).stop 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- 172.21.15.70:0/2771430253 shutdown_connections 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- 172.21.15.70:0/2771430253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50101605e0 0x7f501006f2b0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- 172.21.15.70:0/2771430253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f501015bf30 0x7f501006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- 172.21.15.70:0/2771430253 >> v2:172.21.15.98:3300/0 conn(0x7f501015bb50 0x7f50100cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- 172.21.15.70:0/2771430253 >> 172.21.15.70:0/2771430253 conn(0x7f50100cf9e0 msgr2=0x7f50100cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- 172.21.15.70:0/2771430253 shutdown_connections 2024-04-22T01:19:19.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- 172.21.15.70:0/2771430253 wait complete. 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 Processor -- start 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- start start 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 0x7f50101653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f501015bf30 0x7f5010158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50101605e0 0x7f501014d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f501006e730 con 0x7f501015bb50 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f501006e5b0 con 0x7f501015bf30 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.654+0000 7f50159f5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f501006e8b0 con 0x7f50101605e0 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 0x7f50101653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 0x7f50101653c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45308/0 (socket says 172.21.15.70:45308) 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 -- 172.21.15.70:0/508322625 learned_addr learned my addr 172.21.15.70:0/508322625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500f7fe640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50101605e0 0x7f501014d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500e7fc640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f501015bf30 0x7f5010158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 -- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50101605e0 msgr2=0x7f501014d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50101605e0 0x7f501014d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 -- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f501015bf30 msgr2=0x7f5010158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f501015bf30 0x7f5010158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 -- 172.21.15.70:0/508322625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50101590d0 con 0x7f501015bb50 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500effd640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 0x7f50101653c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f500000ce40 tx=0x7f500000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000003ba0 con 0x7f501015bb50 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5000003d40 con 0x7f501015bb50 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50101f82b0 con 0x7f501015bb50 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000004240 con 0x7f501015bb50 2024-04-22T01:19:19.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50101f8870 con 0x7f501015bb50 2024-04-22T01:19:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f501004f510 con 0x7f501015bb50 2024-04-22T01:19:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50000043e0 con 0x7f501015bb50 2024-04-22T01:19:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f4ff7fff640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4ffc0678c0 0x7f4ffc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500e7fc640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4ffc0678c0 0x7f4ffc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f5000086630 con 0x7f501015bb50 2024-04-22T01:19:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.658+0000 7f500e7fc640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4ffc0678c0 0x7f4ffc069d80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.662+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f50000572e0 con 0x7f501015bb50 2024-04-22T01:19:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.822+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f50100d0740 con 0x7f4ffc0678c0 2024-04-22T01:19:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.822+0000 7f4ff7fff640 1 -- 172.21.15.70:0/508322625 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f50100d0740 con 0x7f4ffc0678c0 2024-04-22T01:19:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 3 to deep-scrub 2024-04-22T01:19:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4ffc0678c0 msgr2=0x7f4ffc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4ffc0678c0 0x7f4ffc069d80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).stop 2024-04-22T01:19:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 msgr2=0x7f50101653c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 0x7f50101653c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f500000ce40 tx=0x7f500000a660 comp rx=0 tx=0).stop 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 shutdown_connections 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f4ffc0678c0 0x7f4ffc069d80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50101605e0 0x7f501014d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f501015bf30 0x7f5010158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 --2- 172.21.15.70:0/508322625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f501015bb50 0x7f50101653c0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 >> 172.21.15.70:0/508322625 conn(0x7f50100cf9e0 msgr2=0x7f50101544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 shutdown_connections 2024-04-22T01:19:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.826+0000 7f50159f5640 1 -- 172.21.15.70:0/508322625 wait complete. 2024-04-22T01:19:19.842 INFO:tasks.ceph:Scrubbing osd.4 2024-04-22T01:19:19.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 Processor -- start 2024-04-22T01:19:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 -- start start 2024-04-22T01:19:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 0x7ff80c15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff80c155370 0x7ff80c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff80c15f000 con 0x7ff80c0ce920 2024-04-22T01:19:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff80c0b3800 con 0x7ff80c155370 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff8108d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c0b3ae0 con 0x7ff80c15e6e0 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff809d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff80c0ce920 0x7ff80c154c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36554/0 (socket says 172.21.15.70:36554) 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff809d9b640 1 -- 172.21.15.70:0/153656603 learned_addr learned my addr 172.21.15.70:0/153656603 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/153656603 >> v2:172.21.15.98:3300/0 conn(0x7ff80c155370 0x7ff80c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80a59c640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 0x7ff80c15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707253425 0 0) 0x7ff80c15f000 con 0x7ff80c0ce920 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7f4003530 con 0x7ff80c0ce920 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 149238760 0 0) 0x7ff7f4003530 con 0x7ff80c0ce920 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 msgr2=0x7ff80c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 0x7ff80c15eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 >> v2:172.21.15.98:3300/0 conn(0x7ff80c155370 msgr2=0x7ff80c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 --2- 172.21.15.70:0/153656603 >> v2:172.21.15.98:3300/0 conn(0x7ff80c155370 0x7ff80c06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c0cde40 con 0x7ff80c0ce920 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7ff800001680 con 0x7ff80c0ce920 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 >> v1:172.21.15.70:6789/0 conn(0x7ff80c0ce920 legacy=0x7ff80c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f40039c0 0x7ff7f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 0x7ff7f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4004d50 0x7ff7f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c0b3800 con 0x7ff7f4004d50 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c0b3ae0 con 0x7ff7f40039c0 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f400ba50 con 0x7ff80c15e6e0 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4004d50 0x7ff7f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 -- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 msgr2=0x7ff7f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c15e6e0 0x7ff7f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 -- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f40039c0 msgr2=0x7ff7f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f40039c0 0x7ff7f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.962+0000 7ff80ad9d640 1 -- 172.21.15.70:0/153656603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f4003530 con 0x7ff7f4004d50 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4004d50 0x7ff7f40051d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7ff804009fc0 tx=0x7ff80401eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b7533ee099b35a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8040b4070 con 0x7ff7f4004d50 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8040af440 con 0x7ff7f4004d50 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff80959a640 1 -- 172.21.15.70:0/153656603 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804009c10 con 0x7ff7f4004d50 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 -- 172.21.15.70:0/153656603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4004d50 msgr2=0x7ff7f40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 --2- 172.21.15.70:0/153656603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4004d50 0x7ff7f40051d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7ff804009fc0 tx=0x7ff80401eed0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff80ad9d640 1 -- 172.21.15.70:0/153656603 reap_dead start 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 -- 172.21.15.70:0/153656603 shutdown_connections 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 -- 172.21.15.70:0/153656603 >> 172.21.15.70:0/153656603 conn(0x7ff80c0b1d40 msgr2=0x7ff80c0cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 -- 172.21.15.70:0/153656603 shutdown_connections 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 -- 172.21.15.70:0/153656603 wait complete. 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 Processor -- start 2024-04-22T01:19:19.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 -- start start 2024-04-22T01:19:19.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.966+0000 7ff8108d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c1453b0 0x7ff80c158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c145790 0x7ff80c159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c15ce30 0x7ff80c15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c06f0e0 con 0x7ff80c145790 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c0ce400 con 0x7ff80c15ce30 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c06f260 con 0x7ff80c1453b0 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c145790 0x7ff80c159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c145790 0x7ff80c159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45340/0 (socket says 172.21.15.70:45340) 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 -- 172.21.15.70:0/942175806 learned_addr learned my addr 172.21.15.70:0/942175806 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff80a59c640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c1453b0 0x7ff80c158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c15ce30 0x7ff80c15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 -- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c1453b0 msgr2=0x7ff80c158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff80c1453b0 0x7ff80c158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 -- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c15ce30 msgr2=0x7ff80c15d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c15ce30 0x7ff80c15d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 -- 172.21.15.70:0/942175806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c1e3f20 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff809d9b640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c145790 0x7ff80c159250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ff80000caa0 tx=0x7ff80000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800018070 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff800004cc0 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c15dbb0 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800007310 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c1e41e0 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff80c068210 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff80001b070 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7ec0678c0 0x7ff7ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7ff800085650 con 0x7ff80c145790 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff80a59c640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7ec0678c0 0x7ff7ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7ff7ec06dd60 0x7ff7ec0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 --> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7ec070870 con 0x7ff7ec06dd60 2024-04-22T01:19:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7ff800085ae0 con 0x7ff80c145790 2024-04-22T01:19:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff80a59c640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7ec0678c0 0x7ff7ec069d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0047c0 tx=0x7ff7fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7ff7ec06dd60 0x7ff7ec0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.970+0000 7ff80ad9d640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7ff7ec06dd60 0x7ff7ec0701a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.974+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== osd.4 v2:172.21.15.98:6804/3056066255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7ec070870 con 0x7ff7ec06dd60 2024-04-22T01:19:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 --> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff80c0d08e0 con 0x7ff7ec06dd60 2024-04-22T01:19:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff7faffd640 1 -- 172.21.15.70:0/942175806 <== osd.4 v2:172.21.15.98:6804/3056066255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff80c0d08e0 con 0x7ff7ec06dd60 2024-04-22T01:19:19.988 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:19.988 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7ff7ec06dd60 msgr2=0x7ff7ec0701a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.98:6804/3056066255,v1:172.21.15.98:6805/3056066255] conn(0x7ff7ec06dd60 0x7ff7ec0701a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7ec0678c0 msgr2=0x7ff7ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7ec0678c0 0x7ff7ec069d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0047c0 tx=0x7ff7fc008040 comp rx=0 tx=0).stop 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c145790 msgr2=0x7ff80c159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 --2- 172.21.15.70:0/942175806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c145790 0x7ff80c159250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ff80000caa0 tx=0x7ff80000cf60 comp rx=0 tx=0).stop 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff80ad9d640 1 -- 172.21.15.70:0/942175806 reap_dead start 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 shutdown_connections 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 >> 172.21.15.70:0/942175806 conn(0x7ff80c0b1d40 msgr2=0x7ff80c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 shutdown_connections 2024-04-22T01:19:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:19.986+0000 7ff8108d2640 1 -- 172.21.15.70:0/942175806 wait complete. 2024-04-22T01:19:20.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-22T01:19:20.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 Processor -- start 2024-04-22T01:19:20.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 -- start start 2024-04-22T01:19:20.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400d17b0 0x7fd2400d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2400ce3f0 0x7fd2400ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2400cee40 con 0x7fd2400cc610 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd24015c150 con 0x7fd2400ce3f0 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd247926640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd2400cf400 con 0x7fd2400d17b0 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd244ec1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2400ce3f0 0x7fd2400ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd245ec3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd2400cc610 0x7fd2400d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.70:48826/0 (socket says 172.21.15.70:48826) 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd2456c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400d17b0 0x7fd2400d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd245ec3640 1 -- 172.21.15.70:0/4187328149 learned_addr learned my addr 172.21.15.70:0/4187328149 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:20.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd244ec1640 1 -- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400d17b0 msgr2=0x7fd2400d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:20.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd244ec1640 1 --2- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400d17b0 0x7fd2400d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd244ec1640 1 -- 172.21.15.70:0/4187328149 >> v1:172.21.15.98:6789/0 conn(0x7fd2400cc610 legacy=0x7fd2400d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.110+0000 7fd244ec1640 1 -- 172.21.15.70:0/4187328149 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2400cf120 con 0x7fd2400ce3f0 2024-04-22T01:19:20.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd244ec1640 1 --2- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400ce3f0 0x7fd2400ce850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd2300097c0 tx=0x7fd23001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeb4477440b0ca84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd237fff640 1 -- 172.21.15.70:0/4187328149 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2300b0070 con 0x7fd2400ce3f0 2024-04-22T01:19:20.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd237fff640 1 -- 172.21.15.70:0/4187328149 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2300ab600 con 0x7fd2400ce3f0 2024-04-22T01:19:20.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd237fff640 1 -- 172.21.15.70:0/4187328149 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2300ab9d0 con 0x7fd2400ce3f0 2024-04-22T01:19:20.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 -- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400ce3f0 msgr2=0x7fd2400ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 --2- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400ce3f0 0x7fd2400ce850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd2300097c0 tx=0x7fd23001c4a0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 -- 172.21.15.70:0/4187328149 shutdown_connections 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 --2- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400ce3f0 0x7fd2400ce850 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 --2- 172.21.15.70:0/4187328149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400d17b0 0x7fd2400d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 -- 172.21.15.70:0/4187328149 >> 172.21.15.70:0/4187328149 conn(0x7fd24008b1b0 msgr2=0x7fd24008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 -- 172.21.15.70:0/4187328149 shutdown_connections 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.114+0000 7fd247926640 1 -- 172.21.15.70:0/4187328149 wait complete. 2024-04-22T01:19:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 Processor -- start 2024-04-22T01:19:20.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- start start 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400cc610 0x7fd24015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400d17b0 0x7fd24015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 0x7fd24015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd24015fb00 con 0x7fd24015c810 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd24015f980 con 0x7fd2400d17b0 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd2401693a0 con 0x7fd2400cc610 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 0x7fd24015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 0x7fd24015a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45342/0 (socket says 172.21.15.70:45342) 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 -- 172.21.15.70:0/1036462339 learned_addr learned my addr 172.21.15.70:0/1036462339 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd2456c2640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400cc610 0x7fd24015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd244ec1640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400d17b0 0x7fd24015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 -- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400cc610 msgr2=0x7fd24015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400cc610 0x7fd24015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 -- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400d17b0 msgr2=0x7fd24015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400d17b0 0x7fd24015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 -- 172.21.15.70:0/1036462339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24015ccb0 con 0x7fd24015c810 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd2456c2640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400cc610 0x7fd24015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd245ec3640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 0x7fd24015a820 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fd23800cbc0 tx=0x7fd23800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238017070 con 0x7fd24015c810 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23800ad40 con 0x7fd24015c810 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24015cf90 con 0x7fd24015c810 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238012600 con 0x7fd24015c810 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd24015d550 con 0x7fd24015c810 2024-04-22T01:19:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24004f510 con 0x7fd24015c810 2024-04-22T01:19:20.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd23801a070 con 0x7fd24015c810 2024-04-22T01:19:20.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd235ffb640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2240678c0 0x7fd224069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7fd238085790 con 0x7fd24015c810 2024-04-22T01:19:20.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.118+0000 7fd2456c2640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2240678c0 0x7fd224069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.122+0000 7fd2456c2640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2240678c0 0x7fd224069d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd2400ca4c0 tx=0x7fd23c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.122+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fd238055de0 con 0x7fd24015c810 2024-04-22T01:19:20.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.274+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fd240070090 con 0x7fd2240678c0 2024-04-22T01:19:20.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd235ffb640 1 -- 172.21.15.70:0/1036462339 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd240070090 con 0x7fd2240678c0 2024-04-22T01:19:20.278 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 4 to deep-scrub 2024-04-22T01:19:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2240678c0 msgr2=0x7fd224069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2240678c0 0x7fd224069d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd2400ca4c0 tx=0x7fd23c008040 comp rx=0 tx=0).stop 2024-04-22T01:19:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 msgr2=0x7fd24015a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 0x7fd24015a820 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fd23800cbc0 tx=0x7fd23800a430 comp rx=0 tx=0).stop 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 shutdown_connections 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fd2240678c0 0x7fd224069d80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24015c810 0x7fd24015a820 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2400d17b0 0x7fd24015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 --2- 172.21.15.70:0/1036462339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2400cc610 0x7fd24015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 >> 172.21.15.70:0/1036462339 conn(0x7fd24008b1b0 msgr2=0x7fd2400af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 shutdown_connections 2024-04-22T01:19:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.278+0000 7fd247926640 1 -- 172.21.15.70:0/1036462339 wait complete. 2024-04-22T01:19:20.294 INFO:tasks.ceph:Scrubbing osd.5 2024-04-22T01:19:20.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:20.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 Processor -- start 2024-04-22T01:19:20.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 -- start start 2024-04-22T01:19:20.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f624015f410 0x7f624006bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f624006c140 0x7f62400ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62400cd5a0 con 0x7f62400af990 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f624006b400 con 0x7f624015f410 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f624515a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6240067460 con 0x7f624006c140 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f624015f410 0x7f624006bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f624015f410 0x7f624006bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45352/0 (socket says 172.21.15.70:45352) 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 -- 172.21.15.70:0/3645076814 learned_addr learned my addr 172.21.15.70:0/3645076814 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623e59c640 1 --2- 172.21.15.70:0/3645076814 >> v2:172.21.15.98:3300/0 conn(0x7f624006c140 0x7f62400ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 -- 172.21.15.70:0/3645076814 >> v2:172.21.15.98:3300/0 conn(0x7f624006c140 msgr2=0x7f62400ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 --2- 172.21.15.70:0/3645076814 >> v2:172.21.15.98:3300/0 conn(0x7f624006c140 0x7f62400ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 -- 172.21.15.70:0/3645076814 >> v1:172.21.15.98:6789/0 conn(0x7f62400af990 legacy=0x7f62400cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:20.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623ed9d640 1 -- 172.21.15.70:0/3645076814 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62400cd880 con 0x7f624015f410 2024-04-22T01:19:20.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.418+0000 7f623e59c640 1 --2- 172.21.15.70:0/3645076814 >> v2:172.21.15.98:3300/0 conn(0x7f624006c140 0x7f62400ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:20.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f623ed9d640 1 --2- 172.21.15.70:0/3645076814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f624006bc00 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f62400736f0 tx=0x7f623801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d53c126f60e82a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f623dd9b640 1 -- 172.21.15.70:0/3645076814 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62380a7070 con 0x7f624015f410 2024-04-22T01:19:20.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f623dd9b640 1 -- 172.21.15.70:0/3645076814 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62380b0070 con 0x7f624015f410 2024-04-22T01:19:20.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f623dd9b640 1 -- 172.21.15.70:0/3645076814 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62380ab890 con 0x7f624015f410 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 -- 172.21.15.70:0/3645076814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 msgr2=0x7f624006bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 --2- 172.21.15.70:0/3645076814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f624006bc00 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f62400736f0 tx=0x7f623801c450 comp rx=0 tx=0).stop 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 -- 172.21.15.70:0/3645076814 shutdown_connections 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 --2- 172.21.15.70:0/3645076814 >> v2:172.21.15.98:3300/0 conn(0x7f624006c140 0x7f62400ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 --2- 172.21.15.70:0/3645076814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f624006bc00 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 -- 172.21.15.70:0/3645076814 >> 172.21.15.70:0/3645076814 conn(0x7f6240099a60 msgr2=0x7f62400b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 -- 172.21.15.70:0/3645076814 shutdown_connections 2024-04-22T01:19:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 -- 172.21.15.70:0/3645076814 wait complete. 2024-04-22T01:19:20.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 Processor -- start 2024-04-22T01:19:20.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.422+0000 7f624515a640 1 -- start start 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62400af990 0x7f624014fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f6240154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62400cea30 0x7f62400c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62401604b0 con 0x7f624015f410 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6240160330 con 0x7f62400cea30 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6240160630 con 0x7f62400af990 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f6240154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f6240154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45364/0 (socket says 172.21.15.70:45364) 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 -- 172.21.15.70:0/335988129 learned_addr learned my addr 172.21.15.70:0/335988129 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623f59e640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62400cea30 0x7f62400c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623ed9d640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62400af990 0x7f624014fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 -- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62400af990 msgr2=0x7f624014fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62400af990 0x7f624014fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 -- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62400cea30 msgr2=0x7f62400c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62400cea30 0x7f62400c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 -- 172.21.15.70:0/335988129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6240150300 con 0x7f624015f410 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623f59e640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62400cea30 0x7f62400c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623e59c640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f6240154180 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f622800c820 tx=0x7f622800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228015070 con 0x7f624015f410 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6228007a30 con 0x7f624015f410 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62400ceec0 con 0x7f624015f410 2024-04-22T01:19:20.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228007e00 con 0x7f624015f410 2024-04-22T01:19:20.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62400cf480 con 0x7f624015f410 2024-04-22T01:19:20.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6240068920 con 0x7f624015f410 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6228018070 con 0x7f624015f410 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62200678f0 0x7f6220069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f6228085ae0 con 0x7f624015f410 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623ed9d640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62200678f0 0x7f6220069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f622006dd90 0x7f62200701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 --> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f622801be50 con 0x7f622006dd90 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f622808a2c0 con 0x7f624015f410 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.426+0000 7f623ed9d640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62200678f0 0x7f6220069db0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f62380af000 tx=0x7f6238008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.430+0000 7f623f59e640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f622006dd90 0x7f62200701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.430+0000 7f623f59e640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f622006dd90 0x7f62200701d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.430+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== osd.5 v2:172.21.15.98:6816/3651416447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f622801be50 con 0x7f622006dd90 2024-04-22T01:19:20.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 --> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6240067780 con 0x7f622006dd90 2024-04-22T01:19:20.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f62377fe640 1 -- 172.21.15.70:0/335988129 <== osd.5 v2:172.21.15.98:6816/3651416447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6240067780 con 0x7f622006dd90 2024-04-22T01:19:20.444 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:20.444 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:20.444 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f622006dd90 msgr2=0x7f62200701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.98:6816/3651416447,v1:172.21.15.98:6817/3651416447] conn(0x7f622006dd90 0x7f62200701d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62200678f0 msgr2=0x7f6220069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f62200678f0 0x7f6220069db0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f62380af000 tx=0x7f6238008040 comp rx=0 tx=0).stop 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 msgr2=0x7f6240154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 --2- 172.21.15.70:0/335988129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624015f410 0x7f6240154180 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f622800c820 tx=0x7f622800cce0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f623f59e640 1 -- 172.21.15.70:0/335988129 reap_dead start 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 shutdown_connections 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 >> 172.21.15.70:0/335988129 conn(0x7f6240099a60 msgr2=0x7f62400cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.442+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 shutdown_connections 2024-04-22T01:19:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.446+0000 7f624515a640 1 -- 172.21.15.70:0/335988129 wait complete. 2024-04-22T01:19:20.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-22T01:19:20.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 Processor -- start 2024-04-22T01:19:20.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 -- start start 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f028c099c50 0x7f028c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f028c0cc1c0 0x7f028c0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f028c0d0750 con 0x7f028c09a030 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f028c1619b0 con 0x7f028c099c50 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f02910d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f028c0d17f0 con 0x7f028c0cc1c0 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f028c099c50 0x7f028c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f028c099c50 0x7f028c06d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45380/0 (socket says 172.21.15.70:45380) 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 -- 172.21.15.70:0/377314477 learned_addr learned my addr 172.21.15.70:0/377314477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 -- 172.21.15.70:0/377314477 >> v2:172.21.15.98:3300/0 conn(0x7f028c0cc1c0 msgr2=0x7f028c0cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:20.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 --2- 172.21.15.70:0/377314477 >> v2:172.21.15.98:3300/0 conn(0x7f028c0cc1c0 0x7f028c0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 -- 172.21.15.70:0/377314477 >> v1:172.21.15.98:6789/0 conn(0x7f028c09a030 legacy=0x7f028c0cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:20.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.566+0000 7f028ad9d640 1 -- 172.21.15.70:0/377314477 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c0d0a30 con 0x7f028c099c50 2024-04-22T01:19:20.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028ad9d640 1 --2- 172.21.15.70:0/377314477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c099c50 0x7f028c06d3f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f028c067df0 tx=0x7f028401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cc4dbc0cd96c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f0289d9b640 1 -- 172.21.15.70:0/377314477 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840b5070 con 0x7f028c099c50 2024-04-22T01:19:20.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f0289d9b640 1 -- 172.21.15.70:0/377314477 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02840a89d0 con 0x7f028c099c50 2024-04-22T01:19:20.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f0289d9b640 1 -- 172.21.15.70:0/377314477 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840a8da0 con 0x7f028c099c50 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- 172.21.15.70:0/377314477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c099c50 msgr2=0x7f028c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 --2- 172.21.15.70:0/377314477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c099c50 0x7f028c06d3f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f028c067df0 tx=0x7f028401ecd0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- 172.21.15.70:0/377314477 shutdown_connections 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 --2- 172.21.15.70:0/377314477 >> v2:172.21.15.98:3300/0 conn(0x7f028c0cc1c0 0x7f028c0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 --2- 172.21.15.70:0/377314477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c099c50 0x7f028c06d3f0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- 172.21.15.70:0/377314477 >> 172.21.15.70:0/377314477 conn(0x7f028c0b3660 msgr2=0x7f028c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- 172.21.15.70:0/377314477 shutdown_connections 2024-04-22T01:19:20.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- 172.21.15.70:0/377314477 wait complete. 2024-04-22T01:19:20.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 Processor -- start 2024-04-22T01:19:20.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- start start 2024-04-22T01:19:20.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f028c099c50 0x7f028c15e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c09a030 0x7f028c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 0x7f028c15c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f028c06d930 con 0x7f028c0cc1c0 2024-04-22T01:19:20.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f028c0d1360 con 0x7f028c09a030 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f02910d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f028c0d0ff0 con 0x7f028c099c50 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 0x7f028c15c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 0x7f028c15c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45384/0 (socket says 172.21.15.70:45384) 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 -- 172.21.15.70:0/3116481551 learned_addr learned my addr 172.21.15.70:0/3116481551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028a59c640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c09a030 0x7f028c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028ad9d640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f028c099c50 0x7f028c15e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 -- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f028c099c50 msgr2=0x7f028c15e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f028c099c50 0x7f028c15e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 -- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c09a030 msgr2=0x7f028c157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c09a030 0x7f028c157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028b59e640 1 -- 172.21.15.70:0/3116481551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c0c6590 con 0x7f028c0cc1c0 2024-04-22T01:19:20.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.570+0000 7f028ad9d640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f028c099c50 0x7f028c15e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f028b59e640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 0x7f028c15c9e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b9a0 tx=0x7f027c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c019070 con 0x7f028c0cc1c0 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027c00cbc0 con 0x7f028c0cc1c0 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028c15eb90 con 0x7f028c0cc1c0 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c014440 con 0x7f028c0cc1c0 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028c15f150 con 0x7f028c0cc1c0 2024-04-22T01:19:20.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f028c04f510 con 0x7f028c0cc1c0 2024-04-22T01:19:20.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f027c01c070 con 0x7f028c0cc1c0 2024-04-22T01:19:20.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02837fe640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f026c0678c0 0x7f026c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f027c085810 con 0x7f028c0cc1c0 2024-04-22T01:19:20.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f028ad9d640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f026c0678c0 0x7f026c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.574+0000 7f028ad9d640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f026c0678c0 0x7f026c069d80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f02840b2000 tx=0x7f0284008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.578+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f027c056550 con 0x7f028c0cc1c0 2024-04-22T01:19:20.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f028c070ac0 con 0x7f026c0678c0 2024-04-22T01:19:20.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02837fe640 1 -- 172.21.15.70:0/3116481551 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f028c070ac0 con 0x7f026c0678c0 2024-04-22T01:19:20.726 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 5 to deep-scrub 2024-04-22T01:19:20.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f026c0678c0 msgr2=0x7f026c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f026c0678c0 0x7f026c069d80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f02840b2000 tx=0x7f0284008040 comp rx=0 tx=0).stop 2024-04-22T01:19:20.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 msgr2=0x7f028c15c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 0x7f028c15c9e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b9a0 tx=0x7f027c00be60 comp rx=0 tx=0).stop 2024-04-22T01:19:20.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 shutdown_connections 2024-04-22T01:19:20.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f026c0678c0 0x7f026c069d80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c0cc1c0 0x7f028c15c9e0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c09a030 0x7f028c157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 --2- 172.21.15.70:0/3116481551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f028c099c50 0x7f028c15e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 >> 172.21.15.70:0/3116481551 conn(0x7f028c0b3660 msgr2=0x7f028c0cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 shutdown_connections 2024-04-22T01:19:20.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.726+0000 7f02910d4640 1 -- 172.21.15.70:0/3116481551 wait complete. 2024-04-22T01:19:20.743 INFO:tasks.ceph:Scrubbing osd.6 2024-04-22T01:19:20.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:20.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 Processor -- start 2024-04-22T01:19:20.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 -- start start 2024-04-22T01:19:20.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff7ec0cc5d0 0x7ff7ec0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec06b1f0 0x7ff7ec06b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec0b2440 con 0x7ff7ec0cc9b0 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ec06bba0 con 0x7ff7ec0cc5d0 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f39e4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ec0b18b0 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff7ec0cc5d0 0x7ff7ec0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff7ec0cc5d0 0x7ff7ec0ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45400/0 (socket says 172.21.15.70:45400) 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 -- 172.21.15.70:0/2077326689 learned_addr learned my addr 172.21.15.70:0/2077326689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1f81640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec06b1f0 0x7ff7ec06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 -- 172.21.15.70:0/2077326689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec06b1f0 msgr2=0x7ff7ec06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec06b1f0 0x7ff7ec06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 -- 172.21.15.70:0/2077326689 >> v1:172.21.15.98:6789/0 conn(0x7ff7ec0cc9b0 legacy=0x7ff7ec0b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1780640 1 -- 172.21.15.70:0/2077326689 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec0b0f80 con 0x7ff7ec0cc5d0 2024-04-22T01:19:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.866+0000 7ff7f1f81640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec06b1f0 0x7ff7ec06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:20.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f1780640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0ce050 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80098a0 tx=0x7ff7e801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bebc9213a15ad12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7e3fff640 1 -- 172.21.15.70:0/2077326689 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80ab680 con 0x7ff7ec0cc5d0 2024-04-22T01:19:20.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7e3fff640 1 -- 172.21.15.70:0/2077326689 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e80b0070 con 0x7ff7ec0cc5d0 2024-04-22T01:19:20.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7e3fff640 1 -- 172.21.15.70:0/2077326689 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80aba40 con 0x7ff7ec0cc5d0 2024-04-22T01:19:20.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- 172.21.15.70:0/2077326689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec0cc5d0 msgr2=0x7ff7ec0ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0ce050 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80098a0 tx=0x7ff7e801c480 comp rx=0 tx=0).stop 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- 172.21.15.70:0/2077326689 shutdown_connections 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec06b1f0 0x7ff7ec06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 --2- 172.21.15.70:0/2077326689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0ce050 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- 172.21.15.70:0/2077326689 >> 172.21.15.70:0/2077326689 conn(0x7ff7ec099a60 msgr2=0x7ff7ec099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- 172.21.15.70:0/2077326689 shutdown_connections 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- 172.21.15.70:0/2077326689 wait complete. 2024-04-22T01:19:20.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 Processor -- start 2024-04-22T01:19:20.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- start start 2024-04-22T01:19:20.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec06b1f0 0x7ff7ec157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec0cc9b0 0x7ff7ec0b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec0d0d50 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec0d0bd0 con 0x7ff7ec0cc5d0 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.870+0000 7ff7f39e4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ec0d0ed0 con 0x7ff7ec0cc9b0 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec06b1f0 0x7ff7ec157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec06b1f0 0x7ff7ec157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45402/0 (socket says 172.21.15.70:45402) 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 -- 172.21.15.70:0/1219152412 learned_addr learned my addr 172.21.15.70:0/1219152412 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f0f7f640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1f81640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec0cc9b0 0x7ff7ec0b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 -- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec0cc9b0 msgr2=0x7ff7ec0b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7ec0cc9b0 0x7ff7ec0b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 -- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec0cc5d0 msgr2=0x7ff7ec0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 -- 172.21.15.70:0/1219152412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec0c66a0 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f0f7f640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7ec0cc5d0 0x7ff7ec0b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1780640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec06b1f0 0x7ff7ec157c00 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7ff7e801a0b0 tx=0x7ff7e801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80b4800 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e80a7070 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80b4c20 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec0c6980 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ec158140 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7ec068210 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff7e80b3380 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7d00678c0 0x7ff7d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7ff7e81202d0 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f0f7f640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7d00678c0 0x7ff7d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7ff7d006dd60 0x7ff7d00701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:20.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 --> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7d0070870 con 0x7ff7d006dd60 2024-04-22T01:19:20.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7ff7e8120760 con 0x7ff7ec06b1f0 2024-04-22T01:19:20.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f0f7f640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7d00678c0 0x7ff7d0069d80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec1f9c70 tx=0x7ff7dc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.874+0000 7ff7f1f81640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7ff7d006dd60 0x7ff7d00701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:20.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.878+0000 7ff7f1f81640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7ff7d006dd60 0x7ff7d00701a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:20.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.878+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== osd.6 v2:172.21.15.98:6800/3535628973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7d0070870 con 0x7ff7d006dd60 2024-04-22T01:19:20.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.890+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 --> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff7ec0afb60 con 0x7ff7d006dd60 2024-04-22T01:19:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.890+0000 7ff7e1ffb640 1 -- 172.21.15.70:0/1219152412 <== osd.6 v2:172.21.15.98:6800/3535628973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff7ec0afb60 con 0x7ff7d006dd60 2024-04-22T01:19:20.892 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:20.892 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:20.892 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7ff7d006dd60 msgr2=0x7ff7d00701a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.98:6800/3535628973,v1:172.21.15.98:6801/3535628973] conn(0x7ff7d006dd60 0x7ff7d00701a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7d00678c0 msgr2=0x7ff7d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7ff7d00678c0 0x7ff7d0069d80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec1f9c70 tx=0x7ff7dc009040 comp rx=0 tx=0).stop 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec06b1f0 msgr2=0x7ff7ec157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 --2- 172.21.15.70:0/1219152412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7ec06b1f0 0x7ff7ec157c00 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7ff7e801a0b0 tx=0x7ff7e801acc0 comp rx=0 tx=0).stop 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f1f81640 1 -- 172.21.15.70:0/1219152412 reap_dead start 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 shutdown_connections 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 >> 172.21.15.70:0/1219152412 conn(0x7ff7ec099a60 msgr2=0x7ff7ec15b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 shutdown_connections 2024-04-22T01:19:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:20.894+0000 7ff7f39e4640 1 -- 172.21.15.70:0/1219152412 wait complete. 2024-04-22T01:19:20.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-22T01:19:21.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 Processor -- start 2024-04-22T01:19:21.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 -- start start 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb668161a50 0x7fb6680ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6680ceef0 0x7fb6680d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb668161ee0 con 0x7fb6681614d0 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb6680d0170 con 0x7fb6680ceef0 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66cf25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb6680cf360 con 0x7fb668161a50 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb6681614d0 0x7fb6680b9670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36560/0 (socket says 172.21.15.70:36560) 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 -- 172.21.15.70:0/1535141214 learned_addr learned my addr 172.21.15.70:0/1535141214 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2063280850 0 0) 0x7fb668161ee0 con 0x7fb6681614d0 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb666d9d640 1 --2- 172.21.15.70:0/1535141214 >> v2:172.21.15.98:3300/0 conn(0x7fb6680ceef0 0x7fb6680d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb654003530 con 0x7fb6681614d0 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb665d9b640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb668161a50 0x7fb6680ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1244300188 0 0) 0x7fb654003530 con 0x7fb6681614d0 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb668161a50 msgr2=0x7fb6680ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb668161a50 0x7fb6680ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 >> v2:172.21.15.98:3300/0 conn(0x7fb6680ceef0 msgr2=0x7fb6680d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 --2- 172.21.15.70:0/1535141214 >> v2:172.21.15.98:3300/0 conn(0x7fb6680ceef0 0x7fb6680d2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6681621c0 con 0x7fb6681614d0 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7fb660001a00 con 0x7fb6681614d0 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 >> v1:172.21.15.70:6789/0 conn(0x7fb6681614d0 legacy=0x7fb6680b9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb654003a30 0x7fb654003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654004810 0x7fb654004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654005190 0x7fb65400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6680d0170 con 0x7fb654003a30 2024-04-22T01:19:21.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6680cf360 con 0x7fb654005190 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb65400bf10 con 0x7fb654004810 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb654003a30 0x7fb654003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb666d9d640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654005190 0x7fb65400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb665d9b640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654004810 0x7fb654004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 -- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654004810 msgr2=0x7fb654004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654004810 0x7fb654004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 -- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654005190 msgr2=0x7fb65400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654005190 0x7fb65400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb66659c640 1 -- 172.21.15.70:0/1535141214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb654003530 con 0x7fb654003a30 2024-04-22T01:19:21.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.018+0000 7fb666d9d640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654005190 0x7fb65400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:21.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66659c640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb654003a30 0x7fb654003e50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fb66000a870 tx=0x7fb66001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab57c7d7f2a60962 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66000a4c0 con 0x7fb654003a30 2024-04-22T01:19:21.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6600ad630 con 0x7fb654003a30 2024-04-22T01:19:21.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66559a640 1 -- 172.21.15.70:0/1535141214 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6600ada20 con 0x7fb654003a30 2024-04-22T01:19:21.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66cf25640 1 -- 172.21.15.70:0/1535141214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb654003a30 msgr2=0x7fb654003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1535141214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb654003a30 0x7fb654003e50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fb66000a870 tx=0x7fb66001d3a0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb666d9d640 1 -- 172.21.15.70:0/1535141214 reap_dead start 2024-04-22T01:19:21.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66cf25640 1 -- 172.21.15.70:0/1535141214 shutdown_connections 2024-04-22T01:19:21.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66cf25640 1 -- 172.21.15.70:0/1535141214 >> 172.21.15.70:0/1535141214 conn(0x7fb66806c9d0 msgr2=0x7fb66806cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66cf25640 1 -- 172.21.15.70:0/1535141214 shutdown_connections 2024-04-22T01:19:21.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.022+0000 7fb66cf25640 1 -- 172.21.15.70:0/1535141214 wait complete. 2024-04-22T01:19:21.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 Processor -- start 2024-04-22T01:19:21.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- start start 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654003a30 0x7fb66814d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 0x7fb6680cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb66815da00 0x7fb66815de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6680d3650 con 0x7fb66814ccb0 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6680d34d0 con 0x7fb66815da00 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb6680d37d0 con 0x7fb654003a30 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 0x7fb6680cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 0x7fb6680cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45434/0 (socket says 172.21.15.70:45434) 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 -- 172.21.15.70:0/1631881561 learned_addr learned my addr 172.21.15.70:0/1631881561 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb666d9d640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb66815da00 0x7fb66815de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66659c640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654003a30 0x7fb66814d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 -- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654003a30 msgr2=0x7fb66814d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654003a30 0x7fb66814d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 -- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb66815da00 msgr2=0x7fb66815de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb66815da00 0x7fb66815de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 -- 172.21.15.70:0/1631881561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6680ccfd0 con 0x7fb66814ccb0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66659c640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654003a30 0x7fb66814d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb666d9d640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb66815da00 0x7fb66815de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb665d9b640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 0x7fb6680cafe0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb65000e870 tx=0x7fb65000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650017070 con 0x7fb66814ccb0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65000c8d0 con 0x7fb66814ccb0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65000f040 con 0x7fb66814ccb0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66814d130 con 0x7fb66814ccb0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6680cbb10 con 0x7fb66814ccb0 2024-04-22T01:19:21.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66804f510 con 0x7fb66814ccb0 2024-04-22T01:19:21.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.026+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb65001a070 con 0x7fb66814ccb0 2024-04-22T01:19:21.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.030+0000 7fb65effd640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb6480678c0 0x7fb648069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.030+0000 7fb66659c640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb6480678c0 0x7fb648069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.030+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7fb650085690 con 0x7fb66814ccb0 2024-04-22T01:19:21.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.030+0000 7fb66659c640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb6480678c0 0x7fb648069d80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb6600ad000 tx=0x7fb660001370 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.034+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fb650056340 con 0x7fb66814ccb0 2024-04-22T01:19:21.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.178+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fb6680d19b0 con 0x7fb6480678c0 2024-04-22T01:19:21.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.178+0000 7fb65effd640 1 -- 172.21.15.70:0/1631881561 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb6680d19b0 con 0x7fb6480678c0 2024-04-22T01:19:21.179 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 6 to deep-scrub 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb6480678c0 msgr2=0x7fb648069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb6480678c0 0x7fb648069d80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb6600ad000 tx=0x7fb660001370 comp rx=0 tx=0).stop 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 msgr2=0x7fb6680cafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 0x7fb6680cafe0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb65000e870 tx=0x7fb65000ed30 comp rx=0 tx=0).stop 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 shutdown_connections 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fb6480678c0 0x7fb648069d80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb66815da00 0x7fb66815de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb66814ccb0 0x7fb6680cafe0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 --2- 172.21.15.70:0/1631881561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb654003a30 0x7fb66814d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 >> 172.21.15.70:0/1631881561 conn(0x7fb66806c9d0 msgr2=0x7fb66814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 shutdown_connections 2024-04-22T01:19:21.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.182+0000 7fb66cf25640 1 -- 172.21.15.70:0/1631881561 wait complete. 2024-04-22T01:19:21.196 INFO:tasks.ceph:Scrubbing osd.7 2024-04-22T01:19:21.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-22T01:19:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86bdeca640 1 Processor -- start 2024-04-22T01:19:21.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86bdeca640 1 -- start start 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86bdeca640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f86b80b98a0 0x7f86b80d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86bdeca640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b80b9ce0 con 0x7f86b815de20 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86bdeca640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b80b75e0 con 0x7f86b80cea10 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86bdeca640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f86b80b78c0 con 0x7f86b80b98a0 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f86b815de20 0x7f86b80b6db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36570/0 (socket says 172.21.15.70:36570) 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 -- 172.21.15.70:0/1720875796 learned_addr learned my addr 172.21.15.70:0/1720875796 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86affff640 1 --2- 172.21.15.70:0/1720875796 >> v2:172.21.15.98:3300/0 conn(0x7f86b80b98a0 0x7f86b80d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309161138 0 0) 0x7f86b80b9ce0 con 0x7f86b815de20 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86a4003530 con 0x7f86b815de20 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794599032 0 0) 0x7f86a4003530 con 0x7f86b815de20 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 >> v2:172.21.15.98:3300/0 conn(0x7f86b80b98a0 msgr2=0x7f86b80d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 --2- 172.21.15.70:0/1720875796 >> v2:172.21.15.98:3300/0 conn(0x7f86b80b98a0 0x7f86b80d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 >> v1:172.21.15.98:6789/0 conn(0x7f86b80cea10 legacy=0x7f86b80cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b80b7ba0 con 0x7f86b815de20 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f86a8001680 con 0x7f86b815de20 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 >> v1:172.21.15.70:6789/0 conn(0x7f86b815de20 legacy=0x7f86b80b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40039c0 0x7f86a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a4004810 0x7f86a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a40051d0 0x7f86a400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86b80b78c0 con 0x7f86a40039c0 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86a400c080 con 0x7f86a40051d0 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f86a400c280 con 0x7f86a4004810 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40039c0 0x7f86a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 -- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a4004810 msgr2=0x7f86a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a4004810 0x7f86a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 -- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a40051d0 msgr2=0x7f86a400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b7fff640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a40051d0 0x7f86a400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86a40051d0 0x7f86a400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b7fff640 1 -- 172.21.15.70:0/1720875796 reap_dead start 2024-04-22T01:19:21.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.310+0000 7f86b77fe640 1 -- 172.21.15.70:0/1720875796 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a4003530 con 0x7f86a40039c0 2024-04-22T01:19:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86b77fe640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40039c0 0x7f86a4003de0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f86a800c1c0 tx=0x7f86a801f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84995484cabb89f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8007d00 con 0x7f86a40039c0 2024-04-22T01:19:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86a8005b00 con 0x7f86a40039c0 2024-04-22T01:19:21.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86b6ffd640 1 -- 172.21.15.70:0/1720875796 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8007070 con 0x7f86a40039c0 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- 172.21.15.70:0/1720875796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40039c0 msgr2=0x7f86a4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40039c0 0x7f86a4003de0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f86a800c1c0 tx=0x7f86a801f3d0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- 172.21.15.70:0/1720875796 shutdown_connections 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 --2- 172.21.15.70:0/1720875796 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86a40039c0 0x7f86a4003de0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- 172.21.15.70:0/1720875796 >> 172.21.15.70:0/1720875796 conn(0x7f86b8072d20 msgr2=0x7f86b8073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- 172.21.15.70:0/1720875796 shutdown_connections 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- 172.21.15.70:0/1720875796 wait complete. 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 Processor -- start 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- start start 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a40039c0 0x7f86b8160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86b815ed70 0x7f86b81f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b80ccf90 0x7f86b80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86b815e200 con 0x7f86b815ed70 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86b80b8160 con 0x7f86b80ccf90 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86bdeca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f86b80cee60 con 0x7f86a40039c0 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86b815ed70 0x7f86b81f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86b815ed70 0x7f86b81f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45454/0 (socket says 172.21.15.70:45454) 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86affff640 1 -- 172.21.15.70:0/3732672438 learned_addr learned my addr 172.21.15.70:0/3732672438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86b7fff640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b80ccf90 0x7f86b80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.314+0000 7f86b77fe640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a40039c0 0x7f86b8160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86affff640 1 -- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a40039c0 msgr2=0x7f86b8160ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86affff640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a40039c0 0x7f86b8160ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86affff640 1 -- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b80ccf90 msgr2=0x7f86b80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86affff640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b80ccf90 0x7f86b80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86affff640 1 -- 172.21.15.70:0/3732672438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b8161ae0 con 0x7f86b815ed70 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b77fe640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86a40039c0 0x7f86b8160ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86affff640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86b815ed70 0x7f86b81f2fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f86a000e8a0 tx=0x7f86a000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a0017070 con 0x7f86b815ed70 2024-04-22T01:19:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86a000c940 con 0x7f86b815ed70 2024-04-22T01:19:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b815f550 con 0x7f86b815ed70 2024-04-22T01:19:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a000cd30 con 0x7f86b815ed70 2024-04-22T01:19:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b8162060 con 0x7f86b815ed70 2024-04-22T01:19:21.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86b8068210 con 0x7f86b815ed70 2024-04-22T01:19:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f86a001a070 con 0x7f86b815ed70 2024-04-22T01:19:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86980678f0 0x7f8698069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b77fe640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86980678f0 0x7f8698069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f86a0085db0 con 0x7f86b815ed70 2024-04-22T01:19:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f869806dd90 0x7f86980701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 --> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86a0013e50 con 0x7f869806dd90 2024-04-22T01:19:21.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f86a0086240 con 0x7f86b815ed70 2024-04-22T01:19:21.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b77fe640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86980678f0 0x7f8698069db0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f86a80ae000 tx=0x7f86a8007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b7fff640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f869806dd90 0x7f86980701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.318+0000 7f86b7fff640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f869806dd90 0x7f86980701d0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.322+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== osd.7 v2:172.21.15.98:6824/2897852499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86a0013e50 con 0x7f869806dd90 2024-04-22T01:19:21.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 --> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f86b80b8f80 con 0x7f869806dd90 2024-04-22T01:19:21.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86b4ff9640 1 -- 172.21.15.70:0/3732672438 <== osd.7 v2:172.21.15.98:6824/2897852499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f86b80b8f80 con 0x7f869806dd90 2024-04-22T01:19:21.335 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-22T01:19:21.335 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T01:19:21.335 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-22T01:19:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f869806dd90 msgr2=0x7f86980701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.98:6824/2897852499,v1:172.21.15.98:6825/2897852499] conn(0x7f869806dd90 0x7f86980701d0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86980678f0 msgr2=0x7f8698069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f86980678f0 0x7f8698069db0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f86a80ae000 tx=0x7f86a8007040 comp rx=0 tx=0).stop 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86b815ed70 msgr2=0x7f86b81f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 --2- 172.21.15.70:0/3732672438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86b815ed70 0x7f86b81f2fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f86a000e8a0 tx=0x7f86a000ed60 comp rx=0 tx=0).stop 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86b7fff640 1 -- 172.21.15.70:0/3732672438 reap_dead start 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 shutdown_connections 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 >> 172.21.15.70:0/3732672438 conn(0x7f86b8072d20 msgr2=0x7f86b80b84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 shutdown_connections 2024-04-22T01:19:21.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.334+0000 7f86bdeca640 1 -- 172.21.15.70:0/3732672438 wait complete. 2024-04-22T01:19:21.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-22T01:19:21.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 Processor -- start 2024-04-22T01:19:21.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 -- start start 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f78b815bd00 0x7f78b815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b815c470 con 0x7f78b806b1f0 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f78b80b1cd0 con 0x7f78b815bd00 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bf55d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f78b80b1f80 con 0x7f78b80cdfe0 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f78b815bd00 0x7f78b815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f78b815bd00 0x7f78b815cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45462/0 (socket says 172.21.15.70:45462) 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 -- 172.21.15.70:0/2052890760 learned_addr learned my addr 172.21.15.70:0/2052890760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bdafa640 1 --1- 172.21.15.70:0/2052890760 >> v1:172.21.15.70:6789/0 conn(0x7f78b806b1f0 0x7f78b815f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:36582/0 (socket says 172.21.15.70:36582) 2024-04-22T01:19:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bcaf8640 1 --2- 172.21.15.70:0/2052890760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 -- 172.21.15.70:0/2052890760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 msgr2=0x7f78b80b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 --2- 172.21.15.70:0/2052890760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 -- 172.21.15.70:0/2052890760 >> v1:172.21.15.70:6789/0 conn(0x7f78b806b1f0 legacy=0x7f78b815f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.462+0000 7f78bd2f9640 1 -- 172.21.15.70:0/2052890760 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b815c710 con 0x7f78b815bd00 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bd2f9640 1 --2- 172.21.15.70:0/2052890760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b815cf10 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009870 tx=0x7f78b401cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a141aab5376db12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78affff640 1 -- 172.21.15.70:0/2052890760 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40aa620 con 0x7f78b815bd00 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78affff640 1 -- 172.21.15.70:0/2052890760 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b40af070 con 0x7f78b815bd00 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78affff640 1 -- 172.21.15.70:0/2052890760 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40aa9e0 con 0x7f78b815bd00 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- 172.21.15.70:0/2052890760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 msgr2=0x7f78b815cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 --2- 172.21.15.70:0/2052890760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b815cf10 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009870 tx=0x7f78b401cf00 comp rx=0 tx=0).stop 2024-04-22T01:19:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- 172.21.15.70:0/2052890760 shutdown_connections 2024-04-22T01:19:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 --2- 172.21.15.70:0/2052890760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 --2- 172.21.15.70:0/2052890760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b815cf10 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- 172.21.15.70:0/2052890760 >> 172.21.15.70:0/2052890760 conn(0x7f78b8099a60 msgr2=0x7f78b8099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- 172.21.15.70:0/2052890760 shutdown_connections 2024-04-22T01:19:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- 172.21.15.70:0/2052890760 wait complete. 2024-04-22T01:19:21.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 Processor -- start 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- start start 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b806b1f0 0x7f78b8153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b80b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78b806ba10 con 0x7f78b815bd00 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8168cf0 con 0x7f78b806b1f0 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bf55d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f78b80ca1b0 con 0x7f78b80cdfe0 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bdafa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b80b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bdafa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b80b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45476/0 (socket says 172.21.15.70:45476) 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bdafa640 1 -- 172.21.15.70:0/3928495105 learned_addr learned my addr 172.21.15.70:0/3928495105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bd2f9640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b806b1f0 0x7f78b8153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.466+0000 7f78bcaf8640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bdafa640 1 -- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 msgr2=0x7f78b80c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bdafa640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bdafa640 1 -- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b806b1f0 msgr2=0x7f78b8153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bdafa640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b806b1f0 0x7f78b8153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bdafa640 1 -- 172.21.15.70:0/3928495105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b80ca450 con 0x7f78b815bd00 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bcaf8640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bd2f9640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b806b1f0 0x7f78b8153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bdafa640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b80b0ef0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f78b000c730 tx=0x7f78b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0017070 con 0x7f78b815bd00 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b000ce40 con 0x7f78b815bd00 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b000a820 con 0x7f78b815bd00 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b80ca730 con 0x7f78b815bd00 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b80b53e0 con 0x7f78b815bd00 2024-04-22T01:19:21.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b80b51d0 con 0x7f78b815bd00 2024-04-22T01:19:21.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78b001a070 con 0x7f78b815bd00 2024-04-22T01:19:21.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78adffb640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f789c0678c0 0x7f789c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f78b0085180 con 0x7f78b815bd00 2024-04-22T01:19:21.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bd2f9640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f789c0678c0 0x7f789c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.470+0000 7f78bd2f9640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f789c0678c0 0x7f789c069d80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f78b40ae000 tx=0x7f78b4004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.474+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f78b0055d90 con 0x7f78b815bd00 2024-04-22T01:19:21.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.622+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f78b804f510 con 0x7f789c0678c0 2024-04-22T01:19:21.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.622+0000 7f78adffb640 1 -- 172.21.15.70:0/3928495105 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f78b804f510 con 0x7f789c0678c0 2024-04-22T01:19:21.623 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 7 to deep-scrub 2024-04-22T01:19:21.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f789c0678c0 msgr2=0x7f789c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f789c0678c0 0x7f789c069d80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f78b40ae000 tx=0x7f78b4004740 comp rx=0 tx=0).stop 2024-04-22T01:19:21.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 msgr2=0x7f78b80b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b80b0ef0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f78b000c730 tx=0x7f78b000cbf0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 shutdown_connections 2024-04-22T01:19:21.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f789c0678c0 0x7f789c069d80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78b815bd00 0x7f78b80b0ef0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b80cdfe0 0x7f78b80c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 --2- 172.21.15.70:0/3928495105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b806b1f0 0x7f78b8153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 >> 172.21.15.70:0/3928495105 conn(0x7f78b8099a60 msgr2=0x7f78b815e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 shutdown_connections 2024-04-22T01:19:21.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.626+0000 7f78bf55d640 1 -- 172.21.15.70:0/3928495105 wait complete. 2024-04-22T01:19:21.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:19:21.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 Processor -- start 2024-04-22T01:19:21.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 -- start start 2024-04-22T01:19:21.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7b0815bc70 0x7f7b0815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b0806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b0806b730 0x7f7b080ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7b0806bb80 con 0x7f7b0815bc70 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7b080cfd10 con 0x7f7b0806b730 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0d0ba640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0815fd10 con 0x7f7b080cd740 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7b0815bc70 0x7f7b0815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7b0815bc70 0x7f7b0815f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45492/0 (socket says 172.21.15.70:45492) 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 -- 172.21.15.70:0/1398359562 learned_addr learned my addr 172.21.15.70:0/1398359562 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0759e640 1 --2- 172.21.15.70:0/1398359562 >> v2:172.21.15.98:3300/0 conn(0x7f7b0806b730 0x7f7b080ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b0659c640 1 --2- 172.21.15.70:0/1398359562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b0806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 -- 172.21.15.70:0/1398359562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 msgr2=0x7f7b0806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 --2- 172.21.15.70:0/1398359562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b0806b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 -- 172.21.15.70:0/1398359562 >> v2:172.21.15.98:3300/0 conn(0x7f7b0806b730 msgr2=0x7f7b080ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 --2- 172.21.15.70:0/1398359562 >> v2:172.21.15.98:3300/0 conn(0x7f7b0806b730 0x7f7b080ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.758+0000 7f7b06d9d640 1 -- 172.21.15.70:0/1398359562 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0815c050 con 0x7f7b0815bc70 2024-04-22T01:19:21.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 --2- 172.21.15.70:0/1398359562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0815bc70 0x7f7b0815f7d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7b0806c090 tx=0x7f7af801c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ee8f3ba244311d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b05d9b640 1 -- 172.21.15.70:0/1398359562 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b0070 con 0x7f7b0815bc70 2024-04-22T01:19:21.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b05d9b640 1 -- 172.21.15.70:0/1398359562 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af80ab600 con 0x7f7b0815bc70 2024-04-22T01:19:21.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b05d9b640 1 -- 172.21.15.70:0/1398359562 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80ab9d0 con 0x7f7b0815bc70 2024-04-22T01:19:21.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/1398359562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0815bc70 msgr2=0x7f7b0815f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/1398359562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0815bc70 0x7f7b0815f7d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7b0806c090 tx=0x7f7af801c230 comp rx=0 tx=0).stop 2024-04-22T01:19:21.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/1398359562 shutdown_connections 2024-04-22T01:19:21.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/1398359562 >> v2:172.21.15.98:3300/0 conn(0x7f7b0806b730 0x7f7b080ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/1398359562 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b0806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/1398359562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0815bc70 0x7f7b0815f7d0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/1398359562 >> 172.21.15.70:0/1398359562 conn(0x7f7b08099a60 msgr2=0x7f7b08099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/1398359562 shutdown_connections 2024-04-22T01:19:21.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/1398359562 wait complete. 2024-04-22T01:19:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 Processor -- start 2024-04-22T01:19:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- start start 2024-04-22T01:19:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 0x7f7b08153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b081545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b0815bc70 0x7f7b08154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080d0480 con 0x7f7b0806b730 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080cff60 con 0x7f7b0815bc70 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7b080d0600 con 0x7f7b080cd740 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 0x7f7b08153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 0x7f7b08153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45508/0 (socket says 172.21.15.70:45508) 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 -- 172.21.15.70:0/2626079864 learned_addr learned my addr 172.21.15.70:0/2626079864 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0759e640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b0815bc70 0x7f7b08154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 -- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 msgr2=0x7f7b081545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b081545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 -- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b0815bc70 msgr2=0x7f7b08154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b0815bc70 0x7f7b08154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 -- 172.21.15.70:0/2626079864 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b080b2150 con 0x7f7b0806b730 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b06d9d640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 0x7f7b08153e90 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ae040 tx=0x7f7af8018fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b0040 con 0x7f7b0806b730 2024-04-22T01:19:21.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af80b9070 con 0x7f7b0806b730 2024-04-22T01:19:21.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b080b2430 con 0x7f7b0806b730 2024-04-22T01:19:21.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b4660 con 0x7f7b0806b730 2024-04-22T01:19:21.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.762+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b080b29f0 con 0x7f7b0806b730 2024-04-22T01:19:21.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.766+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b08068920 con 0x7f7b0806b730 2024-04-22T01:19:21.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.766+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7af80b4800 con 0x7f7b0806b730 2024-04-22T01:19:21.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.766+0000 7f7aff7fe640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7ae80678c0 0x7f7ae8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:21.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.766+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f7af811f7a0 con 0x7f7b0806b730 2024-04-22T01:19:21.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.766+0000 7f7b0659c640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7ae80678c0 0x7f7ae8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:21.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.766+0000 7f7b0659c640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7ae80678c0 0x7f7ae8069d80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7b0815a1b0 tx=0x7f7af0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.770+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f7af8016110 con 0x7f7b0806b730 2024-04-22T01:19:21.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.926+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b080cd520 con 0x7f7ae80678c0 2024-04-22T01:19:21.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7aff7fe640 1 -- 172.21.15.70:0/2626079864 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13506 (secure 0 0 0) 0x7f7b080cd520 con 0x7f7ae80678c0 2024-04-22T01:19:21.929 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:21.929 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:19:21.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7ae80678c0 msgr2=0x7f7ae8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7ae80678c0 0x7f7ae8069d80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7b0815a1b0 tx=0x7f7af0008040 comp rx=0 tx=0).stop 2024-04-22T01:19:21.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 msgr2=0x7f7b08153e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 0x7f7b08153e90 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ae040 tx=0x7f7af8018fd0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 shutdown_connections 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f7ae80678c0 0x7f7ae8069d80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b0815bc70 0x7f7b08154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b080cd740 0x7f7b081545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 --2- 172.21.15.70:0/2626079864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b0806b730 0x7f7b08153e90 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 >> 172.21.15.70:0/2626079864 conn(0x7f7b08099a60 msgr2=0x7f7b0815c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 shutdown_connections 2024-04-22T01:19:21.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:21.930+0000 7f7b0d0ba640 1 -- 172.21.15.70:0/2626079864 wait complete. 2024-04-22T01:19:21.948 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-04-22T01:19:20.666331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":277588,"kb_used_data":11496,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754697132,"statfs":{"total":773094113280,"available":772809863168,"internally_reserved":0,"allocated":11771904,"data_stored":6050288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2770109,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-221,"num_write_kb":-23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001952"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":324,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-22T01:19:16.014316+0000","last_change":"2024-04-22T01:14:35.964934+0000","last_active":"2024-04-22T01:19:16.014316+0000","last_peered":"2024-04-22T01:19:16.014316+0000","last_clean":"2024-04-22T01:19:16.014316+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:19:16.014316+0000","last_undegraded":"2024-04-22T01:19:16.014316+0000","last_fullsized":"2024-04-22T01:19:16.014316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:14:34.662376+0000","last_clean_scrub_stamp":"2024-04-22T01:14:34.662376+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:10.011431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1199062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36632,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1916928,"data_stored":1199062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:19:21.951 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-22T01:14:34.662376+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=1, tm_min=14, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=1, tm_min=19, tm_sec=16, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T01:19:21.951 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-22T01:19:41.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:19:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.086+0000 7fc177b79640 1 Processor -- start 2024-04-22T01:19:42.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.086+0000 7fc177b79640 1 -- start start 2024-04-22T01:19:42.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1701455b0 0x7fc1700cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc1700cc150 0x7fc170071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1700d0250 con 0x7fc170072060 2024-04-22T01:19:42.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc1700724d0 con 0x7fc1700cc150 2024-04-22T01:19:42.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc1700cd7b0 con 0x7fc1701455b0 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc1700cc150 0x7fc170071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc1700cc150 0x7fc170071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43146/0 (socket says 172.21.15.70:43146) 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 -- 172.21.15.70:0/2352213654 learned_addr learned my addr 172.21.15.70:0/2352213654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175915640 1 --2- 172.21.15.70:0/2352213654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1701455b0 0x7fc1700cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 -- 172.21.15.70:0/2352213654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1701455b0 msgr2=0x7fc1700cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 --2- 172.21.15.70:0/2352213654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1701455b0 0x7fc1700cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 -- 172.21.15.70:0/2352213654 >> v1:172.21.15.98:6789/0 conn(0x7fc170072060 legacy=0x7fc17015f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 -- 172.21.15.70:0/2352213654 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1700cc710 con 0x7fc1700cc150 2024-04-22T01:19:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc175114640 1 --2- 172.21.15.70:0/2352213654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1700cc150 0x7fc170071b20 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc160009870 tx=0x7fc16001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6823e00c7606dc2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc174913640 1 -- 172.21.15.70:0/2352213654 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1600b0070 con 0x7fc1700cc150 2024-04-22T01:19:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc174913640 1 -- 172.21.15.70:0/2352213654 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1600ab600 con 0x7fc1700cc150 2024-04-22T01:19:42.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc174913640 1 -- 172.21.15.70:0/2352213654 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1600ab9d0 con 0x7fc1700cc150 2024-04-22T01:19:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 -- 172.21.15.70:0/2352213654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1700cc150 msgr2=0x7fc170071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.090+0000 7fc177b79640 1 --2- 172.21.15.70:0/2352213654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1700cc150 0x7fc170071b20 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc160009870 tx=0x7fc16001c440 comp rx=0 tx=0).stop 2024-04-22T01:19:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- 172.21.15.70:0/2352213654 shutdown_connections 2024-04-22T01:19:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 --2- 172.21.15.70:0/2352213654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1700cc150 0x7fc170071b20 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 --2- 172.21.15.70:0/2352213654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1701455b0 0x7fc1700cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- 172.21.15.70:0/2352213654 >> 172.21.15.70:0/2352213654 conn(0x7fc17006b1f0 msgr2=0x7fc17006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- 172.21.15.70:0/2352213654 shutdown_connections 2024-04-22T01:19:42.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- 172.21.15.70:0/2352213654 wait complete. 2024-04-22T01:19:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 Processor -- start 2024-04-22T01:19:42.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- start start 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc170072060 0x7fc1700b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1700cc150 0x7fc17015ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 0x7fc17015e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc170168b40 con 0x7fc1701455b0 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc17006f210 con 0x7fc1700cc150 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc170168cc0 con 0x7fc170072060 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 0x7fc17015e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 0x7fc17015e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43152/0 (socket says 172.21.15.70:43152) 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 -- 172.21.15.70:0/1364533538 learned_addr learned my addr 172.21.15.70:0/1364533538 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc175114640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1700cc150 0x7fc17015ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc175915640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc170072060 0x7fc1700b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 -- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc170072060 msgr2=0x7fc1700b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc170072060 0x7fc1700b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 -- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1700cc150 msgr2=0x7fc17015ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1700cc150 0x7fc17015ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 -- 172.21.15.70:0/1364533538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1701408a0 con 0x7fc1701455b0 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc176116640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 0x7fc17015e2c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c730 tx=0x7fc16800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168003b80 con 0x7fc1701455b0 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc168003d20 con 0x7fc1701455b0 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17015ee80 con 0x7fc1701455b0 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16800aa30 con 0x7fc1701455b0 2024-04-22T01:19:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.094+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17015b610 con 0x7fc1701455b0 2024-04-22T01:19:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.098+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc170068210 con 0x7fc1701455b0 2024-04-22T01:19:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.098+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc16800abd0 con 0x7fc1701455b0 2024-04-22T01:19:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.098+0000 7fc1667fc640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc1540678c0 0x7fc154069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.098+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7fc1680857f0 con 0x7fc1701455b0 2024-04-22T01:19:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.098+0000 7fc175915640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc1540678c0 0x7fc154069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.098+0000 7fc175915640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc1540678c0 0x7fc154069d80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc16c007a60 tx=0x7fc16c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.102+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fc168055ee0 con 0x7fc1701455b0 2024-04-22T01:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.254+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 --> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc170144d90 con 0x7fc1540678c0 2024-04-22T01:19:42.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc1667fc640 1 -- 172.21.15.70:0/1364533538 <== mgr.4108 v2:172.21.15.70:6832/4026573221 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7fc170144d90 con 0x7fc1540678c0 2024-04-22T01:19:42.259 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc1540678c0 msgr2=0x7fc154069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc1540678c0 0x7fc154069d80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc16c007a60 tx=0x7fc16c008040 comp rx=0 tx=0).stop 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 msgr2=0x7fc17015e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 0x7fc17015e2c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c730 tx=0x7fc16800cbf0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 shutdown_connections 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fc1540678c0 0x7fc154069d80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1701455b0 0x7fc17015e2c0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1700cc150 0x7fc17015ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 --2- 172.21.15.70:0/1364533538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc170072060 0x7fc1700b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 >> 172.21.15.70:0/1364533538 conn(0x7fc17006b1f0 msgr2=0x7fc1700b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 shutdown_connections 2024-04-22T01:19:42.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.258+0000 7fc177b79640 1 -- 172.21.15.70:0/1364533538 wait complete. 2024-04-22T01:19:42.274 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-04-22T01:19:40.671529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":277580,"kb_used_data":11496,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754697140,"statfs":{"total":773094113280,"available":772809871360,"internally_reserved":0,"allocated":11771904,"data_stored":6050288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":272354866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.0","version":"106'119","reported_seq":335,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-22T01:19:22.510025+0000","last_change":"2024-04-22T01:19:22.510099+0000","last_active":"2024-04-22T01:19:22.510025+0000","last_peered":"2024-04-22T01:19:22.510025+0000","last_clean":"2024-04-22T01:19:22.510025+0000","last_became_active":"2024-04-22T01:14:35.963165+0000","last_became_peered":"2024-04-22T01:14:35.963165+0000","last_unstale":"2024-04-22T01:19:22.510025+0000","last_undegraded":"2024-04-22T01:19:22.510025+0000","last_fullsized":"2024-04-22T01:19:22.510025+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-04-22T01:19:22.509859+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-04-22T01:19:22.509859+0000","last_clean_scrub_stamp":"2024-04-22T01:19:22.509859+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:35:03.712298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1916928,"data_stored":1199062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36632,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":608694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1916928,"data_stored":1199062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:19:42.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T01:19:42.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f405b4a4640 1 Processor -- start 2024-04-22T01:19:42.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f405b4a4640 1 -- start start 2024-04-22T01:19:42.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f405b4a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f405415de20 0x7f40540b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f405b4a4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40540b75e0 con 0x7f40540cea10 2024-04-22T01:19:42.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f405b4a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40540b78c0 con 0x7f40540b6db0 2024-04-22T01:19:42.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f405b4a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f40540b7ba0 con 0x7f405415de20 2024-04-22T01:19:42.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059a41640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f40540cea10 0x7f40540cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56494/0 (socket says 172.21.15.70:56494) 2024-04-22T01:19:42.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059a41640 1 -- 172.21.15.70:0/3666704162 learned_addr learned my addr 172.21.15.70:0/3666704162 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 --2- 172.21.15.70:0/3666704162 >> v2:172.21.15.98:3300/0 conn(0x7f405415de20 0x7f40540b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2179009946 0 0) 0x7f40540b75e0 con 0x7f40540cea10 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4040003530 con 0x7f40540cea10 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3808606197 0 0) 0x7f4040003530 con 0x7f40540cea10 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 >> v2:172.21.15.98:3300/0 conn(0x7f405415de20 msgr2=0x7f40540b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 --2- 172.21.15.70:0/3666704162 >> v2:172.21.15.98:3300/0 conn(0x7f405415de20 0x7f40540b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 >> v1:172.21.15.98:6789/0 conn(0x7f40540b6db0 legacy=0x7f40540d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40540b7e80 con 0x7f40540cea10 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1918312604 0 0) 0x7f404c001a00 con 0x7f40540cea10 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 >> v1:172.21.15.70:6789/0 conn(0x7f40540cea10 legacy=0x7f40540cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40400039c0 0x7f4040003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4040004810 0x7f4040004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40400051d0 0x7f404000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40540b7ba0 con 0x7f40400039c0 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f404000c070 con 0x7f4040004810 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f404000c270 con 0x7f40400051d0 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40400039c0 0x7f4040003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4058a3f640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4040004810 0x7f4040004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 -- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40400051d0 msgr2=0x7f404000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:42.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40400051d0 0x7f404000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 -- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4040004810 msgr2=0x7f4040004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:42.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4040004810 0x7f4040004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059240640 1 -- 172.21.15.70:0/3666704162 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4040003530 con 0x7f40400039c0 2024-04-22T01:19:42.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4058a3f640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4040004810 0x7f4040004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:19:42.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.394+0000 7f4059a41640 1 -- 172.21.15.70:0/3666704162 reap_dead start 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f4059240640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40400039c0 0x7f4040003de0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4050009ac0 tx=0x7f405001ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83292673f8950d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500b2070 con 0x7f40400039c0 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40500a5610 con 0x7f40400039c0 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f404bfff640 1 -- 172.21.15.70:0/3666704162 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050009690 con 0x7f40400039c0 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 -- 172.21.15.70:0/3666704162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40400039c0 msgr2=0x7f4040003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40400039c0 0x7f4040003de0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4050009ac0 tx=0x7f405001ca40 comp rx=0 tx=0).stop 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 -- 172.21.15.70:0/3666704162 shutdown_connections 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3666704162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40400039c0 0x7f4040003de0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 -- 172.21.15.70:0/3666704162 >> 172.21.15.70:0/3666704162 conn(0x7f4054072d20 msgr2=0x7f4054073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 -- 172.21.15.70:0/3666704162 shutdown_connections 2024-04-22T01:19:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 -- 172.21.15.70:0/3666704162 wait complete. 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 Processor -- start 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.398+0000 7f405b4a4640 1 -- start start 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 0x7f40540c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f405414f6d0 0x7f40540cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40540ccfb0 0x7f40540cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40540b7310 con 0x7f405414f2f0 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f40540b7190 con 0x7f405414f6d0 2024-04-22T01:19:42.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f40540d2190 con 0x7f40540ccfb0 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 0x7f40540c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 0x7f40540c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43178/0 (socket says 172.21.15.70:43178) 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 -- 172.21.15.70:0/3076575390 learned_addr learned my addr 172.21.15.70:0/3076575390 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4058a3f640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f405414f6d0 0x7f40540cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 -- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40540ccfb0 msgr2=0x7f40540cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059a41640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40540ccfb0 0x7f40540cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40540ccfb0 0x7f40540cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 -- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f405414f6d0 msgr2=0x7f40540cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f405414f6d0 0x7f40540cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 -- 172.21.15.70:0/3076575390 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40540cb6f0 con 0x7f405414f2f0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4059240640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 0x7f40540c9ba0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f405001d2e0 tx=0x7f40500197e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500b2040 con 0x7f405414f2f0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40500a5e40 con 0x7f405414f2f0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40540cb9d0 con 0x7f405414f2f0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050009690 con 0x7f405414f2f0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40540cbf90 con 0x7f405414f2f0 2024-04-22T01:19:42.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4054068210 con 0x7f405414f2f0 2024-04-22T01:19:42.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f40500ad640 con 0x7f405414f2f0 2024-04-22T01:19:42.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4049ffb640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f40380678c0 0x7f4038069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7f405011fb30 con 0x7f405414f2f0 2024-04-22T01:19:42.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4058a3f640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f40380678c0 0x7f4038069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.402+0000 7f4058a3f640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f40380678c0 0x7f4038069d80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f40540cde10 tx=0x7f4044009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.406+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7f4050016780 con 0x7f405414f2f0 2024-04-22T01:19:42.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.570+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f40540d33e0 con 0x7f405414f2f0 2024-04-22T01:19:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v22)=0 v22) ==== 123+0+0 (secure 0 0 0) 0x7f40500f0220 con 0x7f405414f2f0 2024-04-22T01:19:42.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f4049ffb640 1 -- 172.21.15.70:0/3076575390 <== mon.0 v2:172.21.15.70:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f40500f4520 con 0x7f405414f2f0 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f40380678c0 msgr2=0x7f4038069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f40380678c0 0x7f4038069d80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f40540cde10 tx=0x7f4044009040 comp rx=0 tx=0).stop 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 msgr2=0x7f40540c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 0x7f40540c9ba0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f405001d2e0 tx=0x7f40500197e0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 shutdown_connections 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7f40380678c0 0x7f4038069d80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40540ccfb0 0x7f40540cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f405414f6d0 0x7f40540cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 --2- 172.21.15.70:0/3076575390 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f405414f2f0 0x7f40540c9ba0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 >> 172.21.15.70:0/3076575390 conn(0x7f4054072d20 msgr2=0x7f40540b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 shutdown_connections 2024-04-22T01:19:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.574+0000 7f405b4a4640 1 -- 172.21.15.70:0/3076575390 wait complete. 2024-04-22T01:19:42.593 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:19:42.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:19:42.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 Processor -- start 2024-04-22T01:19:42.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 -- start start 2024-04-22T01:19:42.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa65015dfb0 0x7fa6500b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6500cea80 0x7fa6500ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa6500b5730 0x7fa6500b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa6500cf420 con 0x7fa6500b5730 2024-04-22T01:19:42.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa65015e390 con 0x7fa65015dfb0 2024-04-22T01:19:42.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa657496640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa650161e00 con 0x7fa6500cea80 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa6500b5730 0x7fa6500b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa6500b5730 0x7fa6500b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43190/0 (socket says 172.21.15.70:43190) 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 -- 172.21.15.70:0/3126699701 learned_addr learned my addr 172.21.15.70:0/3126699701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655232640 1 --2- 172.21.15.70:0/3126699701 >> v2:172.21.15.98:3300/0 conn(0x7fa65015dfb0 0x7fa6500b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 -- 172.21.15.70:0/3126699701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6500cea80 msgr2=0x7fa6500ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 --2- 172.21.15.70:0/3126699701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6500cea80 0x7fa6500ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 -- 172.21.15.70:0/3126699701 >> v2:172.21.15.98:3300/0 conn(0x7fa65015dfb0 msgr2=0x7fa6500b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 --2- 172.21.15.70:0/3126699701 >> v2:172.21.15.98:3300/0 conn(0x7fa65015dfb0 0x7fa6500b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 -- 172.21.15.70:0/3126699701 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6501620e0 con 0x7fa6500b5730 2024-04-22T01:19:42.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.714+0000 7fa655a33640 1 --2- 172.21.15.70:0/3126699701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500b5730 0x7fa6500b5b90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa6500d4d80 tx=0x7fa64801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ebbec1c8c257890 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa647fff640 1 -- 172.21.15.70:0/3126699701 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480ab5f0 con 0x7fa6500b5730 2024-04-22T01:19:42.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa647fff640 1 -- 172.21.15.70:0/3126699701 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa6480b0070 con 0x7fa6500b5730 2024-04-22T01:19:42.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa647fff640 1 -- 172.21.15.70:0/3126699701 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480ab9d0 con 0x7fa6500b5730 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/3126699701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500b5730 msgr2=0x7fa6500b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- 172.21.15.70:0/3126699701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500b5730 0x7fa6500b5b90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa6500d4d80 tx=0x7fa64801c520 comp rx=0 tx=0).stop 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/3126699701 shutdown_connections 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- 172.21.15.70:0/3126699701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500b5730 0x7fa6500b5b90 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- 172.21.15.70:0/3126699701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6500cea80 0x7fa6500ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- 172.21.15.70:0/3126699701 >> v2:172.21.15.98:3300/0 conn(0x7fa65015dfb0 0x7fa6500b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/3126699701 >> 172.21.15.70:0/3126699701 conn(0x7fa65006c9b0 msgr2=0x7fa65006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/3126699701 shutdown_connections 2024-04-22T01:19:42.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/3126699701 wait complete. 2024-04-22T01:19:42.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 Processor -- start 2024-04-22T01:19:42.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- start start 2024-04-22T01:19:42.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500b5730 0x7fa650151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 0x7fa6500b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa65015dfb0 0x7fa6500b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa650161a90 con 0x7fa6500cea80 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6500b9b10 con 0x7fa6500b5730 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa6500d2900 con 0x7fa65015dfb0 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 0x7fa6500b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 0x7fa6500b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43198/0 (socket says 172.21.15.70:43198) 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 -- 172.21.15.70:0/1534168955 learned_addr learned my addr 172.21.15.70:0/1534168955 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa655232640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500b5730 0x7fa650151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 -- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa65015dfb0 msgr2=0x7fa6500b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa65015dfb0 0x7fa6500b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 -- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500b5730 msgr2=0x7fa650151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500b5730 0x7fa650151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 -- 172.21.15.70:0/1534168955 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65015e630 con 0x7fa6500cea80 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa654a31640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 0x7fa6500b3b70 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa64000b550 tx=0x7fa64000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640014070 con 0x7fa6500cea80 2024-04-22T01:19:42.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa640010890 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65015e910 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640010ca0 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.718+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65015eed0 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.722+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa650068210 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.722+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa640017070 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.722+0000 7fa645ffb640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa6340678c0 0x7fa634069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.722+0000 7fa655232640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa6340678c0 0x7fa634069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.722+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5335+0+0 (secure 0 0 0) 0x7fa640085a20 con 0x7fa6500cea80 2024-04-22T01:19:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.722+0000 7fa655232640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa6340678c0 0x7fa634069d80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa6500b98e0 tx=0x7fa64c007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:19:42.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.726+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177503 (secure 0 0 0) 0x7fa640056630 con 0x7fa6500cea80 2024-04-22T01:19:42.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.922+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa65015f0a0 con 0x7fa6500cea80 2024-04-22T01:19:42.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.922+0000 7fa645ffb640 1 -- 172.21.15.70:0/1534168955 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+3866 (secure 0 0 0) 0x7fa640055f30 con 0x7fa6500cea80 2024-04-22T01:19:42.923 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-22T01:19:42.923 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/716406626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":716406626},{"type":"v1","addr":"172.21.15.98:6833","nonce":716406626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6010,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3774812673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3774812673},{"type":"v1","addr":"172.21.15.98:6835","nonce":3774812673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1006953558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1006953558},{"type":"v1","addr":"172.21.15.70:6835","nonce":1006953558}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":7947,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/1206082119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":1206082119},{"type":"v1","addr":"172.21.15.70:6837","nonce":1206082119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[]} 2024-04-22T01:19:42.923 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 52 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa6340678c0 msgr2=0x7fa634069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa6340678c0 0x7fa634069d80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa6500b98e0 tx=0x7fa64c007900 comp rx=0 tx=0).stop 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 msgr2=0x7fa6500b3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 0x7fa6500b3b70 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa64000b550 tx=0x7fa64000ba10 comp rx=0 tx=0).stop 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 shutdown_connections 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:6832/4026573221,v1:172.21.15.70:6833/4026573221] conn(0x7fa6340678c0 0x7fa634069d80 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa65015dfb0 0x7fa6500b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6500cea80 0x7fa6500b3b70 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 --2- 172.21.15.70:0/1534168955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500b5730 0x7fa650151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:19:42.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 >> 172.21.15.70:0/1534168955 conn(0x7fa65006c9b0 msgr2=0x7fa6500b8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:19:42.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 shutdown_connections 2024-04-22T01:19:42.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-22T01:19:42.926+0000 7fa657496640 1 -- 172.21.15.70:0/1534168955 wait complete. 2024-04-22T01:19:42.941 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-22T01:19:42.942 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T01:19:42.942 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-22T01:19:42.942 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:42.951 INFO:tasks.ceph.mds.a:Stopped 2024-04-22T01:19:42.952 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-22T01:19:42.952 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:43.016 INFO:tasks.ceph.mds.c:Stopped 2024-04-22T01:19:43.017 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-22T01:19:43.017 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:43.036 INFO:tasks.ceph.mds.b:Stopped 2024-04-22T01:19:43.037 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-22T01:19:43.037 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:43.102 INFO:tasks.ceph.mds.d:Stopped 2024-04-22T01:19:43.103 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T01:19:43.103 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T01:19:43.103 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:43.179 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T01:19:43.179 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T01:19:43.179 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:43.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:43.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:47.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-22T01:19:49.281 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T01:19:49.281 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T01:19:49.281 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.330 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T01:19:49.331 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T01:19:49.331 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.432 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T01:19:49.433 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-22T01:19:49.433 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.491 INFO:tasks.ceph.osd.4:Stopped 2024-04-22T01:19:49.491 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-22T01:19:49.491 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.591 INFO:tasks.ceph.osd.5:Stopped 2024-04-22T01:19:49.592 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-22T01:19:49.592 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.682 INFO:tasks.ceph.osd.6:Stopped 2024-04-22T01:19:49.683 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-22T01:19:49.683 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.770 INFO:tasks.ceph.osd.7:Stopped 2024-04-22T01:19:49.771 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T01:19:49.771 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-22T01:19:49.771 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.801 INFO:tasks.ceph.mgr.y:Stopped 2024-04-22T01:19:49.802 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T01:19:49.802 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.829 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T01:19:49.830 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T01:19:49.830 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T01:19:49.830 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.863 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T01:19:49.863 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-22T01:19:49.863 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.888 INFO:tasks.ceph.mon.b:Stopped 2024-04-22T01:19:49.888 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T01:19:49.888 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:19:49.942 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T01:19:49.942 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T01:19:49.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-22T01:19:49.960 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:49.961 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T01:19:50.404 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:50.405 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T01:19:50.629 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:50.630 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T01:19:50.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:50.814 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-22T01:19:51.044 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:51.044 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-22T01:19:51.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:51.329 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-22T01:19:51.576 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:51.577 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-22T01:19:51.758 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:51.758 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-22T01:19:51.941 INFO:tasks.ceph:Archiving mon data... 2024-04-22T01:19:51.942 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/data/mon.a.tgz 2024-04-22T01:19:51.942 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-22T01:19:51.948 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.91VUEdE2Cg 2024-04-22T01:19:51.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.91VUEdE2Cg 2024-04-22T01:19:52.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.91VUEdE2Cg 2024-04-22T01:19:52.172 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.91VUEdE2Cg is 492KB 2024-04-22T01:19:52.234 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.91VUEdE2Cg 2024-04-22T01:19:52.240 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/data/mon.b.tgz 2024-04-22T01:19:52.240 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-04-22T01:19:52.246 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.KUPMsi3pp9 2024-04-22T01:19:52.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KUPMsi3pp9 2024-04-22T01:19:52.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.KUPMsi3pp9 2024-04-22T01:19:52.500 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.KUPMsi3pp9 is 638KB 2024-04-22T01:19:52.563 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.KUPMsi3pp9 2024-04-22T01:19:52.568 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/data/mon.c.tgz 2024-04-22T01:19:52.568 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-04-22T01:19:52.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-22T01:19:52.617 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.yyDMw9nRn4 2024-04-22T01:19:52.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yyDMw9nRn4 2024-04-22T01:19:52.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.yyDMw9nRn4 2024-04-22T01:19:52.876 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.yyDMw9nRn4 is 638KB 2024-04-22T01:19:52.939 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.yyDMw9nRn4 2024-04-22T01:19:52.944 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T01:19:52.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T01:19:52.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T01:19:53.006 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T01:19:53.049 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T01:19:53.050 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T01:19:53.051 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/remote/smithi070/crash 2024-04-22T01:19:53.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T01:19:53.066 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/remote/smithi098/crash 2024-04-22T01:19:53.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T01:19:53.105 INFO:tasks.ceph:Shutting down logrotate 2024-04-22T01:19:53.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:53.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:53.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:53.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-22T01:19:53.223 INFO:tasks.ceph:Compressing logs... 2024-04-22T01:19:53.223 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T01:19:53.254 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T01:19:53.278 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36392.log 2024-04-22T01:19:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37719.log 2024-04-22T01:19:53.279 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44622.log 2024-04-22T01:19:53.289 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27717.log 2024-04-22T01:19:53.289 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27813.log 2024-04-22T01:19:53.289 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27693.log 2024-04-22T01:19:53.289 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27717.log.gz 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27813.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27813.log.gz 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.33746.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27693.log.gz 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34649.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.3.33746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34213.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.34649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27741.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.3.33746.log.gz 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.34213.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27837.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27741.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33903.log 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27741.log.gz 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.1.34213.log.gz 2024-04-22T01:19:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27765.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27837.log: 87.5% 91.4% -- replaced with /var/log/ceph/ceph-client.admin.27837.log.gz -- replaced with /var/log/ceph/ceph-client.1.34649.log.gz 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34057.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.33903.log: /var/log/ceph/ceph-client.admin.27765.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.34057.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27765.log.gz 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.1.33903.log.gz 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27669.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34806.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.1.34057.log.gz 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27669.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27669.log.gz 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27789.log 2024-04-22T01:19:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.34806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33626.log 2024-04-22T01:19:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.27789.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27789.log.gz 2024-04-22T01:19:53.294 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T01:19:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.33626.log: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T01:19:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-22T01:19:53.295 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.1.33626.log.gz 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.d.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.b.log: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-22T01:19:53.301 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34380.log 2024-04-22T01:19:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34471.log 2024-04-22T01:19:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.34380.log: 88.3% -- replaced with /var/log/ceph/ceph-client.1.34380.log.gz 2024-04-22T01:19:53.327 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.34471.log: 92.8% -- replaced with /var/log/ceph/ceph-client.1.34471.log.gz 2024-04-22T01:19:53.327 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-22T01:19:53.341 INFO:teuthology.orchestra.run.smithi098.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.1.34806.log.gz 2024-04-22T01:19:53.347 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44282.log 2024-04-22T01:19:53.347 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36392.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34289.log 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44622.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37719.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35466.log 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44622.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44282.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:gzip/var/log/ceph/ceph-client.admin.34289.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43268.log 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35466.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34289.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46863.log 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35466.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37102.log 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43268.log: /var/log/ceph/ceph-client.admin.46863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45884.log 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43268.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46863.log.gz 2024-04-22T01:19:53.348 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45884.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37102.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42930.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45884.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33983.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41299.log: /var/log/ceph/ceph-client.admin.42930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37206.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42930.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38434.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33983.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39803.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37206.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41909.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39803.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38434.log.gz 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44792.log 2024-04-22T01:19:53.349 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39803.log.gz 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33061.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41909.log.gz 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45394.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44792.log.gz 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33395.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33061.log.gz 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46368.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45394.log: /var/log/ceph/ceph-client.admin.33395.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45394.log.gz 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46368.log.gz 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46837.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37518.log 2024-04-22T01:19:53.350 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37648.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.33395.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47076.log 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46837.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34871.log 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37518.log: /var/log/ceph/ceph-client.admin.47076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37518.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47076.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34871.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46738.log 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40330.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.47183.log -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-04-22T01:19:53.351 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.46738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36732.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36124.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36732.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47183.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33801.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36732.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42956.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36124.log: /var/log/ceph/ceph-client.admin.33801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36124.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36863.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33801.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42956.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34290.log 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43508.log: /var/log/ceph/ceph-client.admin.36863.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.0.46738.log.gz 2024-04-22T01:19:53.352 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45336.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36863.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42704.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34290.log: /var/log/ceph/ceph-client.admin.45336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34072.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45336.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34290.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36941.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42704.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.34072.log: /var/log/ceph/ceph-client.admin.45527.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27652.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34072.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36941.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45527.log.gz 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33454.log 2024-04-22T01:19:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.27652.log: /var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47365.log 87.8% 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27652.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33454.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40546.log 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33454.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45030.log 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47365.log: /var/log/ceph/ceph-client.admin.40546.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47365.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45030.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40546.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34687.log 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45030.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45163.log 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.34687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36915.log 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.45163.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34687.log.gz 2024-04-22T01:19:53.354 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45284.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36915.log: /var/log/ceph/ceph-client.admin.45284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46941.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36915.log.gz 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45284.log.gz 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45004.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37154.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46941.log: 91.9% -- replaced with /var/log/ceph/ceph-client.0.45163.log.gz 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47665.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45004.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46941.log.gz 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32561.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37154.log: /var/log/ceph/ceph-client.admin.47665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43324.log 2024-04-22T01:19:53.355 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37154.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37050.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47665.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32561.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.43324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35622.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38566.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37050.log: /var/log/ceph/ceph-client.admin.35622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35701.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37050.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35622.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46915.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43915.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35701.log: /var/log/ceph/ceph-client.admin.46915.log: 92.1% -- replaced with /var/log/ceph/ceph-client.0.43324.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35701.log.gz 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36496.log 2024-04-22T01:19:53.356 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46915.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36229.log 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44336.log 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43915.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36496.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47131.log 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:gzip 88.6%/var/log/ceph/ceph-client.admin.44336.log: -- replaced with /var/log/ceph/ceph-client.admin.36229.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36548.log 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47131.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44336.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38745.log 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47131.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46016.log 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36548.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.0.38566.log.gz 2024-04-22T01:19:53.357 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33827.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36626.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38745.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46016.log: /var/log/ceph/ceph-client.admin.33827.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46016.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47462.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33827.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45446.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36626.log: /var/log/ceph/ceph-client.admin.47462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47391.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36626.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47462.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46998.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47391.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45446.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46998.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47391.log.gz 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45258.log 2024-04-22T01:19:53.358 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46998.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38408.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42255.log: /var/log/ceph/ceph-client.admin.45258.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34296.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45258.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5/var/log/ceph/ceph-client.admin.38408.log: --verbose -- /var/log/ceph/ceph-client.admin.40958.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38408.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34296.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45683.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39065.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45472.log: /var/log/ceph/ceph-client.admin.45683.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37693.log 2024-04-22T01:19:53.359 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45683.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42281.log 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39065.log: /var/log/ceph/ceph-client.admin.37693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39169.log 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39065.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37693.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36150.log 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42281.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39169.log: /var/log/ceph/ceph-client.admin.39449.log 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36150.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.42333.log -- replaced with /var/log/ceph/ceph-client.admin.39169.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39449.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36150.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33611.log 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39449.log.gz 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2024-04-22T01:19:53.360 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9%/var/log/ceph/ceph-client.admin.33611.log: -- replaced with /var/log/ceph/ceph-client.admin.42333.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33611.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41883.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46333.log: /var/log/ceph/ceph-client.admin.40520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37258.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34278.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42070.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37258.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41883.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34278.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37258.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45605.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34278.log.gz 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43996.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42070.log: /var/log/ceph/ceph-client.admin.45605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2024-04-22T01:19:53.361 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45605.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42070.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33715.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.43996.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34479.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44310.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36574.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37024.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33715.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34479.log: /var/log/ceph/ceph-client.admin.36574.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34479.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47024.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36574.log.gz 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40806.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37024.log: /var/log/ceph/ceph-client.admin.47024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45735.log 2024-04-22T01:19:53.362 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47024.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37024.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40387.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.37544.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40806.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45735.log: /var/log/ceph/ceph-client.admin.40387.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45735.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40387.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38304.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37544.log: /var/log/ceph/ceph-client.admin.46528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47977.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37544.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44414.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38304.log.gz 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43806.log 2024-04-22T01:19:53.363 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47951.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44414.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47977.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43806.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44414.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27752.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43806.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42770.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47951.log: /var/log/ceph/ceph-client.admin.27752.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47951.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27752.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46792.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40201.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.2.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42770.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37921.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40201.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46792.log.gz 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35304.log 2024-04-22T01:19:53.364 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40201.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33532.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35753.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8%/var/log/ceph/ceph-client.admin.35304.log: -- replaced with /var/log/ceph/ceph-client.admin.37921.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46502.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35304.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33532.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45761.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35753.log: /var/log/ceph/ceph-client.admin.46502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39606.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35753.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32662.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46502.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45631.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45761.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39606.log.gz 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-04-22T01:19:53.365 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32727.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32662.log.gz 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45631.log: /var/log/ceph/ceph-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39911.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45631.log.gz 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32727.log: gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz /var/log/ceph/ceph-client.admin.42508.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32727.log.gz 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46251.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5%/var/log/ceph/ceph-client.admin.42508.log: -- replaced with /var/log/ceph/ceph-client.admin.39911.log.gz 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 88.4% 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42508.log.gz 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46251.log.gz 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35935.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-04-22T01:19:53.366 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35388.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.35935.log.gz -5 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40854.log: /var/log/ceph/ceph-client.admin.35388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37180.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35388.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48108.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38103.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37180.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33637.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37180.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48108.log.gz 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-22T01:19:53.367 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38103.log: /var/log/ceph/ceph-client.admin.33637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44531.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: 89.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33637.log.gz 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34793.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38103.log.gz 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33506.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44531.log.gz 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34531.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35330.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34793.log.gz 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33506.log: /var/log/ceph/ceph-client.admin.34531.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33506.log.gz 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33480.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34531.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35330.log: /var/log/ceph/ceph-client.admin.33480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48132.log 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35330.log.gz 2024-04-22T01:19:53.368 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43754.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33480.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36967.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48132.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38220.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43754.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41101.log: /var/log/ceph/ceph-client.admin.36967.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33775.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36967.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38220.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36039.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32782.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33775.log.gz 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-04-22T01:19:53.369 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2%/var/log/ceph/ceph-client.admin.36039.log: -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36039.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40252.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47873.log 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33003.log 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose 89.2% -- /var/log/ceph/ceph-client.admin.41167.log -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47873.log: /var/log/ceph/ceph-client.admin.33003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34557.log 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47873.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33003.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37466.log 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41167.log.gz 2024-04-22T01:19:53.370 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47847.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5%/var/log/ceph/ceph-client.admin.37466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39885.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34557.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42203.log 89.6% 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37466.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37947.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47847.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39885.log: /var/log/ceph/ceph-client.admin.42203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42203.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36652.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39885.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37947.log: /var/log/ceph/ceph-client.admin.33191.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37947.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37232.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32977.log 2024-04-22T01:19:53.371 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.37232.log: -- replaced with /var/log/ceph/ceph-client.admin.36652.log.gz 2024-04-22T01:19:53.372 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36758.log 2024-04-22T01:19:53.429 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37232.log.gz 2024-04-22T01:19:53.429 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43088.log 2024-04-22T01:19:53.429 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-04-22T01:19:53.429 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36758.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32977.log.gz 2024-04-22T01:19:53.429 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43088.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44388.log 2024-04-22T01:19:53.429 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36758.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43088.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41550.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45056.log 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35414.log 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44388.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35649.log 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37973.log 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35414.log: /var/log/ceph/ceph-client.admin.35649.log: 88.1%gzip -5 91.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45056.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr: /var/log/ceph/ceph-client.admin.42639.log 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35414.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35649.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37973.log.gz 2024-04-22T01:19:53.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44926.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42639.log.gz 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42096.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7%/var/log/ceph/ceph-client.admin.44926.log: -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42096.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44926.log.gz 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46889.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44596.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42096.log.gz /var/log/ceph/ceph-client.admin.40572.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47925.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38194.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46889.log.gz 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40572.log: /var/log/ceph/ceph-client.admin.47925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47339.log 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40572.log.gz 2024-04-22T01:19:53.431 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47925.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38194.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38194.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36522.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35596.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47339.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39117.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35596.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36522.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39117.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35596.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37777.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39117.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41416.log: /var/log/ceph/ceph-client.admin.37777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34609.log 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-04-22T01:19:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47235.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37777.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34609.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34609.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45553.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45553.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47235.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36257.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0%/var/log/ceph/ceph-client.admin.41992.log: -- replaced with /var/log/ceph/ceph-client.admin.45553.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47717.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36257.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47717.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36257.log.gz 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44440.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35675.log 2024-04-22T01:19:53.433 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47717.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44048.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37336.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35675.log: gzip/var/log/ceph/ceph-client.admin.44048.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34713.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35675.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37336.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48029.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44048.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34713.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.37336.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42613.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39775.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34713.log.gz 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48029.log: /var/log/ceph/ceph-client.admin.42613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36283.log 2024-04-22T01:19:53.434 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42613.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48029.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37622.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39775.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35857.log 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41362.log 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36283.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45964.log 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35805.log 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41362.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35857.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45964.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38828.log 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41362.log.gz 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35805.log: gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.45964.log.gz --verbose 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr: -- /var/log/ceph/ceph-client.admin.45657.log 2024-04-22T01:19:53.435 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35805.log.gz 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36678.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45657.log: /var/log/ceph/ceph-client.admin.39221.log: 89.1% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45657.log.gz 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38828.log.gz 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36678.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36678.log.gz 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40278.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35492.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40278.log: /var/log/ceph/ceph-client.admin.38667.log: gzip -5 --verbose 88.6% -- -- replaced with /var/log/ceph/ceph-client.admin.40278.log.gz /var/log/ceph/ceph-client.admin.39292.log 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.436 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35492.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35492.log.gz 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39684.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr: 87.3%/var/log/ceph/ceph-client.admin.39684.log: -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33905.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48055.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40598.log: /var/log/ceph/ceph-client.admin.33905.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33905.log.gz 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5%/var/log/ceph/ceph-client.admin.43560.log: -- replaced with /var/log/ceph/ceph-client.admin.48055.log.gzgzip 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36309.log 2024-04-22T01:19:53.437 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37492.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42411.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45787.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42411.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36309.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8%/var/log/ceph/ceph-client.admin.45787.log: -- replaced with /var/log/ceph/ceph-client.admin.42411.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35883.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33217.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45787.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46972.log 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-04-22T01:19:53.438 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35883.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46173.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46632.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46972.log: /var/log/ceph/ceph-client.admin.46173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38693.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45579.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46972.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0%/var/log/ceph/ceph-client.admin.38693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46173.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2%/var/log/ceph/ceph-client.admin.45579.log: -- replaced with /var/log/ceph/ceph-client.admin.38693.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.46632.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45579.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45858.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41935.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35570.log 2024-04-22T01:19:53.439 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:gzip/var/log/ceph/ceph-client.admin.45858.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40015.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39580.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45858.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.40015.log: -- replaced with /var/log/ceph/ceph-client.admin.35570.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39580.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40015.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46554.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39580.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47769.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45310.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46554.log: 89.1%/var/log/ceph/ceph-client.admin.47769.log: -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36470.log 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2024-04-22T01:19:53.440 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.38620.log -- replaced with /var/log/ceph/ceph-client.admin.47769.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45310.log: /var/log/ceph/ceph-client.admin.36470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44100.log 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38620.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45310.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36470.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38961.log 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44100.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36600.log 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44100.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37440.log 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38961.log: gzip/var/log/ceph/ceph-client.admin.36600.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43192.log 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37440.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36600.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38131.log 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38961.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43192.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37440.log.gz 2024-04-22T01:19:53.441 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35109.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43192.log.gz 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33558.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38131.log: /var/log/ceph/ceph-client.0.35109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44729.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43404.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38131.log.gz 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44729.log: 91.1% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33558.log.gz 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.0.35109.log.gzgzip 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44729.log.gz 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39482.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44900.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.1.log: 88.4%/var/log/ceph/ceph-client.admin.39482.log: -- replaced with /var/log/ceph/ceph-client.admin.43404.log.gz 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42560.log 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39482.log.gz 2024-04-22T01:19:53.442 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44900.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44900.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36072.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43837.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8%/var/log/ceph/ceph-client.admin.36072.log: -- replaced with /var/log/ceph/ceph-client.admin.42560.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34583.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36072.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43837.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43837.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36784.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47795.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34583.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47491.log 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36784.log: /var/log/ceph/ceph-client.admin.41010.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36784.log.gz 2024-04-22T01:19:53.443 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40356.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40091.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0%/var/log/ceph/ceph-client.admin.40356.log: -- replaced with /var/log/ceph/ceph-client.admin.47491.log.gzgzip 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40356.log.gz 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40091.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40091.log.gz 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38987.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41576.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40465.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr: 90.8% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38987.log.gz 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41576.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41576.log.gz 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-04-22T01:19:53.444 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37128.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40465.log: /var/log/ceph/ceph-client.admin.41678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37128.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40465.log.gz 88.1% 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22725.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37128.log.gz 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45938.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph.tmp-client.admin.22725.log: -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34084.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22725.log.gz 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45938.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45938.log.gz 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47261.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33741.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45912.log 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34084.log.gz 2024-04-22T01:19:53.445 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33741.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47261.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33741.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32914.log 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42456.log 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47105.log 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45912.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32914.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42456.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42456.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43889.log 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39829.log 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43889.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.2.35254.log 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47105.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39829.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43889.log.gz 2024-04-22T01:19:53.446 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39829.log.gz 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48003.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.2.35254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34087.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48003.log.gz/var/log/ceph/ceph-client.admin.34087.log: 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.2.35254.log.gz 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34087.log.gz 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33087.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37596.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33087.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36993.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36098.log 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33087.log.gz/var/log/ceph/ceph-client.admin.37596.log: 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36993.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37596.log.gz 2024-04-22T01:19:53.447 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36993.log.gz 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33931.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33931.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36098.log.gz 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42848.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33113.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9%/var/log/ceph/ceph-client.admin.34661.log: -- replaced with /var/log/ceph/ceph-client.admin.33931.log.gz 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42848.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42848.log.gz 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46476.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39396.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46476.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33113.log.gz 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36889.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37842.log 2024-04-22T01:19:53.448 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8%/var/log/ceph/ceph-client.admin.39396.log: -- replaced with /var/log/ceph/ceph-client.admin.46476.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36889.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39396.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36889.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33957.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37570.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37842.log: /var/log/ceph/ceph-client.admin.33957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35987.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37842.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5%/var/log/ceph/ceph-client.admin.37570.log: -- replaced with /var/log/ceph/ceph-client.admin.33957.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37570.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46277.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41069.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46277.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35987.log.gz 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47899.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T01:19:53.449 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46277.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41069.log: /var/log/ceph/ceph-client.admin.47899.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41069.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47899.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44874.log 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40624.log 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44874.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44874.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38356.log 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:gzip/var/log/ceph/ceph-client.admin.40624.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34293.log 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38356.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36335.log 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40624.log.gz 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38356.log.gz/var/log/ceph/ceph-client.admin.34293.log: 2024-04-22T01:19:53.450 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34293.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33267.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34984.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36335.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38159.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33267.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44675.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34984.log: /var/log/ceph/ceph-client.admin.38159.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34984.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38159.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43780.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43780.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44675.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27533.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35727.log 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1%/var/log/ceph/ceph-client.admin.44178.log: -- replaced with /var/log/ceph/ceph-client.admin.43780.log.gz 2024-04-22T01:19:53.451 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.27533.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33879.log 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.27533.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43140.log 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33035.log 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35727.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33879.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43140.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43140.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36176.log 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6%/var/log/ceph/ceph-client.admin.39318.log: -- replaced with /var/log/ceph/ceph-client.admin.33035.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35961.log 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36176.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36176.log.gz 2024-04-22T01:19:53.452 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45106.log 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43062.log 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2%/var/log/ceph/ceph-client.admin.45106.log: -- replaced with /var/log/ceph/ceph-client.admin.35961.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38025.log 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45106.log.gz 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43114.log 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33689.log 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1%/var/log/ceph/ceph-client.admin.38025.log: -- replaced with /var/log/ceph/ceph-client.admin.43062.log.gz 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43114.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38025.log.gz 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43114.log.gz 2024-04-22T01:19:53.453 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36444.log 2024-04-22T01:19:53.471 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2024-04-22T01:19:53.471 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2%/var/log/ceph/ceph-client.admin.36444.log: -- replaced with /var/log/ceph/ceph-client.admin.33689.log.gz 2024-04-22T01:19:53.471 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34845.log 2024-04-22T01:19:53.471 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36444.log.gz 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42229.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47587.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32942.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6%/var/log/ceph/ceph-client.admin.47587.log: -- replaced with /var/log/ceph/ceph-client.admin.34845.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47587.log.gz 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32942.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32942.log.gz 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39143.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1%/var/log/ceph/ceph-client.admin.39143.log: -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39143.log.gz 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38077.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44499.log 2024-04-22T01:19:53.472 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38802.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9%/var/log/ceph/ceph-client.admin.44499.log: -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43008.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44499.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38802.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38802.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47209.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34897.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43008.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38460.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34897.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47209.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39039.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34298.log 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34897.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38460.log: /var/log/ceph/ceph-client.admin.39039.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39039.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.38460.log.gz 2024-04-22T01:19:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2%/var/log/ceph/ceph-client.admin.40439.log: -- replaced with /var/log/ceph/ceph-client.admin.34298.log.gz 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46075.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41857.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43702.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.40676.log: -- replaced with /var/log/ceph/ceph-client.admin.46075.log.gz 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43702.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43702.log.gz 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36361.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44204.log 2024-04-22T01:19:53.474 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4%/var/log/ceph/ceph-client.admin.36361.log: -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36013.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36361.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44204.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44204.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44362.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47691.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4%/var/log/ceph/ceph-client.admin.44362.log: -- replaced with /var/log/ceph/ceph-client.admin.36013.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42151.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44362.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47691.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47691.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45813.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6%/var/log/ceph/ceph-client.admin.37362.log: -- replaced with /var/log/ceph/ceph-client.admin.42151.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38051.log 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2024-04-22T01:19:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45813.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45813.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47517.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42385.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47517.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38051.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34081.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39091.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47517.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42385.log: /var/log/ceph/ceph-client.admin.34081.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34081.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42385.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43166.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43034.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6%/var/log/ceph/ceph-client.admin.43166.log: -- replaced with /var/log/ceph/ceph-client.admin.39091.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39013.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43166.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43034.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43034.log.gz 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37076.log 2024-04-22T01:19:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37076.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39013.log.gz 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37076.log.gz 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43612.log: /var/log/ceph/ceph.audit.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34505.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39370.log: /var/log/ceph/ceph-client.admin.34505.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34505.log.gz 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33139.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46450.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33165.log 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.46450.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46450.log.gz 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43638.log 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46199.log 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33165.log: /var/log/ceph/ceph-client.admin.43638.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43638.log.gz 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42796.log 2024-04-22T01:19:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mds.a.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46199.log.gz 2024-04-22T01:19:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44952.log 2024-04-22T01:19:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42796.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42796.log.gz 2024-04-22T01:19:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43941.log 2024-04-22T01:19:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44952.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44952.log.gz 2024-04-22T01:19:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38382.log 2024-04-22T01:19:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43941.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43941.log.gz 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35909.log 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38382.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38382.log.gz 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34082.log 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35909.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35909.log.gz 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41524.log 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34082.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34082.log.gz 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41524.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35518.log 2024-04-22T01:19:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41446.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2024-04-22T01:19:53.484 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-22T01:19:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38330.log 2024-04-22T01:19:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35518.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35518.log.gz 2024-04-22T01:19:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-04-22T01:19:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38330.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38330.log.gz 2024-04-22T01:19:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40880.log 2024-04-22T01:19:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42018.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-04-22T01:19:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2024-04-22T01:19:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40880.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40880.log.gz 2024-04-22T01:19:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2024-04-22T01:19:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44701.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2024-04-22T01:19:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35362.log 2024-04-22T01:19:53.490 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46420.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2024-04-22T01:19:53.490 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2024-04-22T01:19:53.491 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35362.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35362.log.gz 2024-04-22T01:19:53.492 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39658.log 2024-04-22T01:19:53.492 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40932.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2024-04-22T01:19:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48158.log 2024-04-22T01:19:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39658.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39658.log.gz 2024-04-22T01:19:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47821.log 2024-04-22T01:19:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48158.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48158.log.gz 2024-04-22T01:19:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32860.log 2024-04-22T01:19:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47821.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47821.log.gz 2024-04-22T01:19:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47287.log 2024-04-22T01:19:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32860.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32860.log.gz 2024-04-22T01:19:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43728.log 2024-04-22T01:19:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47287.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47287.log.gz 2024-04-22T01:19:53.499 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37999.log 2024-04-22T01:19:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43728.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43728.log.gz 2024-04-22T01:19:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42902.log 2024-04-22T01:19:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37999.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37999.log.gz 2024-04-22T01:19:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40491.log 2024-04-22T01:19:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42902.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42902.log.gz 2024-04-22T01:19:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-04-22T01:19:53.504 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40491.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40491.log.gz 2024-04-22T01:19:53.504 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32808.log 2024-04-22T01:19:53.505 INFO:teuthology.orchestra.run.smithi098.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-22T01:19:53.506 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36202.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-04-22T01:19:53.506 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37868.log 2024-04-22T01:19:53.506 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32808.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32808.log.gz 2024-04-22T01:19:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42359.log 2024-04-22T01:19:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37868.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37868.log.gz 2024-04-22T01:19:53.508 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46580.log 2024-04-22T01:19:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42359.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42359.log.gz 2024-04-22T01:19:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2024-04-22T01:19:53.510 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46580.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46580.log.gz 2024-04-22T01:19:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47561.log 2024-04-22T01:19:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40780.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2024-04-22T01:19:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44978.log 2024-04-22T01:19:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47561.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47561.log.gz 2024-04-22T01:19:53.513 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39422.log 2024-04-22T01:19:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44978.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44978.log.gz 2024-04-22T01:19:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39712.log 2024-04-22T01:19:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39422.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39422.log.gz 2024-04-22T01:19:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43378.log 2024-04-22T01:19:53.517 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39712.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39712.log.gz 2024-04-22T01:19:53.517 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45420.log 2024-04-22T01:19:53.518 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43378.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43378.log.gz 2024-04-22T01:19:53.518 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34635.log 2024-04-22T01:19:53.519 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45420.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45420.log.gz 2024-04-22T01:19:53.520 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2024-04-22T01:19:53.520 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34635.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34635.log.gz 2024-04-22T01:19:53.521 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42177.log 2024-04-22T01:19:53.522 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43218.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2024-04-22T01:19:53.523 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-04-22T01:19:53.523 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42177.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz 2024-04-22T01:19:53.524 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36418.log 2024-04-22T01:19:53.525 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44757.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-04-22T01:19:53.525 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33853.log 2024-04-22T01:19:53.526 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36418.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36418.log.gz 2024-04-22T01:19:53.526 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43586.log 2024-04-22T01:19:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33853.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33853.log.gz 2024-04-22T01:19:53.528 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39632.log 2024-04-22T01:19:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43586.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43586.log.gz 2024-04-22T01:19:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2024-04-22T01:19:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39632.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39632.log.gz 2024-04-22T01:19:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-04-22T01:19:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47436.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2024-04-22T01:19:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42586.log 2024-04-22T01:19:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42125.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gz 2024-04-22T01:19:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38278.log 2024-04-22T01:19:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42586.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42586.log.gz 2024-04-22T01:19:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2024-04-22T01:19:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38278.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38278.log.gz 2024-04-22T01:19:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-04-22T01:19:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46042.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2024-04-22T01:19:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2024-04-22T01:19:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39514.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-04-22T01:19:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-04-22T01:19:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44230.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2024-04-22T01:19:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-04-22T01:19:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38854.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-04-22T01:19:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37388.log 2024-04-22T01:19:53.542 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44126.log: 88.7% 91.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-22T01:19:53.542 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-04-22T01:19:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46107.log 2024-04-22T01:19:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43430.log 2024-04-22T01:19:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37388.log: /var/log/ceph/ceph-client.admin.46107.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37388.log.gz 2024-04-22T01:19:53.543 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46107.log.gz 2024-04-22T01:19:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41734.log 2024-04-22T01:19:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-04-22T01:19:53.544 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43430.log: /var/log/ceph/ceph-client.0.41734.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43430.log.gz 2024-04-22T01:19:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32753.log 2024-04-22T01:19:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40413.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-04-22T01:19:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38906.log 2024-04-22T01:19:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32753.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32753.log.gz 2024-04-22T01:19:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34009.log 2024-04-22T01:19:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38906.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38906.log.gz 2024-04-22T01:19:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39195.log 2024-04-22T01:19:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34009.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34009.log.gz 2024-04-22T01:19:53.550 INFO:teuthology.orchestra.run.smithi070.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.0.41734.log.gz 2024-04-22T01:19:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32886.log 2024-04-22T01:19:53.551 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45362.log 2024-04-22T01:19:53.551 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39195.log: /var/log/ceph/ceph-client.admin.32886.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2024-04-22T01:19:53.551 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32886.log.gz 2024-04-22T01:19:53.551 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37745.log 2024-04-22T01:19:53.552 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2024-04-22T01:19:53.552 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45362.log: /var/log/ceph/ceph-client.admin.37745.log: 88.6% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37745.log.gz -- replaced with /var/log/ceph/ceph-client.admin.45362.log.gz 2024-04-22T01:19:53.552 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44648.log 2024-04-22T01:19:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-04-22T01:19:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40650.log: /var/log/ceph/ceph-client.admin.44648.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2024-04-22T01:19:53.554 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44648.log.gz 2024-04-22T01:19:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43970.log 2024-04-22T01:19:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41327.log 2024-04-22T01:19:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41193.log: /var/log/ceph/ceph-client.admin.43970.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2024-04-22T01:19:53.555 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43970.log.gz 2024-04-22T01:19:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36836.log 2024-04-22T01:19:53.556 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38510.log 2024-04-22T01:19:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41327.log: /var/log/ceph/ceph-client.admin.36836.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41327.log.gz 2024-04-22T01:19:53.557 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36836.log.gz 2024-04-22T01:19:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44152.log 2024-04-22T01:19:53.558 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40728.log 2024-04-22T01:19:53.558 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38510.log: /var/log/ceph/ceph-client.admin.44152.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38510.log.gz 2024-04-22T01:19:53.558 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44152.log.gz 2024-04-22T01:19:53.559 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36706.log 2024-04-22T01:19:53.559 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47313.log 2024-04-22T01:19:53.559 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1%/var/log/ceph/ceph-client.admin.36706.log: -- replaced with /var/log/ceph/ceph-client.admin.40728.log.gz 2024-04-22T01:19:53.559 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36706.log.gz 2024-04-22T01:19:53.560 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-04-22T01:19:53.560 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42982.log 2024-04-22T01:19:53.560 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47313.log: /var/log/ceph/ceph-client.admin.41628.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47313.log.gz 2024-04-22T01:19:53.561 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37894.log 2024-04-22T01:19:53.562 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42982.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-04-22T01:19:53.562 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42982.log.gz 2024-04-22T01:19:53.562 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34292.log 2024-04-22T01:19:53.562 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37310.log 2024-04-22T01:19:53.563 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37894.log: /var/log/ceph/ceph-client.admin.34292.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37894.log.gz 2024-04-22T01:19:53.563 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34292.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47613.log 2024-04-22T01:19:53.564 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.564 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37310.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37310.log.gz 2024-04-22T01:19:53.564 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39989.log 2024-04-22T01:19:53.564 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39740.log 2024-04-22T01:19:53.565 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47613.log: /var/log/ceph/ceph-client.admin.39989.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47613.log.gz 2024-04-22T01:19:53.565 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39989.log.gz 2024-04-22T01:19:53.565 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42307.log 2024-04-22T01:19:53.566 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39740.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39740.log.gz 2024-04-22T01:19:53.566 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-04-22T01:19:53.567 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33663.log 2024-04-22T01:19:53.568 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42307.log: /var/log/ceph/ceph-client.admin.39937.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42307.log.gz 2024-04-22T01:19:53.568 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-04-22T01:19:53.568 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43668.log 2024-04-22T01:19:53.568 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47639.log 2024-04-22T01:19:53.569 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33663.log: /var/log/ceph/ceph-client.admin.43668.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33663.log.gz 2024-04-22T01:19:53.569 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43668.log.gz 2024-04-22T01:19:53.569 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40041.log 2024-04-22T01:19:53.570 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47639.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47639.log.gz 2024-04-22T01:19:53.570 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35440.log 2024-04-22T01:19:53.571 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46682.log 2024-04-22T01:19:53.571 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40041.log: /var/log/ceph/ceph-client.admin.35440.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35440.log.gz 2024-04-22T01:19:53.571 INFO:teuthology.orchestra.run.smithi070.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40041.log.gz 2024-04-22T01:19:53.572 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32629.log 2024-04-22T01:19:53.573 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46305.log 2024-04-22T01:19:53.573 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46682.log: /var/log/ceph/ceph-client.admin.32629.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46682.log.gz 2024-04-22T01:19:53.573 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32629.log.gz 2024-04-22T01:19:53.573 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34086.log 2024-04-22T01:19:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2024-04-22T01:19:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46305.log: /var/log/ceph/ceph-client.admin.34086.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46305.log.gz 2024-04-22T01:19:53.575 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34086.log.gz 2024-04-22T01:19:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-04-22T01:19:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32834.log 2024-04-22T01:19:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38935.log: /var/log/ceph/ceph-client.admin.41498.log: 88.1% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz 2024-04-22T01:19:53.576 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-04-22T01:19:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2024-04-22T01:19:53.577 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35159.log 2024-04-22T01:19:53.577 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32834.log: /var/log/ceph/ceph-client.admin.42822.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32834.log.gz 2024-04-22T01:19:53.577 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2024-04-22T01:19:53.577 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48081.log 2024-04-22T01:19:53.578 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47157.log 2024-04-22T01:19:53.578 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35159.log: /var/log/ceph/ceph-client.admin.48081.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35159.log.gz 2024-04-22T01:19:53.579 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48081.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-04-22T01:19:53.579 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.579 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47157.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47157.log.gz 2024-04-22T01:19:53.580 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44466.log 2024-04-22T01:19:53.580 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37284.log 2024-04-22T01:19:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41788.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz/var/log/ceph/ceph-client.admin.44466.log: 2024-04-22T01:19:53.581 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47743.log 2024-04-22T01:19:53.581 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44466.log.gz 2024-04-22T01:19:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45709.log 2024-04-22T01:19:53.582 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37284.log.gz 2024-04-22T01:19:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47743.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47743.log.gz 2024-04-22T01:19:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2024-04-22T01:19:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-04-22T01:19:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45709.log: /var/log/ceph/ceph-client.admin.38771.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45709.log.gz 2024-04-22T01:19:53.584 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2024-04-22T01:19:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34299.log 2024-04-22T01:19:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41961.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-04-22T01:19:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42874.log 2024-04-22T01:19:53.586 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33584.log 2024-04-22T01:19:53.586 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34299.log: /var/log/ceph/ceph-client.admin.42874.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34299.log.gz 2024-04-22T01:19:53.587 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42874.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36810.log 2024-04-22T01:19:53.587 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33584.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33584.log.gz 2024-04-22T01:19:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34035.log 2024-04-22T01:19:53.588 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-04-22T01:19:53.589 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.36810.log: /var/log/ceph/ceph-client.admin.34035.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36810.log.gz 2024-04-22T01:19:53.589 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34035.log.gz 2024-04-22T01:19:53.589 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46394.log 2024-04-22T01:19:53.589 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43863.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-04-22T01:19:53.590 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34739.log 2024-04-22T01:19:53.590 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-04-22T01:19:53.591 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46394.log: /var/log/ceph/ceph-client.admin.34739.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46394.log.gz 2024-04-22T01:19:53.592 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34739.log.gz 2024-04-22T01:19:53.592 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2024-04-22T01:19:53.592 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40906.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-04-22T01:19:53.592 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39963.log 2024-04-22T01:19:53.593 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2024-04-22T01:19:53.593 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44022.log: /var/log/ceph/ceph-client.admin.39963.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2024-04-22T01:19:53.593 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-04-22T01:19:53.593 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44256.log 2024-04-22T01:19:53.595 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42534.log 2024-04-22T01:19:53.595 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41271.log: /var/log/ceph/ceph-client.admin.44256.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-04-22T01:19:53.595 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44256.log.gz 2024-04-22T01:19:53.595 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42044.log 2024-04-22T01:19:53.596 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34088.log 2024-04-22T01:19:53.596 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42534.log: /var/log/ceph/ceph-client.admin.42044.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42534.log.gz 2024-04-22T01:19:53.596 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42044.log.gz 2024-04-22T01:19:53.596 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44844.log 2024-04-22T01:19:53.597 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47050.log 2024-04-22T01:19:53.597 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34088.log: /var/log/ceph/ceph-client.admin.44844.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34088.log.gz 2024-04-22T01:19:53.597 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44844.log.gz 2024-04-22T01:19:53.597 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45990.log 2024-04-22T01:19:53.599 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45498.log 2024-04-22T01:19:53.599 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47050.log: /var/log/ceph/ceph-client.admin.45990.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47050.log.gz 2024-04-22T01:19:53.599 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45990.log.gz 2024-04-22T01:19:53.599 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37414.log 2024-04-22T01:19:53.600 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46606.log 2024-04-22T01:19:53.600 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45498.log: /var/log/ceph/ceph-client.admin.37414.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37414.log.gz 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45498.log.gz 2024-04-22T01:19:53.600 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:53.601 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35831.log 2024-04-22T01:19:53.601 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2024-04-22T01:19:53.601 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46606.log: /var/log/ceph/ceph-client.admin.35831.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46606.log.gz 2024-04-22T01:19:53.602 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35831.log.gz 2024-04-22T01:19:53.602 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34089.log 2024-04-22T01:19:53.603 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2024-04-22T01:19:53.603 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42672.log: /var/log/ceph/ceph-client.admin.34089.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gz 2024-04-22T01:19:53.603 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34089.log.gz 2024-04-22T01:19:53.603 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-04-22T01:19:53.603 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.40147.log 2024-04-22T01:19:53.604 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41245.log: /var/log/ceph/ceph-client.admin.45217.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2024-04-22T01:19:53.604 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-04-22T01:19:53.604 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38719.log 2024-04-22T01:19:53.605 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35779.log 2024-04-22T01:19:53.605 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.40147.log: /var/log/ceph/ceph-client.admin.38719.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38719.log.gz 2024-04-22T01:19:53.607 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.0.40147.log.gz 2024-04-22T01:19:53.608 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.35779.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35779.log.gz 2024-04-22T01:19:53.644 INFO:teuthology.orchestra.run.smithi070.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-22T01:19:53.944 INFO:teuthology.orchestra.run.smithi070.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-22T01:19:54.171 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-22T01:19:54.223 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-22T01:19:54.257 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-22T01:19:54.285 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-22T01:19:54.287 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-04-22T01:19:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m1.018s 2024-04-22T01:19:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m4.214s 2024-04-22T01:19:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.135s 2024-04-22T01:19:54.340 INFO:teuthology.orchestra.run.smithi070.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-22T01:19:54.368 INFO:teuthology.orchestra.run.smithi070.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-22T01:19:54.461 INFO:teuthology.orchestra.run.smithi070.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-22T01:19:54.504 INFO:teuthology.orchestra.run.smithi070.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-22T01:19:54.506 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-22T01:19:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m1.250s 2024-04-22T01:19:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m5.581s 2024-04-22T01:19:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.352s 2024-04-22T01:19:54.506 INFO:tasks.ceph:Archiving logs... 2024-04-22T01:19:54.506 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/remote/smithi070/log 2024-04-22T01:19:54.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T01:19:55.624 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/remote/smithi098/log 2024-04-22T01:19:55.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T01:19:56.131 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T01:19:56.140 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T01:19:56.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T01:19:56.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T01:19:56.205 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T01:19:56.215 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T01:19:56.215 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:19:56.218 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:19:56.229 INFO:teuthology.orchestra.run.smithi070.stdout: remote refid st t when poll reach delay offset jitter 2024-04-22T01:19:56.229 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================== 2024-04-22T01:19:56.229 INFO:teuthology.orchestra.run.smithi070.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 377 0.082 -2.631 1.171 2024-04-22T01:19:56.229 INFO:teuthology.orchestra.run.smithi070.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.082 -2.078 0.892 2024-04-22T01:19:56.229 INFO:teuthology.orchestra.run.smithi070.stdout:+hv03.front.sepi 74.6.168.72 3 u 12 64 377 0.093 -1.874 0.981 2024-04-22T01:19:56.229 INFO:teuthology.orchestra.run.smithi070.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:19:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-04-22T01:19:56.232 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-04-22T01:19:56.232 INFO:teuthology.orchestra.run.smithi098.stdout:*hv01.front.sepi 44.4.53.2 2 u 12 64 377 0.089 -0.318 0.534 2024-04-22T01:19:56.232 INFO:teuthology.orchestra.run.smithi098.stdout:+hv02.front.sepi 63.231.80.2 3 u 7 64 377 0.072 -0.728 0.318 2024-04-22T01:19:56.232 INFO:teuthology.orchestra.run.smithi098.stdout:+hv03.front.sepi 74.6.168.72 3 u 13 64 377 0.081 -0.862 0.482 2024-04-22T01:19:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T01:19:56.233 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T01:19:56.242 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T01:19:56.243 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T01:19:56.252 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T01:19:56.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T01:19:56.271 INFO:teuthology.task.internal:Duration was 962.063426 seconds 2024-04-22T01:19:56.272 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T01:19:56.281 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T01:19:56.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T01:19:56.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T01:19:56.319 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T01:19:56.319 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-22T01:19:56.320 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T01:19:56.368 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T01:19:56.369 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T01:19:56.379 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T01:19:56.379 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T01:19:56.413 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T01:19:56.446 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T01:19:56.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T01:19:56.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T01:19:56.560 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T01:19:56.571 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T01:19:56.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T01:19:56.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T01:19:56.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T01:19:56.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:56.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:56.640 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-22T01:19:56.650 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-04-22T01:19:56.661 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:56.699 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:56.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T01:19:56.719 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T01:19:56.720 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/remote/smithi070 2024-04-22T01:19:56.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T01:19:56.759 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667725/remote/smithi098 2024-04-22T01:19:56.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T01:19:56.780 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T01:19:56.780 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T01:19:56.797 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T01:19:56.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T01:19:56.833 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T01:19:56.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T01:19:56.843 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T01:19:56.843 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T01:19:56.845 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T01:19:56.849 INFO:teuthology.orchestra.run.smithi070.stdout: 658638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 01:19 /home/ubuntu/cephtest 2024-04-22T01:19:56.869 INFO:teuthology.orchestra.run.smithi098.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 01:19 /home/ubuntu/cephtest 2024-04-22T01:19:56.870 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T01:19:56.881 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T01:19:56.905 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quota} duration: 962.0634257793427 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-22T01:19:56.906 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T01:19:56.994 INFO:teuthology.run:pass